Comparison of Join algorithms in MapReduce Framework

Comparison of Join algorithms in MapReduce Framework Mani Bhushan, Balaraj J, Oinam Martina Devi Abstract: In the current technological world, there is generation of enormous data each and every day by different media and social networks. The MapReduce framework is increasingly being used widely to analyse large volumes of data. One of the techniques that framework is join algorithm. Join algorithms can be divided into two groups: Reduce-side join and Map-side join. The aim of our work is to compare existing join algorithms which are used by the MapReduce framework. We have compared Reducer-side merge join and Map-side replication-join in terms of pre-processing, the number of phases involved, whether it is sensitive to data skew, whether there is need for distributed Cache, memory overflow. The objective is to determine which algorithm holds well in given scenario. I INTRODUCTION Data-intensive applications include large-scale data warehouse systems, cloud computing, data-intensive analysis. Applications for large-scale data analysis use MapReduce (MR) paradigm [6]. MAPREDUCE is a programming model for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs and a reduce function that merges all intermediate values associated with the same intermediate key [5]. Let us look upon the execution of MapReduce execution. MapReduce Execution: The Map/Reduce framework consists of two operations, â€Å"map† and â€Å"reduce†, which are executed on a cluster of shared-nothing commodity nodes. In a map operation, the input data available through a distributed file system, is distributed among a number of nodes in the cluster in the form of key-value pairs. Each of these mapper nodes transforms a key-value pair into a list of intermediate key-value pairs [1]. The intermediate key-value pairs are propagated to the reducer nodes such that each reduce process receives values related to one key. The values are processed and the result is written to the file system [1]. Figure 1.1: MR execution in detail [7]. In [3], the authors have described crucial implementation details of a number of well-known join strategies in MapReduce, and present a comprehensive experimental comparison of these join techniques on a 100-node Hadoop cluster. The authors have provided the overview of MapReduce overall. They have described how to implement several equijoin algorithms for log processing in MapReduce. They have used the MapReduce framework as it is, without any modification. Therefore, the support for fault tolerance and load balancing in MapReduce is preserved. They have worked on Repartition Join, Broadcast Join, Semi-Join, and Per-Split Semi-Join. The authors have revealed many details that make the implementation more efficient. We have evaluated the join methods on a 100-node system and shown the unique tradeoffs of these join algorithms in the context of MapReduce. We have also explored how our join algorithms can benefit from certain types of practical preprocessing techniques. In [4], the authors have examined the algorithms for performing equi-joins between datasets over Map/Reduce and have provided a comparative analysis. The results indicate that all join algorithms are significantly affected by certain properties of the input datasets (size, selectivity factor, etc.) and that each algorithm performs better under certain circumstances. Our cost model manages to capture these factors and estimates fairly accurately the performance of each algorithm. II COMPARISON OF ALGORITHMS Data-intensive applications required to process multiple data sets. This implies the need to perform several join operation. Its known join operation is one of the most expensive operations in terms both I / O and CPU costs [6]. Now let us see two of the join algorithms analysed in the earlier work: 2.1 Reducer-side merge join: It is the most straightforward way to join two datasets over the Hadoop framework. It can be considered as the Hadoop version of the parallel sort-merge join algorithm. The main idea is to sort the input splits on the join column, forward them to the appropriate reducer and then merge them during the reduce phase. The performance of the algorithm is dominated by two main factors. The first is the communication overhead required to shuffle the datasets through the network from mapper to reducer. The second one is the time required to sort and write the datasets to disk before forwarding them to the reducers. However, the drawback of the the Reduce-side merge join is that the map function does not apply any filter and the output size remains at the same size with the input and also the reducer loads in memory all the tuples of each split. Figure 1.2 Reducer-side merge join [4] 2.2 Map-side replication-join The Map-Side Replication join tries to address the drawbacks of the previous approach. The concept was initially conceived in the database literature [2]. The implementation is much simpler compared to the previous algorithm. We start by replicating the small table to all nodes by using the distributed cache facility. Then, during the setup2 of the mapper we load the table into a hash table. For each value of the hash table we nest an array list for storing multiple rows with the same join attribute. Hence, for each row of the bigger table we search over only the unique keys of the small table. In the case we have many rows per join attribute it results in substantial performance gain. The hash table provides constant time search for a key value. During the execution of the mapper for each key-value pair of the input split we extract the join attribute and probe the hash table. If the value exists we combine the tuples of the matching keys and submit the new tuple. The algorithm is i llustrated in figure 1.3. The main disadvantage of this algorithm is that it is restricted by the memory size of the nodes. If the small table does not fit in memory we cannot use the algorithm at all. Figure 2.2 Map-side replication-join. III CONCLUSION IV REFERENCES [1] Fariha Atta. Implementation and analysis of join algorithms to handle skew for the hadoop mapreduce framework. Master’s thesis, MSc Informatics, School of Informatics, University of Edinburgh, 2010. [2] Shivnath Babu. Towards automatic optimization of mapreduce programs. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC ’10, pages 137–142, New York, NY, USA, 2010. ACM. [3] Spyros Blanas, Jignesh M. Patel, Vuk Ercegovac, Jun Rao, Eugene J. Shekita, and Yuanyuan Tian. A comparison of join algorithms for log processing in mapreduce. In Proceedings of the 2010 international conference on Management of data, SIGMOD ’10, pages 975–986, New York, NY, USA, 2010. ACM. [4] A Chatzistergiou. Designing a parallel query engine over map/reduce. Master’s thesis, MSc Informatics, School of Informatics, University of Edinburgh, 2010. [5] Jeffrey Dean and Sanjay Ghemawat. Mapreduce: a flexible data processing tool. Commun. ACM, 53:72–77, January 2010. [6] A. Pigul. Comparative Study Parallel Join Algorithms for MapReduce environment. Saint Petersburg State University. [7] S. Blanas, J. M. Patel, V. Ercegovac, J. Rao, E. J. Shekita, and Y. Tian. A comparison of join algorithms for log processing in mapreduce. In SIGMOD ’10: Proceedings of the 2010 international conference on Management of data, pages 975–986, New York, NY, USA, 2010. ACM. [8] Shivnath Babu. Towards automatic optimization of MapReduce programs. In SIGMOD ’10: Proceedings of the 2010 international conference on Management of data. Pages 137-142. New York, NY, USA, 2010. ACM.

My Favorite Show Essay

I have many favorite shows; one of my favorite would have to be Prison Break. There are so many reasons why it’s my favorite. My main reason is the plot of the story/show. Two brothers who broke out of jail with a group of other prisoners. Another reason is the suspense of the show. It’s literally impossible to guess what’s going to happen next. My last reason is probably just me but the little messages the show gives here and there. Messages in the sense of what it shows for what a real friendship should look like. The plot of the show is two brothers of which the young one, Michael Scofield, gets himself thrown in jail to break his older brother, Lincoln Burrows, out who is sentenced to death for a crime he didn’t commit. With just that happening in the show it’s amazing. People escaping you can see happening, but getting thrown in jail to break someone else out is interesting. Also what got my attention is how Michael planned the escape. Besides being a genius, he tattooed the prisons blueprints and needed information on how to stay out on his body. There was never a boring episode, since day one in prison the escape started he just needed to get the people necessary to escape from the prison like: the guy with the money, the guy with the transportation, and finally he needed his cell mate to agree. Finally, the relationship between both of the brothers. It isn’t your typical love story between brothers. They weren’t in each other’s life after they turned 18. And for Michael to break Lincoln out even after they lost touch of each other and Michael never showed any understanding of why Lincoln is a bum. It’s interesting that he still risked his entire career and life for this escape. Another reason is the suspense of the show, and how true it is to its category. You can never guess what’s going to happen next, for example, the first season starts off right, everything is going as planned and then the first attempt to escape isn’t what everyone expects when the escape fails. After that episode, I was completely hooked on it. I also liked that Michael kept part of the plans to himself; he never revealed his complete plan to anyone, keeping the audience in complete suspense as in how is he going to pull it off. Also, it’s very emotional not in your typical love story between both characters but in every relationship within the characters. I think it’s very popular with other shows that it’s just two characters and forget about the others. The show was very true to its point/plot. Finally, my personal reason to loving the show is the little messages/quotes they sent throughout the series. The friendship between Michael and his cellmate, Sucre. It was as if they were the brothers, very close, always had each other back. They had their differences but sometimes people have to realize that there’re more important things when time is limited. The relationship with the brothers was very different and admirable. I appreciated that a lot and I can honestly say it made the relationship with my brother and me a lot closer and stronger. Michael and Lincoln aren’t really brother’s even though they find out at the end Michael never stopped his plan and that was inspiring because it showed that nothing was going to separate them nothing was going to stop them from being free. My favorite quote from the show is you can’t go back and have new beginnings, but you can start now and have a happy ending. I find so much meaning into this quote. You obviously cant start a new beginning, but you can always change the end if it’s going terrible. When Michael told Lincoln this I was convinced that Michael truly loved Lincoln even though they were really brothers. I understood even though we weren’t born brothers, we’re dying brothers. In conclusion, I became obsessed with Prison Break. The show was so creative and amazingly written. The cast in my opinion did an amazing job they were so believable. The suspemse of the show was good and was smart; they never revealed the entire plan. Then there’s the messages the show throws out. You have to stay true to your own. Family is family at the end.

Conceptions of crime Essay

Conceptions of crime are distorted by exposure to media coverage. Discuss. This cours from coursework. inf coga ga† . â€Å"r se† . ga . â€Å"ga† . â€Å"w or†. ga . † † . ga . â€Å"k inga foga † . ga . â€Å"! The media constitutes a major part of modern society and plays a large role in our lives. It is the prime source of information and entertainment, and moreover acts as a strong influence on public belief and opinion. The media is highly influential in coae ae† . â€Å"r se† . ae . â€Å"ae† . â€Å"w or†. ae . † † . ae . â€Å"k inae foae † . ae . â€Å". shaping our views upon the world in which we live. When it comes to crime, coae ae† .†r se† . ae . â€Å"ae† . â€Å"w or†. ae . † † . ae . â€Å"k inae foae † . ae . â€Å". however, do the reports shown by the media present an accurate picture of the true This cours from coursework. inf events? The more horrific the report, the more likely it is to capture the attention of the public, which obviously secures plenty of profit for media barons. Is our increasing reliance on an institution that relies on sensationalism to create profit unwise, as this could well mean that the facts they report lack truth? The media reinforces social myths by selective reporting, and thus presents a distorted view ofreality. This could be leading to an increasingly misconstrued public view of certain crimes. The media is a very powerful influence on the way in which we behave and how we view the world, therefore it is important that it shows a realistic picture of what is taking place in our society, otherwise we will gain a false perception of aspects such as crime. Is the media really portraying crime as it truly is, or is it selectively reporting the most colourful events in order to secure audiences and therefore profit? It is important to understand that there are many crimes that are completely ignored by the media, as they are not seen as ? newsworthy?. Usually only those that will capture public attention are mentioned. This means that the crimes that fill our screens and pages are usually violent or shocking in nature. From a functionalist perspective this can serve a positive function, as the deviant behaviour makes ordinary people feel safe in the knowledge that they are morally better than the criminal, and reporting of crimes and subsequent punishments can also serve as a deterrent to others and a good method of reinforcing the social order. However, there is also a certain degree of bias present in all reporting that stems from the integrated norms and moral background of the journalist involved. It is almost impossible to be completely objective when reporting, and most journalists will slant their stories to appeal to the widest possible audience. This is theorized in the hegemonic model, and works on the basis that the majority of journalists are likely to be white, male and middle-class and will therefore bring with them to their stories their own similar set of norms and values, be this unconsciously or otherwise. The editors will also carry out a selection process on the stories before they hit the public view; therefore it is highly unlikely that any crime report reaches the news in an unbiased state.? Journalists also tend to use a set of stereotypes, which serve to easily identify ? types? of people and reduce complexities of character to a series of ? bad guys? and ? good guys?. These can lead the public to make quick judgments, which may not be based on true fact.? The media? s presentation of stereotypical images can change the public opinion on certain groups in society, and may create a false picture of them that may be harmful. The groups most often portrayed as criminal are the less privileged and minority groups such as those from ethnic minorities. Because our lives are so deeply influenced by the media we begin to see the portrayal of these groups as a reality, and this may cause crimes committed by other groups or individuals (such as corporate crime in the middle class) to be overlooked or seen as less serious.? In addition, the police can make use of the media to help maintain social order by choosing what to tell reporters, as many news stories use police accounts as a major source for their information. A good example of the media distorting aspects of crime is shown with the reporting of women involved in criminal activity. The common media portrayal of women is more passive and much less inclined to commit crime, and when they do it is seen as less violent or shocking in nature, or as influenced by a stronger male personality, for example Myra Hindley was commonly said to have been under the influence of Ian Brady when the horrific Moors murders were committed. Media coverage usually concentrates on the most violent and shocking aspects of crime. This can often be far from the truth, and puts the public in fear when in reality there is a tiny chance of them falling victim. Listening to the news, it appears that the country is riddled with serial killers and rapists however official crime statistics would show that this is not the case. Reporting like this can lead to mass panic and fear across the country, and this aspect was investigated further by Stanley Cohen in his book ? Folk Devils and Moral Panics? where he studied the 1960s mods and rockers conflicts. The media? s handling of the fights between the rivals groups made it sound as if Brighton was full of riots and was a very dangerous place, when in reality this was a false picture of events. However, these stories grabbed the headlines, and as a result appealed to a large audience and gained lots of attention with the public. At the moment there is lots of media coverage about murders in Cornwall, and about the girl who was kidnapped and murdered on Dartmoor. The way these are reported makes people fearful that there is a serial killer on the loose, when in reality there is probably little chance of this happening to them and they should be more concerned with the more common but less ? colourful? crimes that threaten society, most of which would be much more obvious if the public were able to look at official crime statistics. This viewpoint fits in well with the idea of the media as a manipulative entity that is ideologically biased and therefore bases its selection of news on this fundamental bias. This manipulation theory is Marxist in origin and works on the idea that the public are passive and absorb the information that the media presents them with without question, and that the media consist of the ruling class who use media channels to perpetuate their views and opinions in society. Official statistics paint an interesting picture of the true nature of crime levels in society. It is a fact that crime rates have actually remained stable over the last year, after experiencing a period of decline. This challenges the constant media message that crime is everywhere and on the increase. When people were interviewed about crime rates many of them believed that crime was well on the increase, as result of a ? heightened degree of press coverage which particularly focused on a claimed rise in crime.? (Jon Simmons in the Home Office Report 2002).? It is a well-known fact that the media tend to emphasise aspects and types of crime that are violent or shocking, such as murder and rape, which affects the general population by creating huge worry about occurrences of violent crime when it is a statistical fact that one is over four times more likely to fall victim to burglary. Another form of media that can, and often does, present distorted representations of crime are television shows such as The Bill, Merseybeat or reality shows such as Crimewatch. For example Crimewatch focuses on catching criminals who have carried out violent crime and places little or no emphasis on occurrences of burglary or more ? normal? crimes, even though this type of crime is much more frequent. The Bill shows many incidents of violent crime, however also tends to show other less ? newsworthy? forms of crime such as theft which is a more realistic approach. coac ac† . â€Å"r se† . ac . â€Å"ac† . â€Å"w or†. ac . † † . ac .†k inac foac † . ac . â€Å". There are a number of media influence models that attempt to explain the extent and nature of media influence. The manipulation theory presented earlier is one of these theories; however there are others that attempt to explain the media? s effect on society in different ways. The pluralist model argues that the public are not passive receptors but active participants with the ability to choose and form their own opinions from the vast variety of information presented by the media, whether about crime or about other social issues. The media only serve to reinforce the existing values that people possess. Tversky and Kahneman suggest that the easier it is to recall or imagine an event, the more likely we are to see it as risky or as frequent in occurrence (Tversky and Kahneman cit. 1994: 303). Therefore if this is correct then constant media coverage of violent crimes could lead us to believe that such crimes are the ones most likely to affect us. Mass media reporting can also serve as a substitute for direct experience, however research into this supports the pluralist model by showing that people are usually active consumers and interpreters rather than passive recipients. The issue at hand seems to be the growing difference between the way the public perceive crime and the reality as shown by official statistics. For some reason, society has a far greater fear of murder and other violent crime than is justified by the actual number of occurrence, when in reality there is far greater likelihood of incidences of burglary or car crime. So where do these fears stem from? The answer appears to be that they come from the way in which the media report crimes and how they select the most newsworthy aspects so that more papers will get sold and so viewing figures will increase. In conclusion it is an unavoidable fact that our society is so immersed in the media that its influence is hard to get away from. To a large extent, media does form our social world, however it is also true that people are active interpreters of information, and are able to utilize their own opinions on events and stories shown by the media. We are only taking in what the media offer to us not the real truth about what is happening. It is very easy for our conceptions of crime to be distorted by the way that the media portray it, as the only information we are receiving on the events is that which comes from the media. However if people really are actively interpreting and thinking about the true nature of crime levels, while attempting to broaden their horizons of information, then they will be more aware of the reality of the situation and our conceptions will be far less influenced by media constructed images and opinions. Bibliography Maguire, M, R. Reiner and R. Morgan. 2002 The Oxford Handbook Of Criminology New York: Oxford University Press Haralambos, M. 2000? Themes and Perspectives in Sociology? 4th edition Cohen, S and Young, J. 1973? The Manufacture Of News: Deviance, Social Problems and the Mass Media, London: Constable and Co Ltd.

My Beloved Daughter By Susan B. Anthony - 1018 Words

My beloved daughter, I wrote this for you when I was two months away from seventeen years old, and I hope this letter reaches you when you’re about the same. I want to first talk to you about women, and why I was incredibly lucky to grow up a woman in the age that I did and why I hope you are even luckier. The women who came before me fought for their mothers, their sisters, their daughters, and their nieces so that they could have the life they deserved. Women like Susan B. Anthony, Dorothy Height, Jane Addams, Alice Stone Blackwell, and so many others had a unified dream: equality for women. I am so grateful to live in a time they could only dream of. Some of the opportunities I have been given, they wouldn’t even be able to imagine.†¦show more content†¦You’re a prize, all girls are. And I hope you never forget that. Keeping on the topic of women, I want you to remember something: girls aren’t the enemy. A girl isn’t a roadblock in the way to dating someone, or a something to be jealous of. Girls are an incredibly powerful force, in my opinion. But this also means that girls can tear each other down, pretty harshly. In my experience, girls, for some reason, can be really nasty to each other. We know just how to get to each other, how to push each other’s buttons, and we know right where something would hurt the most. For this reason, I have somehow been a part of more girl fights than I can think of. I was either a neutral party in the fight or I was directly involved. I’m not sure why this is, why girls insist or tearing each other down. Girls are really quick to get on the defensive if something happens, but I just wonder what would happen if we channeled that. If girls everywhere fought injustice instead of each other, would we become a force to be reckoned with? G irls aren’t the enemy; they should be working with you against a common enemy. Stick up for your fellow girl, and they’ll stick up for you. There is a trend I keep seeing among my friends: they change themselves to fit someone else’s ideas. They change something just so that they can fit in or just so they can stay in a relationship. I never want you to do that, I never want you to change who you are for someone else. If you’re