learning hbasedocshare04.docshare.tips/files/24721/247212961.pdf · 2019. 9. 16. · chapter...

9781783985944_Learning_HBase_Sample_ChapterUploaded by Packt Publishing

Chapter No. 10 HBase Use Cases Learn the fundamentalsof HBase administration and development with the helpof real-time scenarios Full description

Save Embed Print

RELATED TITLES29 views 0 0

Share

213225820-Cloudera-

HadoopBeginner's Guide

Expert HadoopAdministration

Learning HBase

Learning HBaseLearning HBase

Shashwat ShriparvShashwat Shriparv

Chapter No. 10Chapter No. 10

"HBase Use Cases""HBase Use Cases"



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

In this package, you will find:In this package, you will find:The author The author ’s’s biography biography

A preview chapter from the book, Chapter no.10 "HBase Use Cases"A preview chapter from the book, Chapter no.10 "HBase Use Cases"

A synopsis of the A synopsis of the book’s contentbook’s content

Information on where to buy this bookInformation on where to buy this book

About the Author About the AuthorShashwat ShriparvShashwat Shriparv was born in Muzaffarpur, Bihar. He did his schooling fromwas born in Muzaffarpur, Bihar. He did his schooling fromMuzaffarpur and Shillong, Meghalaya. He received his BCA degree from IGNOU, DelhiMuzaffarpur and Shillong, Meghalaya. He received his BCA degree from IGNOU, Delhiand his MCA degree from Cochin University of Science and Technology, Kerala (C-and his MCA degree from Cochin University of Science and Technology, Kerala (C-DAC Trivandrum).DAC Trivandrum).

He was introduced to Big Data technologies in early 2010 when he was asked to performHe was introduced to Big Data technologies in early 2010 when he was asked to performa proof of concept (POC) on Big Data technologies in storing and processing logs. Hea proof of concept (POC) on Big Data technologies in storing and processing logs. Hewas also given another project, where he was required to store huge binary files withwas also given another project, where he was required to store huge binary files withvariable headers and process them. At this time, he started configuring, setting up, andvariable headers and process them. At this time, he started configuring, setting up, andtesting Hadoop HBase clusters and writing sample code for them. After performing atesting Hadoop HBase clusters and writing sample code for them. After performing asuccessful POC, he initiated serious development using Java REST and SOAP websuccessful POC, he initiated serious development using Java REST and SOAP webservices, building a system to store and process logs to Hadoop using web services, andservices, building a system to store and process logs to Hadoop using web services, andthen storing these logs in HBase using homemade schema and reading data using HBasethen storing these logs in HBase using homemade schema and reading data using HBaseAPIs and HBase-Hive mapped queries. Shashwat successfully implemented the project,APIs and HBase-Hive mapped queries. Shashwat successfully implemented the project,and then moved on to work on huge binary files of size 1 to 3 TB, processing the headerand then moved on to work on huge binary files of size 1 to 3 TB, processing the headerand storing metadata to HBase and files on HDFS.and storing metadata to HBase and files on HDFS.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

Shashwat started his career as a software developer at C-DAC Cyber Forensics,Shashwat started his career as a software developer at C-DAC Cyber Forensics,Trivandrum, building mobile-related software for forensics analysis. Then, he moved toTrivandrum, building mobile-related software for forensics analysis. Then, he moved toGenilok Computer Solutions, where he worked on cluster computing, HPC technologies,Genilok Computer Solutions, where he worked on cluster computing, HPC technologies,and web technologies. After this, he moved to Bangalore from Trivandrum and joinedand web technologies. After this, he moved to Bangalore from Trivandrum and joinedPointCross, where he started working with Big Data technologies, developing softwarePointCross, where he started working with Big Data technologies, developing softwareusing Java, web services, and platform as Big Data. He worked on many projectsusing Java, web services, and platform as Big Data. He worked on many projectsrevolving around Big Data technologies, such as Hadoop, HBase, Hive, Pig, Sqoop,revolving around Big Data technologies, such as Hadoop, HBase, Hive, Pig, Sqoop,Flume, and so on at PointCross. From here, he moved to HCL Infosystems Ltd. to workFlume, and so on at PointCross. From here, he moved to HCL Infosystems Ltd. to workon the UIDAI project, which is one of the most prestigious projects in India, providing aon the UIDAI project, which is one of the most prestigious projects in India, providing aunique identification number to every resident of India. Here, he worked on technologiesunique identification number to every resident of India. Here, he worked on technologiessuch as HBase, Hive, Hadoop, Pig, and Linux, scripting, managing HBase Hadoopsuch as HBase, Hive, Hadoop, Pig, and Linux, scripting, managing HBase Hadoopclusters, writing scripts, automating tasks and processes, and building dashboards forclusters, writing scripts, automating tasks and processes, and building dashboards formonitoring clusters.monitoring clusters.

Currently, he is working with Cognilytics, Inc. on Big Data technologies, HANA, andCurrently, he is working with Cognilytics, Inc. on Big Data technologies, HANA, andother high-performance technologies.other high-performance technologies.

You can find out You can find out more about him at https://github.commore about him at https://github.com/shriparv and/shriparv andhttp://helpmetocode.blogspot.com. You can connect with him on LinkedIn athttp://helpmetocode.blogspot.com. You can connect with him on LinkedIn athttp://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9. You can also e-mail him athttp://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9. You can also e-mail him at

..

Shashwat has worked as a reviewer on the bookShashwat has worked as a reviewer on the book Pig Design Pig Design PatternPattern,, Pradeep Pradeep Pasupuleti Pasupuleti,, Packt Publ Packt Publishing ishing . He also contributed to his college magazine,. He also contributed to his college magazine, InfinityTec InfinityTechh, as an editor., as an editor.

AcknowledgmentsAcknowledgmentsFirst, I would like to First, I would like to thank a few thank a few people from Packt Publishingpeople from Packt Publishing: Kevin for : Kevin for encouragencouragingingme to write this book, Prachi for assisting and guiding me throughout the writing process,me to write this book, Prachi for assisting and guiding me throughout the writing process,Pankaj for helping me out in technical editing, and all other contributors to this book.Pankaj for helping me out in technical editing, and all other contributors to this book.

I would like to thank all the developers, contributors, and forums of Hadoop, HBase, andI would like to thank all the developers, contributors, and forums of Hadoop, HBase, andBig Data technologies for giving the industry such awesome technologies andBig Data technologies for giving the industry such awesome technologies andcontributing to it continuously. Thanks to Lars and Noll for their contribution towardscontributing to it continuously. Thanks to Lars and Noll for their contribution towardsHBase and Hadoop, respectively.HBase and Hadoop, respectively.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

I would like to thank some people who helped me to learn from life, including teachersI would like to thank some people who helped me to learn from life, including teachersat my collegeat my college — — Roshani ma'am (Principal), Namboothari sir, Santosh sir, ManjushRoshani ma'am (Principal), Namboothari sir, Santosh sir, Manjushma'am, Hudlin Leo ma'am, and my seniors Jitesh sir, Nilanchal sir, Vaidhath sir, Jwalama'am, Hudlin Leo ma'am, and my seniors Jitesh sir, Nilanchal sir, Vaidhath sir, Jwalasir, Ashutosh sir, Anzar sir, Kishor sir, and all my friends in Batch 6. I dedicate this booksir, Ashutosh sir, Anzar sir, Kishor sir, and all my friends in Batch 6. I dedicate this bookto my friend, Nikhil, who is not in this world now. Special thanks to Ratnakar Mishrato my friend, Nikhil, who is not in this world now. Special thanks to Ratnakar Mishraand Chandan Jha for always being with me and believing in me. Thanks also go out toand Chandan Jha for always being with me and believing in me. Thanks also go out toVineet, Shashi bhai, Shailesh, Rajeev, Pintu, Darshna, Priya, Amit, Manzar, Sunil, AshokVineet, Shashi bhai, Shailesh, Rajeev, Pintu, Darshna, Priya, Amit, Manzar, Sunil, Ashok

bhai, Pra bhai, Pradeep, Arshdeep, Arshad, Sujiad, Sujith, Vinayth, Vinay, Rachana, Rachana, Ashwathi, , Ashwathi, Rinku, PRinku, Pheona, Lizheona, Lizbeth, Arun,beth, Arun,Kalesh, Chitra, Fatima, Rajesh, Jasmin, and all my friends from C-DAC TrivendrumKalesh, Chitra, Fatima, Rajesh, Jasmin, and all my friends from C-DAC Trivendrumcollege. I thank all my juniors, seniors, and friends in college. Thanks to all mycollege. I thank all my juniors, seniors, and friends in college. Thanks to all mycolleagues at C-DAC Cyber Forensic: Sateesh sir, my project manager; Anwer Reyaz. J,colleagues at C-DAC Cyber Forensic: Sateesh sir, my project manager; Anwer Reyaz. J,an enthusiast who is always encouraging; Bibin bhai sahab; Ramani sir; Potty sir;an enthusiast who is always encouraging; Bibin bhai sahab; Ramani sir; Potty sir;Bhadran sir; Thomas sir; Satish sir; Nabeel sir; Balan sir; Abhin Bhadran sir; Thomas sir; Satish sir; Nabeel sir; Balan sir; Abhin sir; and others. I sir; and others. I wouldwouldalso like to thank Mani sir; Raja sir; my friends and teammates: Maruthi Kiran, Chethan,also like to thank Mani sir; Raja sir; my friends and teammates: Maruthi Kiran, Chethan,Alok, Tariq, Sujatha, Bhagya, and Mukesh; Sri Gopal sir, my team leader; and all myAlok, Tariq, Sujatha, Bhagya, and Mukesh; Sri Gopal sir, my team leader; and all myother colleagueother colleagues from PointCross. I s from PointCross. I thank Ramesh Krishnan sir, Manoj sir, Vinod sir,thank Ramesh Krishnan sir, Manoj sir, Vinod sir,

Nand Kisho Nand Kishor sir, and mr sir, and my teammy teammates Varun bhaates Varun bhai sahab, i sahab, Preeti GupPreeti Gupta, Kuldeta, Kuldeep bhai ep bhai sahab,sahab,and all my colleagues at HCL Infosystems Ltd. and UIDAI. I would also like to thankand all my colleagues at HCL Infosystems Ltd. and UIDAI. I would also like to thankSatish sir; Sudipta sir; my manager, Atul sir; Pradeep; Nikhil; Mohit; Brijesh; Kranth;Satish sir; Sudipta sir; my manager, Atul sir; Pradeep; Nikhil; Mohit; Brijesh; Kranth;Ashish Chopara; Sudhir; and all my colleagues at Cognilytics, Inc.Ashish Chopara; Sudhir; and all my colleagues at Cognilytics, Inc.

Last but not the least, I would like to thank papa, Dr. Rameshwar Dwivedi; mummy, Smt.Last but not the least, I would like to thank papa, Dr. Rameshwar Dwivedi; mummy, Smt.Rewa Dwivedi; bhai, Vishwas Priambud; sister-in-law, Ragini Dwivedi; sweet sister,Rewa Dwivedi; bhai, Vishwas Priambud; sister-in-law, Ragini Dwivedi; sweet sister,Bhumika; brother-in-law, Chandramauli Dwivedi; and new members of my family, VasuBhumika; brother-in-law, Chandramauli Dwivedi; and new members of my family, Vasuand Atmana.and Atmana.

If I missed any names, it does not mean that I am not thankful to them, they all are in myIf I missed any names, it does not mean that I am not thankful to them, they all are in myheart and I am thankful to everyone who has come in my life and left their mark. Also,heart and I am thankful to everyone who has come in my life and left their mark. Also,thanking is not in any order.thanking is not in any order.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

Learning HBaseLearning HBaseThis book will provide a This book will provide a top-down approach to learning HBase, which will be useful fortop-down approach to learning HBase, which will be useful for

both novic both novices and exes and experts. You perts. You will start will start learning clearning configuration, onfiguration, code to mcode to maintenancaintenance, ande, andtroubleshootingtroubleshooting — — a kind of all-in-one HBase knowledge bank. This will be a step-by-stepa kind of all-in-one HBase knowledge bank. This will be a step-by-stepguide, which will help you work on HBase. The book will include day-to-day activitiesguide, which will help you work on HBase. The book will include day-to-day activitiesusing HBase administration, and the implementation of Hadoop, plus HBase cluster setupusing HBase administration, and the implementation of Hadoop, plus HBase cluster setupfrom ground approach. The book will cover a complete list of use cases and explanationsfrom ground approach. The book will cover a complete list of use cases and explanationsto implement HBase as an effective Big Data tool. It will also help you understand theto implement HBase as an effective Big Data tool. It will also help you understand thelayout and structure of HBase. There are lots of books available on the market on HBase,layout and structure of HBase. There are lots of books available on the market on HBase,

but they but they lack somlack something in tething in them; somhem; some of them e of them focuses mfocuses more on configuore on configuration and ration and somesomeon coding, but this book on coding, but this book will provide a kind of will provide a kind of start-to-end approacstart-to-end approach, which will beh, which will beuseful for a person useful for a person with zero knowledge in HBase to the person with zero knowledge in HBase to the person proficient in HBase. Thisproficient in HBase. This

book is a book is a complete complete guide to guide to HBase admHBase administration inistration and deveand development witlopment with real-timh real-timeescenarios and an operation guide.scenarios and an operation guide.

This book will provide an understanding of what HBase is like, where it came from, whoThis book will provide an understanding of what HBase is like, where it came from, whoall are involved, why should we consider using it, why all are involved, why should we consider using it, why people are using it, when people are using it, when to use it,to use it,and how to use it. This book will give overall information about the HBase ecosystem.and how to use it. This book will give overall information about the HBase ecosystem.It's more like an HBase-confusion-buster book, a book to read and implement in real life.It's more like an HBase-confusion-buster book, a book to read and implement in real life.The book has in-depth theory and practical examples on HBase features. This theoreticalThe book has in-depth theory and practical examples on HBase features. This theoreticaland practical approach clears doubts on Hadoop and HBase. It provides completeand practical approach clears doubts on Hadoop and HBase. It provides completeguidance on configuration/management/troubleshooting of HBase clusters and theirguidance on configuration/management/troubleshooting of HBase clusters and theiroperations. The book is targeted at administration and development aspects of HBase;operations. The book is targeted at administration and development aspects of HBase;administration with troubleshooting, setup, and development with client and server APIs.administration with troubleshooting, setup, and development with client and server APIs.This book also enables you to design schema, code in Java, and write shell scripts toThis book also enables you to design schema, code in Java, and write shell scripts towork with HBase.work with HBase.

What This Book CoversWhat This Book CoversChapter 1Chapter 1,, Understanding the HBase EcosystemUnderstanding the HBase Ecosystem, introduces HBase in detail, and, introduces HBase in detail, anddiscusses its features, its evolution, and its architecture. We will compare HBase withdiscusses its features, its evolution, and its architecture. We will compare HBase withtraditional databases and look at add-on features and the various underlying components,traditional databases and look at add-on features and the various underlying components,and its uses in and its uses in the industry.the industry.

Chapter 2Chapter 2,, Let's Be Let's Begin with gin with HBaseHBase, deals with the HBase components in detail, their, deals with the HBase components in detail, theirinternal architecture, communication between different components, how it providesinternal architecture, communication between different components, how it providesscalability, as well as the HBase reading and writing cycle process, HBase housekeepingscalability, as well as the HBase reading and writing cycle process, HBase housekeepingtasks, region-related operations, the different components needed for a HBase clustertasks, region-related operations, the different components needed for a HBase clusterconfiguration, and some basic OS tuning.configuration, and some basic OS tuning.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

Chapter 3Chapter 3,, Let's Sta Let's Start Building Irt Building It t , lets us proceed ahead with building an HBase cluster., lets us proceed ahead with building an HBase cluster.In this chapter, you will find information on the various components and the places weIn this chapter, you will find information on the various components and the places wecan get it from. We will start configuring the cluster and consider all the parameters andcan get it from. We will start configuring the cluster and consider all the parameters andoptimization tweaks while building the Hadoop and HBase cluster. One section in theoptimization tweaks while building the Hadoop and HBase cluster. One section in thechapter will focus on the various component-level and OS-level parameters for anchapter will focus on the various component-level and OS-level parameters for anoptimized cluster.optimized cluster.

Chapter 4Chapter 4,, Optimizing the HBase/Hadoop Cluster Optimizing the HBase/Hadoop Cluster , teaches us to optimize the HBase, teaches us to optimize the HBasecluster according to the production environment and running cluster troubleshootingcluster according to the production environment and running cluster troubleshootingtasks. We will look at optimization on hardware, OS, software, and network parameters.tasks. We will look at optimization on hardware, OS, software, and network parameters.This chapter will also teach us how we can optimize Hadoop for a better HBase.This chapter will also teach us how we can optimize Hadoop for a better HBase.

Chapter 5Chapter 5,, The Storage, Structure Layout, and Data Model of HBaseThe Storage, Structure Layout, and Data Model of HBase , discusses HBase's, discusses HBase'sdata model and its various data model operations for fetching and writing data in HBasedata model and its various data model operations for fetching and writing data in HBasetables. We will also consider some use cases in order to design schema in HBase.tables. We will also consider some use cases in order to design schema in HBase.

Chapter 6 Chapter 6 ,, HBase Clus HBase Cluster Mainter Maintenance tenance and Troubleand Troubleshooting shooting , covers all the aspects of, covers all the aspects ofHBase cluster management, operation, and maintenance. Once a cluster is built and inHBase cluster management, operation, and maintenance. Once a cluster is built and inoperation, we need to look after it, continuously tune it up, operation, we need to look after it, continuously tune it up, and troubleshoot in order toand troubleshoot in order tohave a healthy HBase cluster. We will also study the commands available with HBasehave a healthy HBase cluster. We will also study the commands available with HBaseand Hadoop shell.and Hadoop shell.

Chapter 7 Chapter 7 ,, Scripting in HBaseScripting in HBase, explains an automation process using HBase and shell, explains an automation process using HBase and shellscripts. We will learn to write scripts as an administrator or developer to automatescripts. We will learn to write scripts as an administrator or developer to automatevarious data-model-related tasks. We will also read about various backup and restorevarious data-model-related tasks. We will also read about various backup and restoreoptions available in HBase and how to perform them.options available in HBase and how to perform them.

Chapter 8Chapter 8,, Coding HBase in JavaCoding HBase in Java, teaches Java coding in HBase. We will start with basic, teaches Java coding in HBase. We will start with basicJava coding in HBase and learn about Java APIs available for client requests. You willJava coding in HBase and learn about Java APIs available for client requests. You willalso learn to build a basic client in Java, which can be used to contact an HBase clusteralso learn to build a basic client in Java, which can be used to contact an HBase clusterfor various operations using Java for various operations using Java code.code.

Chapter 9Chapter 9,, Advance C Advance Coding in Joding in Java for HBava for HBasease, focuses more on Java coding in HBase., focuses more on Java coding in HBase.It is a more detailed learning about all the different kind of APIs, classes, methods, andIt is a more detailed learning about all the different kind of APIs, classes, methods, andinterfaces available in Java for HBase. You will also see the different kind of webinterfaces available in Java for HBase. You will also see the different kind of webservices or thrift services, which you can use to services or thrift services, which you can use to ease up the coding and using ease up the coding and using the inbuiltthe inbuiltservice and not implementing the entire architecture code in Java. This chapter has aservice and not implementing the entire architecture code in Java. This chapter has asection that includes a discussion of some special features of HBase and some opensection that includes a discussion of some special features of HBase and some opensource projects available, which can be used in coordination with HBase for a productionsource projects available, which can be used in coordination with HBase for a productioncluster and a project.cluster and a project.

Chapter 10Chapter 10,, HBase Use HBase Use CasesCases, discusses the use cases in the industry, which are being, discusses the use cases in the industry, which are beingused with HBase as their underlying technology.used with HBase as their underlying technology.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

HBase Use CasesHBase Use Cases

In this chapter, we will walk through some HBase user cases, zoom in the projectIn this chapter, we will walk through some HBase user cases, zoom in the projectlayout and design to understand how layout and design to understand how to utilize HBase for different business to utilize HBase for different business usages,usages,and provide some references to you and provide some references to you for further development.for further development.

We will see the use case and architecture used at:We will see the use case and architecture used at:

• Facebook• Facebook

• Groupon• Groupon

• Pinterest• Pinterest

• • LongTail LongTail VideoVideo

This chapter will also show how these leading entities are using HBase for theirThis chapter will also show how these leading entities are using HBase for theirproject and the architecture of the project.project and the architecture of the project.

HBase in industry todayHBase in industry todayWith the increase in demand for Big Data day by day in With the increase in demand for Big Data day by day in order to solve the problemsorder to solve the problemscomprising planet-size data, HBase is one of the comprising planet-size data, HBase is one of the players due to its players due to its characteristic ofcharacteristic ofpersistent read performance. Using these systems, one can store and analyze data inpersistent read performance. Using these systems, one can store and analyze data incoordination with various other tools.coordination with various other tools.

Also features such as HBase running Also features such as HBase running on cloud and expanding and reducing onon cloud and expanding and reducing ondemand will enable even small-scale companies to use demand will enable even small-scale companies to use its services. With theits services. With theincreasing demand of data storage space, the need to process it and representincreasing demand of data storage space, the need to process it and representunstructured data into a structured format can unstructured data into a structured format can be achieved by HBase.be achieved by HBase.

The new sub and open projects that are being created, which enables HBase to supportThe new sub and open projects that are being created, which enables HBase to support



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


The future of HBase against relationalThe future of HBase against relationaldatabasesdatabasesIt would be wrong It would be wrong to say that to say that NoSQL, a column-based database, will replace theNoSQL, a column-based database, will replace theRDBMS. HBase is still evolving and it cannot be used for all the use cases. There willRDBMS. HBase is still evolving and it cannot be used for all the use cases. There willalways be a need for different types of database to work in coordination with eachalways be a need for different types of database to work in coordination with eachother satisfying different use cases other satisfying different use cases to build a complete production environment.to build a complete production environment.

However, the new features of However, the new features of high availability, almost consistent reads, ability tohigh availability, almost consistent reads, ability tostore and retrieve petabytes store and retrieve petabytes of data, and support of various evolving open sourceof data, and support of various evolving open sourcetools totools to fifill up the existing gaps will take HBase and other NoSQL databases very farll up the existing gaps will take HBase and other NoSQL databases very farfor sure. We have just started looking at the power of these for sure. We have just started looking at the power of these systems and are yet tosystems and are yet tounderstand and implement real-life use cases with understand and implement real-life use cases with enormous amount of data.enormous amount of data.

So, for sure, Hadoop plus HBase and other So, for sure, Hadoop plus HBase and other subprojects are the future of subprojects are the future of datadatawarehousing and massive processing.warehousing and massive processing.

Some real-world project examples'Some real-world project examples'use casesuse casesIn this section, we will list out use cases of HBase being used in In this section, we will list out use cases of HBase being used in the industry today.the industry today.References and more details can be found at links provided in theReferences and more details can be found at links provided in the Useful links andUseful links andreferencesreferences section at the end of the section at the end of the chapter.chapter.

HBase at FacebookHBase at FacebookFacebook, as you all know, is Facebook, as you all know, is a social utility that connects people a social utility that connects people with friends andwith friends andothers who work, study, and live others who work, study, and live around them. Facebook uses HBase mainly toaround them. Facebook uses HBase mainly topower their messages infrastructure. The following are the power their messages infrastructure. The following are the services where Facebookservices where Facebookuses HBase:uses HBase:

• • Messages Messages between between usersusers

• Chats• Chats

• E-mails• E-mails

• SMS• SMS

Choosing HBaseChoosing HBaseThe following are the reasons The following are the reasons why Facebook chooses HBase:why Facebook chooses HBase:



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

Chapter 10Chapter 10

• • Horizontal Horizontal scalabilityscalability

• • Automatic Automatic failoverfailover

• • Strong Strong consistencyconsistency

• Bene• Benefifits of HDFS such ts of HDFS such as fault tolerant, scalable, checksums, and MapReduceas fault tolerant, scalable, checksums, and MapReduce

HBase acts like a caching layer on top of Hadoop when bundled these two together,HBase acts like a caching layer on top of Hadoop when bundled these two together,faster data process compare with other NoSQL + faster data process compare with other NoSQL + Hadoop.Hadoop.

Storing in HBaseStoring in HBaseFacebook uses HBase to store data. The following are the types of data that are storedFacebook uses HBase to store data. The following are the types of data that are storedin HBase:in HBase:

• • Small Small messagesmessages

• • Message Message metadata metadata (thread/message (thread/message indices)indices)

• • Search Search indexindex

Facebook also use Haystack (a software that runs on a single machine and stores dataFacebook also use Haystack (a software that runs on a single machine and stores datawithout replication; it only cares about local aggregated blob storage, without replication; it only cares about local aggregated blob storage, concentratingconcentratingupon reducing disk seeks, and speeding upupon reducing disk seeks, and speeding up fifile retrieval) to store attachments andle retrieval) to store attachments andlarge messages.large messages.

The architecture of a Facebook messageThe architecture of a Facebook messageThe following is a diagrammatical representation of how things The following is a diagrammatical representation of how things work:work:



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


In this processIn this process flflow, a client asks the user directory service for the user details, thenow, a client asks the user directory service for the user details, thenafter getting the detail of the after getting the detail of the user, the client sends the request to the user, the client sends the request to the applicationapplicationserver, which may be a server, which may be a Tomcat machine, running a custom application.Tomcat machine, running a custom application.

The application server using a custom service HBase The application server using a custom service HBase index is searched for theindex is searched for theuser-related cell from where a message user-related cell from where a message can be written, appended, or retrieved,can be written, appended, or retrieved,and for the attachments the Haystack is searched.and for the attachments the Haystack is searched.

Facts andFacts and fifiguresguresLet's have a look at some of the Let's have a look at some of the eye-opening facts andeye-opening facts and fifigures:gures:

• • Talking Talking about about Facebook Facebook message message statistics, statistics, number number of of messages messages sentsentand received per day is more and received per day is more than 6 billionthan 6 billion

• • Talking Talking about about traftraffific to HBase, have a look at the following points:c to HBase, have a look at the following points:

Almost 100 billion read/writes per dayAlmost 100 billion read/writes per day

At peak time, more or At peak time, more or less 1.5 million operations per secondless 1.5 million operations per second

About 55 percent reads and 45 percent writesAbout 55 percent reads and 45 percent writes

More than 2 petabytes of data and with replication, it's moreMore than 2 petabytes of data and with replication, it's morethan 6 petabytesthan 6 petabytes

Data is compressed usingData is compressed using Lempel–Ziv–OberhumerLempel–Ziv–Oberhumer ( (LZOLZO))compressioncompression

And the data is growing around 300 terabytes per monthAnd the data is growing around 300 terabytes per month

• • Schema Schema changes changes during during productionproduction

• • Heavy Heavy use use of of bulk bulk importimport

Let's now see some other use cases in the Let's now see some other use cases in the industry.industry.

HBase at PinterestHBase at PinterestPinterest is deployed onPinterest is deployed on Amazon Elastic Compute CloudAmazon Elastic Compute Cloud ( (EC2EC2). Pinterest uses a). Pinterest uses afollow model in which the user follows other users. It needs to update the feed datafollow model in which the user follows other users. It needs to update the feed datafor every user as soon as a follower makes changes in a pin or updates a pin. This isfor every user as soon as a follower makes changes in a pin or updates a pin. This isthe most the most classic social-media kind of application.classic social-media kind of application.

This happens for hundreds of millions of pins This happens for hundreds of millions of pins per month and about a per month and about a billion writesbillion writesper day. So the following are the speciper day. So the following are the specifification of implementation:cation of implementation:

• • They choosThey choose a we a wide schema ide schema where each where each user's follouser's following feed wing feed is a is a single rowsingle row



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


• • They have They have increased increased the per the per region Mregion MemStore siemStore size to ze to 512 MB 512 MB MemStoreMemStorewhich leads to 40 MB HFile instead of the smaller 8 MBwhich leads to 40 MB HFile instead of the smaller 8 MB fifile for defaultle for defaultMemStore; this leads to MemStore; this leads to less-frequent compactionsless-frequent compactions

• Maintains• Maintains mean time to recoverymean time to recovery ( (MTTRMTTR) of less than 2 minutes by reducing) of less than 2 minutes by reducingvarious timeout settings such as socket, connect, and various timeout settings such as socket, connect, and stale node.stale node.

The layout architectureThe layout architectureLet's now see the basic layout architecture. The following layout represents theLet's now see the basic layout architecture. The following layout represents thefollower and the follower and the followee relationship:followee relationship:

And according to this structure, it forms a kind of graph inside. As you can see inAnd according to this structure, it forms a kind of graph inside. As you can see inthe following diagram, it shows the path or writes in the system. Internally, there is athe following diagram, it shows the path or writes in the system. Internally, there is asingle row for each user single row for each user in HBase and all the in HBase and all the followers' and followees' information isfollowers' and followees' information isstored in it.stored in it.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


HBase at GrouponHBase at GrouponGroupon is a deal-of-the-day website that features Groupon is a deal-of-the-day website that features discounted gift certidiscounted gift certifificatescatesusable at local or national companies. Groupon was launched in November 2008.usable at local or national companies. Groupon was launched in November 2008.At Groupon, there are two requirements:At Groupon, there are two requirements:

• • Notify Notify users users about about the the deals deals via via an an e-maile-mail

• • Provide Provide good good user user experience experience on on the the websitewebsite

When they started When they started off, they used Hadoop MapReduce jobs for e-mail deal deliveryoff, they used Hadoop MapReduce jobs for e-mail deal deliveryand used MySQL for their and used MySQL for their online application. Now, they have started relevance andonline application. Now, they have started relevance anda personalization system, both on HBase. They use a very wide schema in HBase asa personalization system, both on HBase. They use a very wide schema in HBase asone column family for user history and one column family for user history and proprofifile and another for le and another for e-mail history.e-mail history.

The layout architectureThe layout architectureNow, let's see their Now, let's see their read/write and architectureread/write and architecture flflow:ow:

The data pipeline collects the data and keeps on writing to HBase, which is used forThe data pipeline collects the data and keeps on writing to HBase, which is used forofofflfline purposes such as sending e-mails from where the data is replicated to theine purposes such as sending e-mails from where the data is replicated to theHBase cluster, which is there for online relevance suggestions on the website.HBase cluster, which is there for online relevance suggestions on the website.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


The following are some of the The following are some of the optimization they performed:optimization they performed:

• • Presplit Presplit tablestables

• • Increased Increased lease lease timeouttimeout

• • Increased Increased scanner scanner timeouttimeout

• • Increased Increased region region size size to to 10 10 GBGB

• • Keep Keep less less number number of of regions regions per per region region serversservers

• • For For heavy heavy write write jobs, jobs, they they set the set the following following values:values:

hbase.hregion.memstore.block.multiplier hbase.hregion.memstore.block.multiplier to to 44

hbase.hregion.memstore.flush.size hbase.hregion.memstore.flush.size to to 134217728134217728

hbase.hstore.blockingstorefiles hbase.hstore.blockingstorefiles to to 100100

hbase.zookeeper.useMulti hbase.zookeeper.useMulti to to falsefalse (for stable replication) (for stable replication)

They performed all these changes in order to optimize and make HBase suit theirThey performed all these changes in order to optimize and make HBase suit theirrequirement and this was done requirement and this was done after evaluating the productivity and performanceafter evaluating the productivity and performanceof the cluster.of the cluster.

HBase at LongTail VideoHBase at LongTail VideoLongTail Video company provides JW Player, which is an online video LongTail Video company provides JW Player, which is an online video player usedplayer usedby nearly 2 by nearly 2 million websites. They are completely deployed on AWS and as million websites. They are completely deployed on AWS and as such usesuch useHBase and EMR from Amazon.HBase and EMR from Amazon.

Their requirements are:Their requirements are:

• • Very Very fast fast queries queries across across datasetsdatasets

• • There There should should be be a a support support for for date-range date-range queriesqueries

• • Ability Ability to to store store a a huge huge amount amount of of aggregated aggregated datadata

• • There There should should bebe flflexibility in dimensions used for rollup tablesexibility in dimensions used for rollup tables

HBaseHBase fifits very well on these requirements. They use full-ts very well on these requirements. They use full-flfledged Python shop so useedged Python shop so useHappyBase and have Thrift running on all the nodes of the HBase cluster for readingHappyBase and have Thrift running on all the nodes of the HBase cluster for readingand writing.and writing.

Some statistics about them are Some statistics about them are as follows:as follows:

• • 156 156 million million unique unique viewersviewers



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


The layout architectureThe layout architectureThis is a diagrammatical representation of the This is a diagrammatical representation of the architecture:architecture:

HBase at Aadhaar (UIDAI)HBase at Aadhaar (UIDAI)UIDAIUIDAI is a government organization of India that is a government organization of India that provides unique identiprovides unique identifificationcationto Indian nationals. At UIDAI, all to Indian nationals. At UIDAI, all the open source software such as the open source software such as Hadoop, HBase,Hadoop, HBase,and Hive are being used. HBase is used to store data about the residents.and Hive are being used. HBase is used to store data about the residents.

The layout architectureThe layout architectureLet's see the architecture of the project that Let's see the architecture of the project that is available on the Internet:is available on the Internet:



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


There are some use cases and the architectureThere are some use cases and the architecture flflow of the use cases that are beingow of the use cases that are beingused in different companies in the industry today. This trend is increasing day byused in different companies in the industry today. This trend is increasing day byday with more people coming and including this day with more people coming and including this technology and implementingtechnology and implementingit in their project requirement.it in their project requirement.

Hadoop and HBase and other Big Data Hadoop and HBase and other Big Data components today are providing a completecomponents today are providing a completeinfrastructure for a production system with data migration tools, data mapping tools,infrastructure for a production system with data migration tools, data mapping tools,and some add-on open source projects.and some add-on open source projects.

Likewise, according to our requirements, we canLikewise, according to our requirements, we can fifirst design the architecture of ourrst design the architecture of ourproject based on the project based on the components and the type of components and the type of functionalities required, and wefunctionalities required, and wecan choose from a wide variety can choose from a wide variety of Hadoop and its subproject components.of Hadoop and its subproject components.

Useful links and referencesUseful links and referencesYou can always use these links for reference.You can always use these links for reference.

The following are the use-cases-related links:The following are the use-cases-related links:

•• http://hbasecon.com/archive.htmlhttp://hbasecon.com/archive.html

•• https://www.facebook.com/UsingHbasehttps://www.facebook.com/UsingHbase

•• http://www.slideshare.net/brizzzdotcom/facebook-messages-hbasehttp://www.slideshare.net/brizzzdotcom/facebook-messages-hbase

•• http://www.slideshare.net/cloudera/case-studies-session-3bhttp://www.slideshare.net/cloudera/case-studies-session-3b

•• http://www.slideshare.net/cloudera/long-tailvideo-http://www.slideshare.net/cloudera/long-tailvideo-

hbasecon2013-24035928hbasecon2013-24035928

•• http://www.slideshare.net/cloudera/operations-session-1http://www.slideshare.net/cloudera/operations-session-1

These are some other references:These are some other references:

•• https://github.com/larsgeorge/hbase-schema-manager/tree/masterhttps://github.com/larsgeorge/hbase-schema-manager/tree/master

•• http://wiki.apache.org/hadoop/HelpContentshttp://wiki.apache.org/hadoop/HelpContents

•• http://www.jnbridge.com/jn/blog/tag/hbase/http://www.jnbridge.com/jn/blog/tag/hbase/

•• http://ianvarley.com/coding/HBaseSchema_HBaseCon2012.pdfhttp://ianvarley.com/coding/HBaseSchema_HBaseCon2012.pdf

•• http://www.slideshare.net/cloudera/5-h-base-schemahbasecon2012http://www.slideshare.net/cloudera/5-h-base-schemahbasecon2012

•• http://hbase.apache.org/book/schema.casestudies.htmlhttp://hbase.apache.org/book/schema.casestudies.html



Save Embed Print


Share

213225820-Cloudera-



Learning HBase


The following links will be useful The following links will be useful for schema designing:for schema designing:

•• http://www.slideshare.net/amansk/hbase-schema-design-big-data-http://www.slideshare.net/amansk/hbase-schema-design-big-data-

techcon-bostontechcon-boston

•• http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.

r43.cf2.rackcdn.com/9353-login1210_khurana.pdfr43.cf2.rackcdn.com/9353-login1210_khurana.pdf

•• http://www.slideshare.net/amansk/hbase-schema-design-big-data-http://www.slideshare.net/amansk/hbase-schema-design-big-data-

techcon-bostontechcon-boston

•• http://hbase.apache.org/book/schema.casestudies.htmlhttp://hbase.apache.org/book/schema.casestudies.html

•• http://www.slideshare.net/cloudera/5-h-base-schemahbasecon2012http://www.slideshare.net/cloudera/5-h-base-schemahbasecon2012

•• https://www.youtube.com/watch?v=_HLoH_PgrLkhttps://www.youtube.com/watch?v=_HLoH_PgrLk

•• http://ianvarley.com/coding/HBaseSchema_HBaseCon2012.pdfhttp://ianvarley.com/coding/HBaseSchema_HBaseCon2012.pdf

The following are the links to HBase books and references:The following are the links to HBase books and references:

•• http://hbase.apache.org/book.htmlhttp://hbase.apache.org/book.html

•• https://www.youtube.com/results?search_query=hbasehttps://www.youtube.com/results?search_query=hbase

•• https://www.youtube.com/results?search_query=hadoophttps://www.youtube.com/results?search_query=hadoop

SummarySummaryIn this chapter, we discussed the future aspect of HBase and the different use casesIn this chapter, we discussed the future aspect of HBase and the different use casesbeing implemented in the being implemented in the industry using HBase, its processindustry using HBase, its process flflow, and architecture.ow, and architecture.

Today, the need of HBase is growing rapidly and we can get the list of companiesToday, the need of HBase is growing rapidly and we can get the list of companieswhose projects are powered by HBase atwhose projects are powered by HBase at http://wiki.apache.org/hadoop/Hbase/http://wiki.apache.org/hadoop/Hbase/

PoweredByPoweredBy..

After reading this book, you should be able to move forward and design the useAfter reading this book, you should be able to move forward and design the usecases, performing administrative tasks and writing codes for HBase. Furthermore,cases, performing administrative tasks and writing codes for HBase. Furthermore,the reader can always the reader can always visit HBase Wiki, the HBase Apache website, visit HBase Wiki, the HBase Apache website, and HBaseand HBasesource site for more updated and recent source site for more updated and recent information.information.

HBase use has grown a lot but it still has HBase use has grown a lot but it still has a long way to go.a long way to go.



Save Embed Print


Share

213225820-Cloudera-



Learning HBase

Where to buy this bookWhere to buy this bookYou can buy Learning HBase from the Packt Publishing website:You can buy Learning HBase from the Packt Publishing website:

..

Free shipping to the US, UK, Europe and selected Asian countries. For more information, pleaseFree shipping to the US, UK, Europe and selected Asian countries. For more information, pleaseread our read our shipping policyshipping policy..

Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals andAlternatively, you can buy the book from Amazon, BN.com, Computer Manuals andmost internet book most internet book retailers.retailers.

learning hbasedocshare04.docshare.tips/files/24721/247212961.pdf · 2019. 9. 16. · chapter...

Documents