big data and hadoop
DESCRIPTION
An Introduction to Big data and problems associated with storing and analyzing big data and How Hadoop solves the problem with its HDFS and MapReduce frameworks. A little intro to HDInsight, Hadoop on windows azure.TRANSCRIPT
Big data and HadoopLearn how Hadoop deals problems associated with Big data Analysis
Srikanth M V
There are 30 billion pieces of content shared on Facebook every day.
Wal-Mart handles more than 1 million customer transactions an hour.
More than 5 billion people are calling, texting, tweeting and browsing websites using Smart phones.
The 3-Vs of Big Data
VolumeGiga Bytes, Tera Bytes,
Peta Bytes or Zeta Bytes….
VelocityThe rate at which data
flows into an organization
Variety
Structured and Unstructured
So What is Big Data?
• Big data is large and complex data sets collected from various sources like Sensors, Social Media, Satellite images, Audio, Video, RFID etc.
• Big data is data that exceeds the processing capacity of conventional database systems.
• How ‘Big’ is big? GB, TB, PB , ZB?? NO.. Data is Big when the organization’s ability to handle,
store and analyze exceeds its capacity.
Problem: Storing and Analyzing
“Big Data”
Solution*: Move compute to data
*One among many, but Hadoop is flexible, Simple and reliable.
Hey there, I’m Hadoop and I can do that
for you..
• Created: 2005• Creators: Doug Cutting and Mike Cafarella• Contributors: Apache, Yahoo, Google• Language: Java
How Hadoop deals with “Big data”
• Primary Components• HDFS – Hadoop Distributed File System • Map Reduce• Hadoop YARN• Job Scheduling and Resource Management
• Hadoop Common• Access to file system
HDFS• Distributed, scalable,
reliable and portable file system.
• Hadoop Cluster is a set of Data Nodes and a Name Node
• Client divides the data to process, into blocks
• Each block of data is replicated in 3 Nodes*
• More Nodes, More Efficiency.
• Robust - Relies on Software instead of hardware
HDFS Cluster
Server
Data Node Name Node
Server
Data Node
Server
Data Node
Server
Data Node
Master
Slaves
B1 B2 B3
Somefile.txt
B1
B2
B3
B2
B3
B1
B3
B1
B3
Map Reduce• Divide and Conquer• Parallel Computing• Map(): Perform Sorting &
Filtering• Reduce(): Perform
Summary Operation • Each node has Task tracker
which communicates with Job Tracker.
• The output files will be available as local files on client.
Hadoop Architecture
Hadoop Secondary Components• Ambari
• Web Tool for provisioning, managing and monitoring Clusters• Hbase
• Scalable distributed database that supports structured data for large tables• Zoo Keeper
– A High performance coordination service for distributed applications• Pig
– A High level data flow language and execution framework for parallel computation• Hive
– A Data warehouse infrastructure that provides data summarization and ad hoc querying• Cassandra
– A scalable multi master database with no single point failures• Chukwa
– A data collection system for managing large distributed systems• Lucene and Solr
– Search engines, currently not part of Hadoop
Real World Example of Big data Analytics using Hadoop
MySql Database
1
7
2 3
56
1. Users interact with Facebook using data in textual, image, video formats.2. Facebook transfers the core data to My SQL database.3. My SQL data is replicated to Hadoop clusters.4. Data is processed using Hadoop MapReduce functions5. The results are transferred back to My SQL6. Facebook uses the data to create recommendations for you based on
your interests.
4
Other users:
Why should an Enterprise move to Big Data Analytics?
• Enterprises will be able to harness relevant data and use it to make the best decisions– Increasing the redemption
rate– Determine optimum prices– Calculate risks in a minute,
and understand future possibilities to mitigate risk
– Enabling new products– Identifying patterns help
identify trends in business
The key lies in collecting quality data, not quantity.
What is in it for us?
Hadoop on Cloud
• Provision Scalable Storage for storing Big data as Blobs– PAAS• Provision Linux VMs on Cloud – IAAS• Language support for JS and C#• Business Intelligence – Connect MS Excel to Hadoop Hive• Remote Access to Hadoop Jobs via REST API, WebHCat REST API.• Easy to access Management Portal for monitoring Hadoop Jobs• .NET SDK to execute Hive Jobs on HDInsight• …..More
+
