cloud computing other mapreduce issues keke chen

22
Cloud Computing Other Mapreduce issues Keke Chen

Upload: gertrude-harrington

Post on 23-Dec-2015

218 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Cloud Computing Other Mapreduce issues Keke Chen

Cloud ComputingOther Mapreduce issues

Keke Chen

Page 2: Cloud Computing Other Mapreduce issues Keke Chen

Outline MapReduce performance tuning Comparing mapreduce with DBMS

Negative arguments from the DB community Experimental study (DB vs. mapreduce)

Page 3: Cloud Computing Other Mapreduce issues Keke Chen

Web interfaces Tracking mapreduce jobs

http://localhost:50030/ http://localhost:50060/ - web UI for task

tracker(s) http://localhost:50070/ - web UI for HDFS

name node(s)

lynx http://localhost:50030/

Page 4: Cloud Computing Other Mapreduce issues Keke Chen

Configuration files Directory /usr/local/hadoop/conf/

hdfs-site.xml mapred-site.xml core-site.xml

Some parameter tuning http://www.youtube.com/watch?v=hIXmpS-

jpoA

Some tips http://www.cloudera.com/blog/2009/12/7-

tips-for-improving-mapreduce-performance/

Page 5: Cloud Computing Other Mapreduce issues Keke Chen

7 tips

1. Configure your cluster correctly2. Use LZO compression3. Tune the number of maps/reduces4. Combiner5. Appropriate and compact Writables6. Reuse Writables7. Profiling tasks

Page 6: Cloud Computing Other Mapreduce issues Keke Chen

Debates on mapreduce Most opinions are from the database

camp Database techniques for distributed data

store Parallel databases

traditional database approaches Scheme SQL

Page 7: Cloud Computing Other Mapreduce issues Keke Chen

DB guys’ initial response to MapReduce “Mapreduce: a giant step backword”

A sub-optimal implementation, in that it uses brute force instead of indexing

Not novel at all -- it represents a specific implementation of well known techniques developed nearly 25 years ago

Missing most of the features that are routinely included in current DBMS

Incompatible with all of the tools DBMS users have come to depend on

Page 8: Cloud Computing Other Mapreduce issues Keke Chen

A giant step backward in the programming paradigm for large-scale data intensive applications Schema Separating schema from code High-level language

Responses MR handles large data having no schema Takes time to clean large data and pump into a DB There are high-level languages developed: pig, hive, etc Some problems that SQL cannot handle Unix style programming (pipelined processing) is used by

many users

Page 9: Cloud Computing Other Mapreduce issues Keke Chen

A sub-optimal implementation, in that it uses brute force instead of indexing

No index Data skew : some reducers take longer time High cost in reduce stage: disk I/O

Responses Google’s experience has shown it can scale well Index is not possible if the data has no schema Mapreduce is used to generate web index Writing back to disk increases fault tolerance

Page 10: Cloud Computing Other Mapreduce issues Keke Chen

Not novel at all -- it represents a specific implementation of well known techniques developed nearly 25 years ago Hash-based join Teradata

Responses The authors do not claim it is novel. many users are already using similar ideas in

their own distributed solutions Mapreduce serves as a well developed library

Page 11: Cloud Computing Other Mapreduce issues Keke Chen

Missing most of the features that are routinely included in current DBMS Bulk loader, transactions, views, integrity

constraints …

Responses Mapreduce is not a DBMS, designed for different

purposes In practice, it does not prevent engineers

implementing solutions quickly Engineers usually take more time to learn DBMS DBMS does not scale to the level of mapreduce

applications

Page 12: Cloud Computing Other Mapreduce issues Keke Chen

Incompatible with all of the tools DBMS users have come to depend on

Responses Again, it is not DBMS DBMS systems and tools have become

obstacles to data analytics

Page 13: Cloud Computing Other Mapreduce issues Keke Chen

Some important problems Experimental study on scalability High-level language

Page 14: Cloud Computing Other Mapreduce issues Keke Chen

Experimental study Sigmod09 “A comparison of approaches

to large scal data analysis” Compare parallel SQL DBMS (anonymous

DBMS-X and Vertica) and mapreduce (hadoop)

Tasks Grep Typical DB tasks: selection, aggregation,

join, UDF

Page 15: Cloud Computing Other Mapreduce issues Keke Chen

Grep task 2 settings: 535M/node, 1TB/cluster

Hadoop is much faster in loading data

Page 16: Cloud Computing Other Mapreduce issues Keke Chen

Grep: task execution

Hadoop is the slowest…

Page 17: Cloud Computing Other Mapreduce issues Keke Chen

Selection task Select pageURL, pageRank from

rankings where pageRank > x

Page 18: Cloud Computing Other Mapreduce issues Keke Chen

Aggregation task Select sourceIP, sum(adRevenue) from

Uservisits group by source IP

Page 19: Cloud Computing Other Mapreduce issues Keke Chen

Join task Mapreduce takes 3 phases to do it

Page 20: Cloud Computing Other Mapreduce issues Keke Chen

UDF task Extract URLs from documents, then

count

Vertica does not support UDF functions; uses a special program

Page 21: Cloud Computing Other Mapreduce issues Keke Chen

Discussion System level

Easy to install/configure MR, more challenging to install parallel DBMSs

Available tools for performance tuning for parallel DBMS

MR takes more time in task start-up Compression does not help MR MR has short loading time, while DBMSs

take some time to reorg the input data MR has better fault tolerance

Page 22: Cloud Computing Other Mapreduce issues Keke Chen

Discussion User level

Easier to program MR Cost to maintain MR code Better tool support for SQL DBMS