8/13/2019 CSE590 Lectures 9 10

1/36

CSE 590: Special Topics Course

( Supercomputing )

Lectures 9 & 10

( MapReduce & Hadoop )

Rezaul A. ChowdhuryDepartment of Computer Science

SUNY Stony Brook

Spring 2012

8/13/2019 CSE590 Lectures 9 10

2/36

MapReduce

,

2004 ( & ).

A .

A .

A C2.

8/13/2019 CSE590 Lectures 9 10

3/36

MapReduce

A,

A

o

o

o

o

8/13/2019 CSE590 Lectures 9 10

4/36

Big Ideas behind MapReduce

A

.

.

C

.

.

.

A

.

8/13/2019 CSE590 Lectures 9 10

5/36

A Simplified View of MapReduce

& ,

8/13/2019 CSE590 Lectures 9 10

6/36

A Simple Word Count Example

C .

& ,

8/13/2019 CSE590 Lectures 9 10

7/36

Combiner & Partitioner

& ,

8/13/2019 CSE590 Lectures 9 10

8/36

Word Count with In-Mapper Combining

& ,

8/13/2019 CSE590 Lectures 9 10

9/36

Word Count with Improved In-Mapper Combining

& ,

8/13/2019 CSE590 Lectures 9 10

10/36

Compute Mean of Values Associated with Each Key

& ,

8/13/2019 CSE590 Lectures 9 10

11/36

Mean of Values with a Separate Combiner

& ,

8/13/2019 CSE590 Lectures 9 10

12/36

& ,

Mean of Values with a Separate Combiner

8/13/2019 CSE590 Lectures 9 10

13/36

Mean of Values with an In-Mapper Combiner

& ,

8/13/2019 CSE590 Lectures 9 10

14/36

Computing Word Co-occurrences

& ,

8/13/2019 CSE590 Lectures 9 10

15/36

Word Co-occurrences ( Stripes Approach )

& ,

8/13/2019 CSE590 Lectures 9 10

16/36

Baseline Inverted Indexing for Text Retrieval

& ,

8/13/2019 CSE590 Lectures 9 10

17/36

& ,

Baseline Inverted Indexing for Text Retrieval

8/13/2019 CSE590 Lectures 9 10

18/36

Scalable Inverted Indexing for Text Retrieval

& ,

8/13/2019 CSE590 Lectures 9 10

19/36

Parallel Breadth-First Search

& ,

8/13/2019 CSE590 Lectures 9 10

20/36

Hadoop Subprojects

,

A

/ ( , C, ).

A , C,

.

A

.

A

.

8/13/2019 CSE590 Lectures 9 10

21/36

Hadoop Subprojects

,

A

. .

A , . B

,

( ).

A , .

.

8/13/2019 CSE590 Lectures 9 10

22/36

Hadoop Subprojects

,

A .

(

)

.

A . C

,

.

8/13/2019 CSE590 Lectures 9 10

23/36

The Building Blocks of Hadoop

.

o

o

o

o

o

8/13/2019 CSE590 Lectures 9 10

24/36

The Building Blocks of Hadoop

:

, ,

.

.

C

,A

8/13/2019 CSE590 Lectures 9 10

25/36

The Building Blocks of Hadoop

, ,

. ,

, .

( )

.

C

,A

8/13/2019 CSE590 Lectures 9 10

26/36

C

,A

.

. C

.

The Building Blocks of Hadoop

8/13/2019 CSE590 Lectures 9 10

27/36

Hadoop Distributed File System ( HDFS ) Design

C

B

,

8/13/2019 CSE590 Lectures 9 10

28/36

Hadoop MapReduce

,

8/13/2019 CSE590 Lectures 9 10

29/36

An Example: Mining Weather DataFind Maximum Temperature Every Year

,

8/13/2019 CSE590 Lectures 9 10

30/36

Maximum Temperature Every Year ( Java )

,

8/13/2019 CSE590 Lectures 9 10

31/36

Maximum Temperature Every Year ( Java )

,

8/13/2019 CSE590 Lectures 9 10

32/36

Maximum Temperature Every Year ( Java )

,

8/13/2019 CSE590 Lectures 9 10

33/36

Maximum Temperature Every Year ( Java )

,

8/13/2019 CSE590 Lectures 9 10

34/36

Maximum Temperature Every Year ( Python )

,

8/13/2019 CSE590 Lectures 9 10

35/36

Maximum Temperature Every Year ( C++ )

,

M i T t E Y ( C )

8/13/2019 CSE590 Lectures 9 10

36/36

Maximum Temperature Every Year ( C++ )

,