apache spark linkedin

Download Apache spark linkedin

Post on 11-Apr-2017

480 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

  • Slide title

    70 pt

    CAPITALS

    Slide subtitle

    minimum 30 pt

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 2

    W h a t ?

    Wh y ?

    H o w ?

    D e m o

    EDR A n a l y t i c s

    AGENDA

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 3

    Spark eco-system

    Technology landscape

    Spark eco-system

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 4

    Fast and general engine for big data processing with libraries for SQL, streaming, advanced analytics(machine learning)

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 5

    WHAT?

    Originally developed in 2009 in

    UC Berkeleys AMPLab

    Fully open sourced in 2010

    now at Apache Software

    Foundation

    http://spark.apache.org

    http://spark.apache.org/http://spark.apache.org/http://spark.apache.org/http://spark.apache.org/

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 6

    Spark is the Most Active Open Source Project in Big Data

    Pro

    jec

    t c

    on

    trib

    uto

    rs in

    pa

    st

    ye

    ar

    Giraph

    Storm

    Tez

    0

    20

    40

    60

    80

    100

    120

    140

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 7

    Distributors Applications

    7

    The Spark Community

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 8

    2015 SNAPSHOT

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 9

    WHY SPARK? Speed

    Run programs up to 100x faster than Hadoop Map Reduce in memory, or 10x faster on disk.

    Ease of Use

    Supports different languages for developing applications using Spark

    Generality

    Combine SQL, streaming, and complex analytics into one platform

    Runs Everywhere Spark runs on Hadoop, Mesos, standalone, or in the cloud.

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 10

    Easy: Get Started Immediately

    Interactive Shell

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 11

    Monitoring

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 12

    FEATURE COMPARISON

    12

    Source: Daytona GraySort benchmark, sortbenchmark.org

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 13

    WORD COUNT

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 14

    Spark eco-system

    Local YARN Mesos

    Spark Streaming Spark SQL GraphX MLLib

    Spark Core Engine (Scala/Java/Python)

    Standalone cluster

    Persistence

    Cluster Manager

    14

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 15

    SPARK ON HDFS

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 16

    HADOOP SPARK

    SQL Query interface HIVE SPARKSQL

    Machine Learning APACHE MAHOUT MLIB

    Graph processing APACHE GIRAPH GRAPHX

    Streaming APACHE STORM SPARK STREAMING

    ECOSYSTEM

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 17

    HOW?

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 18

    So, HOW is It BETTER

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 19

    THE BIG QUESTION?

    Is Spark going to replace Hadoop?

    Answer Yes, Spark will be used on top of Hadoop and replace

    MapReduce Reasons:

    1. Hadoop MapReduce cannot handle real-time

    processing

    2. Hadoop MapReduce is slower than Hadoop Spark

    3. With rise of IOT, Spark is a must

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    minimum 20 pt

    Characters for Embedded font: !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~fifl

    Do not add objects or

    text in the footer area Ericsson Internal | 2015-08-11 | Page 20

    RDD & SPARK COMPONENTS

    Technology landscape

    Spark eco-system

  • Slide title

    44 pt

    Text and bullet level 1

    minimum 24 pt

    Bullets level 2-5

    mini