apache spark linkedin

Slide title

CAPITALS

Slide subtitle

minimum 30 pt

Slide title

Text and bullet level 1

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Characters for Embedded font: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fifl

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or

text in the footer area Ericsson Internal | 2015-08-11 | Page 2

• W h a t ?

• Wh y ?

• H o w ?

• D e m o

• EDR A n a l y t i c s

AGENDA

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Spark eco-system

Technology landscape

Spark eco-system

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

“Fast and general engine for big data processing with libraries for SQL, streaming, advanced analytics(machine learning)

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Originally developed in 2009 in

UC Berkeley’s AMPLab

Fully open sourced in 2010 –

now at Apache Software

Foundation

http://spark.apache.org

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Spark is the Most Active Open Source Project in Big Data

Giraph

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Distributors Applications

The Spark Community

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

2015 SNAPSHOT

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

WHY SPARK? Speed

Run programs up to 100x faster than Hadoop Map Reduce in memory, or 10x faster on disk.

Ease of Use

Supports different languages for developing applications using Spark

Generality

Combine SQL, streaming, and complex analytics into one platform

Runs Everywhere Spark runs on Hadoop, Mesos, standalone, or in the cloud.

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Easy: Get Started Immediately

Interactive Shell

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Monitoring

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

FEATURE COMPARISON

Source: Daytona GraySort benchmark, sortbenchmark.org

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

WORD COUNT

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Spark eco-system

Local YARN Mesos

Spark Streaming Spark SQL GraphX MLLib

Spark Core Engine (Scala/Java/Python)

Standalone cluster

Persistence

Cluster Manager

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

SPARK ON HDFS

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

HADOOP SPARK

SQL Query interface HIVE SPARKSQL

Machine Learning APACHE MAHOUT MLIB

Graph processing APACHE GIRAPH GRAPHX

Streaming APACHE STORM SPARK STREAMING

ECOSYSTEM

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

So, HOW is It BETTER

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

THE BIG QUESTION?

Is Spark going to replace Hadoop?

Answer – Yes, Spark will be used on top of Hadoop and replace

MapReduce Reasons:

1. Hadoop MapReduce cannot handle real-time

processing

2. Hadoop MapReduce is slower than Hadoop Spark

3. With rise of IOT, Spark is a must

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

RDD & SPARK COMPONENTS

Technology landscape

Spark eco-system

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

RESILIENT Distributed Dataset

RDDs track l i n e a g e information that can be used to efficiently

re-compute lost data

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Partitions in the cluster

SparkM

SparkW

SparkW SparkW

SparkW

partition

@doanduyhai

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

RDD TRANSFORMATIONS & ACTIONS

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

PARTITION TRANSFORMATION

map(tuple => (tuple._3, tuple))

groupByKey()

countByKey()

partition

direct transformation

shuffle

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Stage 1 Stages

Shuffle operation

Stage 2

Delimits "shuffle"

frontiers

@doanduyhai

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

SPARK COMPONENTS

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

SPARK STREAMING

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

SPARK SQL

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Let’s try some examples…

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Spark Shell

./bin/spark-shell --master local[2]

The --master option specifies the master URL for a distributed cluster, or local to run locally with one thread, or local[N] to run locally with N threads. You should start by using local for testing.

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

scala> textFile.count() // Number of items in this RDD ees0: Long = 126 scala> textFile.first() // First item in this RDD res1: String = # Apache Spark

scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))

Simplier scala> textFile.filter(line => line.contains("Spark")).count() // How many lines contain "Spark"?

res3: Long = 15

scala> val textFile = sc.textFile(“../README.md")

textFile: spark.RDD[String] = spark.MappedRDD@2ee9b6e3

Basic operations…

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Map - Reduce

scala> textFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b) res4: Long = 15

scala> import java.lang.Math scala> textFile.map(line => line.split(" ").size).reduce((a, b) => Math.max(a, b)) res5: Int = 15

scala> val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)

wordCounts: spark.RDD[(String, Int)] = spark.ShuffledAggregatedRDD@71f027b8

wordCounts.collect()

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

With Caching…

scala> linesWithSpark.cache() res7: spark.RDD[String] = spark.FilteredRDD@17e51082 scala> linesWithSpark.count() res8: Long = 15 scala> linesWithSpark.count() res9: Long = 15

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

With HDFS…

val lines = spark.textFile(“hdfs://...”) val errors = lines.filter(line => line.startsWith(“ERROR”)) println(Total errors: + errors.count())

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Job Submission

$SPARK_HOME/bin/spark-submit \ --class "SimpleApp" \ --master local[4] \ target/scala-2.10/simple-project_2.10-1.0.jar

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Configuration

val conf = new SparkConf() .setMaster("local") .setAppName("CountingSheep") .set("spark.executor.memory", "1g")

val sc = new SparkContext(conf)

Slide title

minimum 24 pt

Bullets level 2-5

minimum 20 pt

SQL to RDD Translation

Projection & selection

SELECT name, age FROM people

WHERE age ≥ 13 AND age ≤ 19

SELECT name, age

WHERE age ≥ 13 AND age ≤ 19

val people:RDD[Person] val teenagers:RDD[(String,Int)]

= people .filter(p => p.age ≥ 13 && p.age ≤ 19) .map(p => (p.name, p.age))

.map(p => (p.name, p.age))

.filter(p => p.age ≥ 13 && p.age ≤ 19)

THANK YOU

apache spark linkedin

Technology

accelerator for apache spark functional specification ·...

using apache spark, apache kafka and apache...

developing apache spark applications · apache spark...

apache spark - lmu

introduction to cassandra • why spark - apache cassandra |...

apache spark 2.0

managed solutions apache spark® · apache spark® apache...

developing apache spark applications - cloudera · apache...

apache spark introduction

apache spark チュートリアル

apache spark briefing

big data with apache spark - wunca · 2017-07-21 · -...

[@naukriengineering] apache spark

introducciÓn a apache spark con...

apache spark - yandex

using apache spark pat mcdonough - databricks. apache spark...

apache spark

r + apache spark

integrating apache hive with kafka, spark, and...

apache spark meetup