bigdata visualization - esri€¦ · bigdata visualization author: esri subject: 2014 esri...

42
BigData Visualization @SpatialAgent @mraad

Upload: others

Post on 31-May-2020

39 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

BigData Visualization @SpatialAgent

@mraad

Page 2: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

BigData ?

Page 3: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

” Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it… - Dan Ariely

Page 4: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 5: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Hadoop Distributed File System (HDFS)

Yet Another Resource Negotiator (YARN)

MapReduce

Commodity Servers

Hadoop Basic Stack

Page 6: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

The Zoo

• Hive - Ad Hoc Query - “SQL” to MapReduce • Pig - High Level Data Analysis Language • Impala - MPP SQL Engine • Mahout - Machine Learning Toolbox • HBase - Columnar KeyValue Database • Cascading - Flow Data Analysis • Avro - Data Serializer • Zookeeper - Centralized State Management

Page 7: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 8: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 9: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

GIS Tools For Hadoop

• Geometry API • Point / Line / Polygon • Operations - Contains, Intersect, Buffer • I/O - WKT, GeoJSON, Shape

• Hive Spatial UDF • ST_POINT, ST_CONTAINS

• GeoProcessing Extensions

Page 10: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Hello, MapReduce !

Page 11: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Density Analysis - Cell Count

Page 12: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

MapReduce Recap

• Map • Extract • Filter • Transform

• Reduce • Group By • Aggregate

Page 13: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Cell Count

function map(lineno,text) { (x,y) = tokenize(text) if(inGrid(x,y)){ (cellX,cellY) = toCell(x,y) emit((cellX,cellY),1) } }

function reduce((cellX,cellY),iterator){ sum = 0 for( one in iterator){ sum = sum + one } emit((cellX,cellY), sum) }

Page 14: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Coolest UX…

Page 15: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 16: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

In Action Demo

Page 17: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

MapReduce Is Hard…

Page 18: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Thinking Of Data As Water

Page 19: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 20: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

To Cell GroupBy

count

X,Y Collection

Cell Count

R M

Source Sink

Filter

Cascading Pipeline

Page 21: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 22: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Spatial Join !

Page 23: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Cascading In Action

Page 24: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

How About No Programming ? What About SQL ?

Page 25: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Hive and Impala

Page 26: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

drop table if exists zipcodes; create external table if not exists zipcodes( id int, lon double, lat double ) row format delimited fields terminated by '\t' lines terminated by '\n' stored as textfile location '/user/cloudera/zipcodes';

Page 27: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

SELECT T.X-180+0.5 AS LON, T.Y-90+0.5 AS LAT, COUNT(*) AS POPULATION FROM ( SELECT FLOOR(LON+180) AS X, FLOOR(LAT+90) AS Y FROM ZIPCODES ) T GROUP BY T.X,T.Y;

Page 28: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Hive and Impala In Action

Page 29: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

ArcGIS & Hadoop

Page 30: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

AIS DATA

• 14.8 Million data points • 1 Month • MMSI, Zulu Time, Lat, Lon, Vessel ID, Draught • Port of Miami - Free

Page 31: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

DEMO Steps

• GP Toolbox • Track Assembly • Hex Generation • Density Analysis

Page 32: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Import Job

AIS CSV

Import Partitioner /ais/YYYY/MM/dd/HH/UUID.csv

HDFS MapReduce

Page 33: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 34: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 35: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 36: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 37: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer
Page 38: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

WebMaps

Page 39: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

/ShippingVolume

/ShippingRank

/ShippingGlobe

/ShippingTracks

http://coolmaps.esri.com/BigData

Page 40: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

Return of KillerApps ?

Page 41: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer

https://github.com/mraad @SpatialAgent

@mraad

Page 42: BigData Visualization - Esri€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer