coping with iot data · 7/19/2016 · coping with iot data on google cloud platform. jen tong...

Jen TongDeveloper Advocate

Coping with IoT DataOn Google Cloud Platform

Jen Tong

Developer AdvocateGoogle Cloud Platform

@MimmingCodesmimming.com

http://twitter.com/MimmingCodes


http://mimming.com

http://mimming.com

Agenda

● IoT Data Challenges● A use case● A recipe● Demos

○ Simulate the IoT○ Capture with Pub/Sub○ Wrangle with Dataflow○ Analyze with BigQuery

Confidential & ProprietaryGoogle Cloud Platform 4

About you

● Electrical engineers?● Web developers?● Data scientists?● Mechanical engineers?● Not engineers at all?

Confidential & ProprietaryGoogle Cloud Platform 5Photo credit: Matt Chan

Data

photo credit - taniwha on flickr


Data

photo credit - wemake_cc on flickr


Data


Big data


Google Research Publications


Open Source Implementations

Bigtable

Flume

Dremel


Managed Cloud Versions

Bigtable

Flume

Dremel

Bigtable

Dataflow

BigQuery


Coping with big data


Big data


Really big data

TuesdayWednesday

Thursday


Infinite data

9:008:00 14:0013:0012:0011:0010:002:001:00 7:006:005:004:003:00


Delayed data

9:008:00 14:0013:0012:0011:0010:00

8:00

8:008:00

8:00


Batch Patterns: Creating Structured Data

MapReduce


Batch Patterns: Repetitive Runs

MapReduce

TuesdayWednesday

Thursday


Batch Patterns: Time Based Windows

MapReduce

Tuesday [11:00 - 12:00)

[12:00 - 13:00)

[13:00 - 14:00)

[14:00 - 15:00)

[15:00 - 16:00)

[16:00 - 17:00)

[18:00 - 19:00)

[19:00 - 20:00)

[21:00 - 22:00)

[22:00 - 23:00)

[23:00 - 0:00)


Batch Patterns: Sessions

MapReduce

TuesdayWednesday

Jose

Lisa

Ingo

Asha

Cheryl

Ari

WednesdayTuesday


Streaming Patterns: Element-wise transformations

13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time


Streaming Patterns: Aggregating Time Based Windows

13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time


Delayed data

9:008:00 14:0013:0012:0011:0010:00

8:00

8:008:00

8:00


Streaming Patterns: Event Time Based Windows

11:0010:00 15:0014:0013:0012:00Event Time

11:0010:00 15:0014:0013:0012:00Processing Time

Input

Output


Streaming Patterns: Session Windows

Event Time

Processing Time 11:0010:00 15:0014:0013:0012:00

11:0010:00 15:0014:0013:0012:00

Input

Output


The use case


The game


Problem scope

● Intermittent connectivity● Inconsistent data delivery timing● Large, endless stream of data● Multiple input and output streams● Bursts of activity● Integrate and synchronize multiple event streams


Solution requirements

● Keep up with the event streams● Respond in real-time● Scale up and down with demand● Process data once● Accommodate late-arriving data● Detect anomalies


A recipe

The recipe

Data

The recipe

Pub/SubData

The recipe

Pub/Sub DataflowData

The recipe

Pub/Sub Dataflow BigQueryData


Cloud Pub/Sub


How Pub/Sub works

Topics Subscriptions Subscribers

Push

Pull

Push


OSS alternative


Cloud Pub/Sub features

● Asynchronous messaging● Many-to-many● Push and pull● At-least-once message delivery● REST/JSON API


Nonfunctional stuff

● Globally available● Automatic scaling● Replicated storage● Encrypted on the wire and at rest


Demo time!

Pub/Sub injector


Cloud Dataflow


Dataflow Pipelines

Data sources


Dataflow Pipelines

Pipeline Steps


Dataflow Pipelines

Destinations


Dataflow Pipelines


OSS alternative


Features

● Unified model for streaming and batch analysis● Once-and-only-once input element processing● Autoscaling● Toolkit of complex transforms● Support for event-time stream processing

○ Handles late data● Session windowing● Real-time analytics, dashboard, and alerts


What it's good at

• Filtering

• Transformation

• Movement

• Extract insights

• Batch

• Continuous

AnalysisETL


Demo time!

Start a pipeline


Google BigQuery


OSS alternative


BigQuery

● Scales flat to petabytes● SQL dialect● User defined functions● REST, Web UI, ODBC● 1TB free each month


Off topic demo!

Count stuff


SELECT count(word)FROM publicdata:samples.shakespeare

Words in Shakespeare


SELECT sum(requests) as totalFROM [fh-bigquery:wikipedia.pagecounts_20150212_01]

Wikipedia hits over 1 hour


SELECT sum(requests) as totalFROM [fh-bigquery:wikipedia.pagecounts_201505]

Wikipedia hits over 1 month


Several years of Wikipedia data

SELECT sum(requests) as totalFROM [fh-bigquery:wikipedia.pagecounts_201105], [fh-bigquery:wikipedia.pagecounts_201106], [fh-bigquery:wikipedia.pagecounts_201107],

...


SELECT SUM(requests) AS totalFROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_2015[0-9]{2}$")')

Several years of Wikipedia data


How about a RegExp

SELECT SUM(requests) AS totalFROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_2015[0-9]{2}$")')WHERE (REGEXP_MATCH(title, '.*[dD]inosaur.*'))


Demo time!

BigQuery


Wrap up


Big data


Data

photo credit - wemake_cc on flickr


Data

photo credit - taniwha on flickr

Thank you!

Jen TongDeveloper AdvocateGoogle Cloud Platform@MimmingCodes

Free trial: cloud.google.com

Slides:mimming.com/presos



https://cloud.google.com

https://cloud.google.com

https://mimming.com/presos/

https://mimming.com/presos/


Bonus stuff


Dataflow

Ozmg code


PCollections -- pipeline collections

● Collection of data in a pipeline

● Bounded or unbounded in size

{Seahawks, NFC, Champions, ...}

{..., “NFC Champions #Seahawks”, “Seahawks third #superbowl!”, ... “Je suis #12thMan”, “#GoHawks”, ...}


ParDo -- Parallel Do transformation

● Process each PCollection element independently using a user-provided DoFn

● Both map and reduce phases in Hadoop.


{KV<S, Seahawks>, KV<C,Champions>, <KV<S, Seattle>, KV<N, NFC>, ...}

Key by initial letter


ParDo example


Lowercase

PCollection<String> tweets = …;

tweets.apply(ParDo.of(

new DoFn<String, String>() {

@Override

public void processElement(ProcessContext c) {

c.output(c.element().toLowerCase());

}));

{seakawhs, nfc, champions, ...}


GroupByKey



GroupByKey

{KV<S, {Seahawks, Seattle, …}, KV<N, {NFC, …} KV<C, {Champion, …}}

● Gathers all PCollection elements with the same key

● Shuffle phase in Hadoop


GroupByKey & Combine

● Compute the most common value for each key with GroupByKey and DoFn

● DoFn needs to see all of the elements

● Easier to optimize than CombineFn

GroupByKey

{KV<S, Seahawks>, KV<C,Champion>, KV<S, Seattle>, KV<N, NFC>, ...}

{KV<S, {Seahawks, Seattle, …}>, KV<N, {NFC, …}>,

KV<C, {Champion, …}>}

Combine.groupedValues(TopFn)

{KV<S, Seahawks>, KV<N, NFC>,

KV<C, Champion>}


Windows

● Divide or group elements of a PCollection into windows○ Fixed Windows: hourly, daily, …○ Sliding Windows○ Sessions

● Required for GroupByKey transforms on an unbounded PCollection

Nighttime Mid-Day Nighttime


Composite PTransforms

● Build new PTransforms from existing transforms

● Some utilities are included in the SDK:○ Count, RemoveDuplicates,

Join, Min, Max, Sum… ● Define your own

○ DoSomething, DoSomethingElse...● Why bother?

○ Code reuse○ Easy to monitor

GroupByKey

Pair With Ones

Sum Values Count


How BigQuery works


Qualities of a good RDBMS



● Inserts & locking● Indexing● Cache● Query planning


Storing data

-- -- -- ---- -- -- ---- -- -- --

Table

Columns

Disks


Reading data: Life of a BigQuery

SELECT sum(requests) as sumFROM ( SELECT requests, title FROM [fh-bigquery:wikipedia.pagecounts_201501] WHERE (REGEXP_MATCH(title, '[Jj]en.+')) )


Life of a BigQuery

L L

MMixer

Leaf

Storage


L L L L

M M

M

Life of a BigQuery

Root Mixer

Mixer

Leaf

Storage


Life of a BigQueryQuery

L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage


Life of a BigQueryLife of a BigQuery

L L L L

M M

MRoot Mixer

Mixer

Leaf

StorageSELECT requests, title



L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage5.4 Bil

SELECT requests, title

WHERE (REGEXP_MATCH(title, '[Jj]en.+'))



L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage5.4 Bil

SELECT sum(requests)

5.8 MilWHERE (REGEXP_MATCH(title, '[Jj]en.+'))




L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage5.4 Bil


5.8 MilWHERE (REGEXP_MATCH(title, '[Jj]en.+'))



coping with iot data · 7/19/2016 · coping with iot data on google cloud platform. jen tong...

Documents