coping with iot data - mimming.com...coping with iot data on google cloud platform. google cloud...

Jen TongDeveloper Advocate

Coping with IoT DataOn Google Cloud Platform

Confidential & ProprietaryGoogle Cloud Platform 2

Jen Tong@MimmingCodes

Felipe Hoffa@felipehoffa

http://twitter.com/MimmingCodes




Agenda

● IoT Data Challenges● A use case● A recipe● Workshop

○ Setup stuff○ Simulate the activity○ Capture it with Pub/Sub○ Wrangle it with Dataflow○ Analyze it with BigQuery


About you

● Electrical engineers?● Web developers?● Data scientists?● Mechanical engineers?● Not engineers at all?

Confidential & ProprietaryGoogle Cloud Platform 5Photo credit: Matt Chan

Data

photo credit - taniwha on flickr


Data

photo credit - wemake_cc on flickr


Data


Big data


Google Research Publications


Open Source Implementations

Bigtable

Flume

Dremel


Managed Cloud Versions

Bigtable

Flume

Dremel

Bigtable

Dataflow

BigQuery


Coping with big data


Big data


Really big data

TuesdayWednesday

Thursday


Infinite data

9:008:00 14:0013:0012:0011:0010:002:001:00 7:006:005:004:003:00


Delayed data

9:008:00 14:0013:0012:0011:0010:00

8:00

8:008:00

8:00


Batch Patterns: Creating Structured Data

MapReduce


Batch Patterns: Repetitive Runs

MapReduce

TuesdayWednesday

Thursday


Batch Patterns: Time Based Windows

MapReduce

Tuesday [11:00 - 12:00)

[12:00 - 13:00)

[13:00 - 14:00)

[14:00 - 15:00)

[15:00 - 16:00)

[16:00 - 17:00)

[18:00 - 19:00)

[19:00 - 20:00)

[21:00 - 22:00)

[22:00 - 23:00)

[23:00 - 0:00)


Batch Patterns: Sessions

MapReduce

TuesdayWednesday

Jose

Lisa

Ingo

Asha

Cheryl

Ari

WednesdayTuesday


Streaming Patterns: Element-wise transformations

13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time


Streaming Patterns: Aggregating Time Based Windows

13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time


Delayed data

9:008:00 14:0013:0012:0011:0010:00

8:00

8:008:00

8:00


Streaming Patterns: Event Time Based Windows

11:0010:00 15:0014:0013:0012:00Event Time

11:0010:00 15:0014:0013:0012:00Processing Time

Input

Output


Streaming Patterns: Session Windows

Event Time

Processing Time 11:0010:00 15:0014:0013:0012:00

11:0010:00 15:0014:0013:0012:00

Input

Output


The use case


The game


Problem scope

● Intermittent connectivity● Inconsistent data delivery timing● Large, endless stream of data● Multiple input and output streams● Bursts of activity● Integrate and synchronize multiple event streams


Solution requirements

● Keep up with the event streams● Respond in real-time● Scale up and down with demand● Process data once● Accommodate late-arriving data● Detect anomalies


A recipe

The recipe

Data

The recipe

Pub/SubData

The recipe

Pub/Sub DataflowData

The recipe

Pub/Sub Dataflow BigQueryData


Code time!

Setup stuff

Setup stuff

1. Log into your codelab project2. Install Java, Maven, and the Cloud SDK3. Download the example code4. Setup service account credentials5. Create a Google Cloud Storage 'staging bucket' for Dataflow6. Create a BigQuery Dataset

Detailed instructions: tinyurl.com/cope-iot-data

http://tinyurl.com/cope-iot-data


Cloud Pub/Sub


How Pub/Sub works

Topics Subscriptions Subscribers

Push

Pull

Push


OSS alternative


Cloud Pub/Sub features

● Asynchronous messaging● Many-to-many● Push and pull● At-least-once message delivery● REST/JSON API


Nonfunctional stuff

● Globally available● Automatic scaling● Replicated storage● Encrypted on the wire and at rest


Code time!

Pub/Sub injector


Cloud Dataflow


Cloud Dataflow

Cloud Dataflow is a collection of SDKs for building batch or

streaming parallelized data processing pipelines.

Cloud Dataflow is a fully managed service for executing optimized

parallelized data processing pipelines.


How Dataflow works● A direct acyclic graph of data

processing transformations● Dataflow Service optimizes and

executes● Supports alternate runners● Multiple inputs and outputs● Logical MapReduce operations● PCollections flow through the

pipeline


OSS alternative


Features

● Unified model for streaming/batch analysis● Once-and-only-once input element processing● Autoscaling● Toolkit of complex transforms● Support for event-time stream processing

○ Handles of late data● Session windowing● Real-time analytics● Real-time dashboard and alerts


What it's good at

• Filtering

• Transformation

• Movement

• Extract insights

• Batch

• Continuous

AnalysisETL


Dataflow concepts


PCollections -- pipeline collections

● A collection of data in a pipeline● Bounded or unbounded in size● Created by:

○ Building from a java.util.collection

○ Reading from a backing data store

○ Transforming an existing PCollection

{Seahawks, NFC, Champions, ...}

{..., “NFC Champions #Seahawks”, “Seahawks third #superbowl!”, ... “Je suis #12thMan”, “#GoHawks”, ...}


ParDo -- Parallel Do transformation

● Process each PCollection element independently using a user-provided DoFn

● Both map and reduce phases in Hadoop.


{KV<S, Seahawks>, KV<C,Champions>, <KV<S, Seattle>, KV<N, NFC>, ...}

Key by initial letter


ParDo example


Lowercase

PCollection<String> tweets = …;

tweets.apply(ParDo.of(

new DoFn<String, String>() {

@Override

public void processElement(ProcessContext c) {

c.output(c.element().toLowerCase());

}));

{seakawhs, nfc, champions, ...}


GroupByKey



GroupByKey

{KV<S, {Seahawks, Seattle, …}, KV<N, {NFC, …} KV<C, {Champion, …}}

● Gathers all PCollection elements with the same key

● Shuffle phase in Hadoop


GroupByKey & Combine

● Compute the most common value for each key with GroupByKey and DoFn

● DoFn needs to see all of the elements

● Easier to optimize than CombineFn

GroupByKey

{KV<S, Seahawks>, KV<C,Champion>, KV<S, Seattle>, KV<N, NFC>, ...}

{KV<S, {Seahawks, Seattle, …}>, KV<N, {NFC, …}>,

KV<C, {Champion, …}>}

Combine.groupedValues(TopFn)

{KV<S, Seahawks>, KV<N, NFC>,

KV<C, Champion>}


Windows

● Divide or group elements of a PCollection into windows○ Fixed Windows: hourly, daily, …○ Sliding Windows○ Sessions

● Required for GroupByKey transforms on an unbounded PCollection

Nighttime Mid-Day Nighttime


Composite PTransforms

● Build new PTransforms from existing transforms

● Some utilities are included in the SDK:○ Count, RemoveDuplicates,

Join, Min, Max, Sum… ● Define your own

○ DoSomething, DoSomethingElse...● Why bother?

○ Code reuse○ Easy to monitor

GroupByKey

Pair With Ones

Sum Values Count


Code time!

Start a pipeline


Google BigQuery


OSS alternative


BigQuery

● Scales flat to petabytes● SQL dialect● User defined functions● REST, Web UI, ODBC● 1TB free each month


Demo: count stuff


SELECT count(word)FROM publicdata:samples.shakespeare

Words in Shakespeare


SELECT sum(requests) as totalFROM [fh-bigquery:wikipedia.pagecounts_20150212_01]

Wikipedia hits over 1 hour


SELECT sum(requests) as totalFROM [fh-bigquery:wikipedia.pagecounts_201505]

Wikipedia hits over 1 month


Several years of Wikipedia data

SELECT sum(requests) as totalFROM [fh-bigquery:wikipedia.pagecounts_201105], [fh-bigquery:wikipedia.pagecounts_201106], [fh-bigquery:wikipedia.pagecounts_201107],

...


SELECT SUM(requests) AS totalFROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_2015[0-9]{2}$")')

Several years of Wikipedia data


How about a RegExp

SELECT SUM(requests) AS totalFROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_2015[0-9]{2}$")')WHERE (REGEXP_MATCH(title, '.*[dD]inosaur.*'))


How BigQuery works


Qualities of a good RDBMS



● Inserts & locking● Indexing● Cache● Query planning


Storing data

-- -- -- ---- -- -- ---- -- -- --

Table

Columns

Disks


Reading data: Life of a BigQuery

SELECT sum(requests) as sumFROM ( SELECT requests, title FROM [fh-bigquery:wikipedia.pagecounts_201501] WHERE (REGEXP_MATCH(title, '[Jj]en.+')) )


Life of a BigQuery

L L

MMixer

Leaf

Storage


L L L L

M M

M

Life of a BigQuery

Root Mixer

Mixer

Leaf

Storage


Life of a BigQueryQuery

L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage


Life of a BigQueryLife of a BigQuery

L L L L

M M

MRoot Mixer

Mixer

Leaf

StorageSELECT requests, title



L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage5.4 Bil

SELECT requests, title

WHERE (REGEXP_MATCH(title, '[Jj]en.+'))



L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage5.4 Bil

SELECT sum(requests)

5.8 MilWHERE (REGEXP_MATCH(title, '[Jj]en.+'))




L L L L

M M

MRoot Mixer

Mixer

Leaf

Storage5.4 Bil


5.8 MilWHERE (REGEXP_MATCH(title, '[Jj]en.+'))




Code time!

BigQuery


Wrap up


Big data


Data

photo credit - wemake_cc on flickr


Data

photo credit - taniwha on flickr


Thank you!

Jen Tong@MimmingCodes

Felipe Hoffa@felipehoffa

Slides:mimming.com/presos



http://twitter.com/felipehoffa

http://twitter.com/felipehoffa

https://mimming.com/presos/

https://mimming.com/presos/


Bonus stuff


Cloud BigtableCloud Bigtable


Bigness


Google Internal Bigtable in Numbers

• Storage: 100s of PB

• Throughput: 1,000,000s of QPS

• Bandwidth: 100s of GB/sec


How much is that?

Several Datas worthPhoto credit: jdhancock

https://www.flickr.com/photos/jdhancock/8031897271/

https://www.flickr.com/photos/jdhancock/8031897271/


How much is that?

Millennia of DVD videoPhoto credit: illinoislibrary

https://www.flickr.com/photos/illinoislibrary/14442820810/

https://www.flickr.com/photos/illinoislibrary/14442820810/


Engineering


Engineering

Hundreds of engineer-years worth


Bigtable - The early years

• Jeff and Sanjay decided to build a database service that could scale linearly across thousands and thousands of commodity servers

• Systems will fail, retain performance at scale

• Abandon traditional relational model

• The first generation was about:

• Prototyping and build the service to do its first scaling

• Migrate initial applications to Bigtable

• Figure out replication, and first multi-tenant version of Bigtable


Bigtable - Stabilized

• From batch only, to serving web traffic

• Low latency for 99th percentile of requests

• Polish the Bigtable service

• React better to abusive usage

• Mixed media clusters - mixture of SSD + HDD storage with configurable affinity

• Bring tablet server recovery time from 10s of seconds to 1 second or less

• Easier replication


Google Cloud Bigtable

• A fully-managed service

• Focus more on your business, less on infrastructure

• Straightforward pricing model


Data Model


Data model


How it works


HBase Architecture

HBase Cluster

Region Server

Region Server

Region Server

Region Server

Master

Region Server

Bloomfilter

Memory Table

WAL

Block Cache

RegionRegion

Region Region

ZooKeeper

HBase Client

HDFS


Bigtable Architecture

Bigtable Cell

Tabletserver

Tabletserver

Tabletserver

Tabletserver

Master

Tabletserver

Bloomfilter

Memtable

Sharedlog

Block Cache

TabletTablet

Tablet Tablet

Chubby

HBase Client

Colossus


Bigtable System Architecture

Bigtable Cell

Tabletserver

Tabletserver

Tabletserver

Tabletserver

Master

Tabletserver

Bloomfilter

Memtable

Sharedlog

Block Cache

TabletTablet

Tablet Tablet

Chubby

HBase Client

Colossus



Bigtable Cell

Tabletserver

Tabletserver

Tabletserver

Tabletserver

Master

Tabletserver

Bloomfilter

Memtable

Sharedlog

Block Cache

TabletTablet

Tablet Tablet

Chubby

HBase Client

Colossus


Life of Bigtable data


When it's awesome


Management

Who in the audience have used HBase before?

Things you will not see in Cloud Bigtable:

Compactions

Pre-splitting

Lots of configuration settings

1 minute regionserver outages

Coprocessors (for now)


Throughput

Write Throughput (MB/s)Mixed Read/Write Throughput(MB/s)


Latency

late

ncy

(ms)

at

99%

read

update


Financial ServicesFaster risk analysis, credit card fraud/abuse

Marketing/ Digital MediaUser engagement, clickstream analysis, real-time adaptive content

Internet of ThingsSensor data dashboards and anomaly detection

TelecommunicationsSampled traffic patterns, metric collection and reporting

EnergyOil well sensors, anomaly detection, predictive modeling

BiomedicalGenomics sequencing data analysis

Cloud Bigtable Use Cases


When not to use it

• Relational joins, like for online transaction processing

• Interactive querying

• Blobs over 10MB

• ACID transactions

• Automatic cross-zone replication

• You don't have much data yet


When not to use it

• Relational joins, like for online transaction processing - Cloud SQL

• Interactive querying - BigQuery

• Blobs over 10MB - Cloud Storage

• ACID transactions - Datastore

• Automatic cross-zone replication - Datastore

• You don't have much data yet - Datastore, Firebase, or Cloud SQL


More bonus stuff

Build one of something with Firebase

http://mimming.com/presos/internet-of-nodebots/

http://mimming.com/presos/internet-of-nodebots/

coping with iot data - mimming.com...coping with iot data on google cloud platform. google cloud...

Documents