building iot and big data solutions on azure

27
Building IoT and Big Data Solutions on Azure Ido Flatow Senior Architect, Sela Group Microsoft MVP & RD @idoflatow Level: Intermediate

Upload: ido-flatow

Post on 16-Apr-2017

127 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: Building IoT and Big Data Solutions on Azure

Building IoT and Big Data Solutions on Azure

Ido FlatowSenior Architect, Sela Group

Microsoft MVP & RD @idoflatow

Level: Intermediate

Page 2: Building IoT and Big Data Solutions on Azure

Modern Data – The Big Picture

IoT

User Data

Media Files

Documents

Machine Data

Log Files

Page 3: Building IoT and Big Data Solutions on Azure

IoT 2010

Cell phone

VoIP phone

HVAC

Computer

Vending

Printer

Security

Media player

Oven

Automobile

Smart scale

Refrigerator

Television

Microwave

Coffee maker

Alarm clock

HOME HOMEWORKPLACE

Page 4: Building IoT and Big Data Solutions on Azure

Sleep tracking

COMMUTE COMMUTE

Home security Home automation Leak detection

Smart appliances

Indoor navigation

Health monitoring

Smart lighting

Pet tracking

Information capture

Trip tracking and car health

Control

Child and eldermonitoring

Sports and fitness

Air conditioning and temperature control Environmental sensors

Behavior modification

Garden, lawn and plant care

Food and nutrition tracking

Beacons and proximity

New devices and sensors

Object tracking

Identity Smart vending machines

Medication adherence

Bike ride stats and protection

Entertainment systems

Office equipment

IoT 2016

HOME HOMEWORKPLACE

Page 5: Building IoT and Big Data Solutions on Azure

What We Need• An integrated data solution that will be:

– Able to process events from external sources– Able to walk data through different pipelines– Fast and responsive– Big-Data ready

Page 6: Building IoT and Big Data Solutions on Azure

In Other Words

Consume

BI Dashboards Applications

ProcessETL Aggregations Computation Analysis Querying

PersistHadoop SQL NoSQL

IngestStructured Data Un-Structured Data

Page 7: Building IoT and Big Data Solutions on Azure

Microsoft Azure Services forIoT and BigData

Devices Device Connectivity Storage Analytics Presentation & Action

Event Hubs SQL Database Machine Learning App Service

IoT Hub Table/Blob Storage Stream Analytics Power BI

External Data Sources DocumentDB HDInsight Notification Hubs

External Data Sources Data Factory Mobile Apps

BizTalk Services

{ }

Page 8: Building IoT and Big Data Solutions on Azure

Microsoft Azure Services forIoT and BigData

Devices Device Connectivity Storage Analytics Presentation & Action

Event Hubs SQL Database Machine Learning App Service

IoT Hub Table/Blob Storage Stream Analytics Power BI

External Data Sources DocumentDB HDInsight Notification Hubs

External Data Sources Data Factory Mobile Apps

BizTalk Services

{ }

Page 9: Building IoT and Big Data Solutions on Azure

Field Gateway

Device Connectivity & Management

Analytics & Operationalized Insights

IoT & Data Processing Patterns

Devic

esRT

OS, L

inux

, Wind

ows,

Andr

oid, iO

S

Protocol Adaptation

Batch Analytics & VisualizationsAzure HDInsight, AzureML, Power BI, Azure Data Factory

Hot Path AnalyticsAzure Stream Analytics, Azure HDInsight Storm

Hot Path Business LogicService Fabric & Actor Framework

Cloud GatewayEvent Hubs&IoT Hub

Field Gateway

Protocol Adaptation

Find insights to• Power new services• Improve your “things”

Operationalize your insights in real time

IoT Scale Object Models & Business Logic

Page 10: Building IoT and Big Data Solutions on Azure

Field Gateway

Device Connectivity & Management

IoT with Event HubsDe

vices

RTOS

, Lin

ux, W

indow

s, An

droid

, iOS

Cloud GatewayEvent Hubs

Field Gateway

Protocol Adaptation

Event Hubs• High scale telemetry ingestion service• HTTP/AMQP protocol support• Each Event Hub supports

• 1 million publishers• 1GB/s ingress

• Generally available worldwide• Billions of messages per day• TB of ingested data per day

Page 11: Building IoT and Big Data Solutions on Azure

IoT Hub Cloud Gateway Endpoints

device

Event processing(hot and cold path)

Device provisioning and management

Your IoT Hub

Device id

C2D queueendpoint

D2C send endpoint

Device …

Device …

Device …

D2C receive endpoint

C2D send endpoint

Msg feedback and monitoring endpoint

Device identity managementIoT Hub

management

Device business logic,Connectivity monitoring

Field GW /Cloud GW

Page 12: Building IoT and Big Data Solutions on Azure

IoT Hub Features • Connection

– Bidirectional communication– Reliable & secure channel– Per-device authentication– Multiplexing

• Features– Device to cloud telemetry– Cloud to device commands and notifications (with TTL & feedback)– File uploads/downloads– Monitoring devices (connection, activity, ...)– Multi protocols (AMQP, HTTP) IoT Protocol Gateway (MQTT)

Page 13: Building IoT and Big Data Solutions on Azure

IOT HUBDemo

Page 14: Building IoT and Big Data Solutions on Azure

Azure Stream AnalyticsMission critical reliability and scale

Enables rapid development

Fully managed real-time analytics

• Automatic recovery• Monitoring and

alerting• Scale on demand

• Managed Cloud Service

• Each unit handles 1MB/s

• Can scale up to 1GB/s

• SQL like language• temporal windowing

semantics• support for reference

data

Page 15: Building IoT and Big Data Solutions on Azure

Tumbling Windows• How many vehicles enter each toll booth

every 5 minutes?

SELECT TollId, COUNT(*) FROM EntryStream GROUP BY TollId, TumblingWindow(minute,5)

Page 16: Building IoT and Big Data Solutions on Azure

STREAM ANALYTICSDemo

Page 17: Building IoT and Big Data Solutions on Azure

What is Azure Data Factory?

Azure Data Factory is a managed service to produce trusted information from data stored in the cloud and on-premises. Easily create, orchestrate and schedule highly-available, fault tolerant work flows to move and transform your data at scale.

Page 18: Building IoT and Big Data Solutions on Azure

Evolving Approaches to Analytics

ETL Tool(SSIS, etc)

EDW(SQL Svr, Teradata, etc)

Extract

Original Data

Load

Transformed Data

Transform

OLTP

ERP LOB

…BI Tools

Devices

Web

Sensors

SocialIngestOriginal Data

Scale-out Storage & Compute

(HDFS, Blob Storage, etc)

Transform & Load

Data MartsData

Lake(s)Dashboards

Apps

Streaming data

Page 19: Building IoT and Big Data Solutions on Azure

Data Factory Concepts

Call Log Files

Azure Storage

On Premises Data Mart

Customer Table

Azure DB

Customer Churn Table

Visualize

Data Set(Collection of files, DB table, etc)

Activity: a processing step (Hadoop job, custom code, ML model, etc)

Pipeline: a sequence of activities (logical group)

Customer Call

Details

Customers Likely to Churn

Transform,

Combine, etc

Analyze Move

Page 20: Building IoT and Big Data Solutions on Azure

DATA FACTORYDemo

Page 21: Building IoT and Big Data Solutions on Azure

DocumentDB and Azure Data Services

fully managed, scalable, queryable, schema free JSON document database service for modern applications

fully featured RDBMStransactional processing

rich query managed as a service

elastic scale

internet accessible http/rest

schema-free data model

arbitrary data formats

Page 22: Building IoT and Big Data Solutions on Azure

Data size

Access

Updates

Structure

Integrity

Scaling

Hadoop vs. Relational DB

Page 23: Building IoT and Big Data Solutions on Azure

Hadoop Ecosystem and OSS vs.Azure IoT and BigData Services

Azure Services OSS SolutionsEvent Hubs KafkaIoT Hub Kafka + Mosquitto (MQTT broker)Stream Analytics StormHDInsight Hadoop

Map Reduce Map ReduceHive HiveSpark SparkHBase HBase

Azure ML MahoutData Factory PigDocumentDB MongoDB / Couchbase

Page 24: Building IoT and Big Data Solutions on Azure

DOCUMENTDB & HDINSIGHT

Demo

Page 25: Building IoT and Big Data Solutions on Azure

Azure IoT Hub vs Event HubArea IoT Hub Event Hubs

Communication patterns

Device-to-Cloud Cloud-to-Device

event ingress (device-to-cloud)

Device protocol support

AMQP, AMQP over WebSockets, HTTP, and MQTT

AMQP, AMQP over WebSockets, and HTTP

Security Per-device identity and revocable access control

Event Hubs-wide shared access policies, with limited revocation support

Operations monitoring

Rich set of device identity management Only aggregate metrics

File upload File notification endpoint for workflow integration

Manually request files from devices

Scale Millions of simultaneously connected devices

Limited number of simultaneous connections--up to 5,000 AMQP connections

Device SDKs C, Node.js, Java, .NET, Python Java, .NETC and Node.js in preview

Page 26: Building IoT and Big Data Solutions on Azure

The Bigger PicturePresentation and action

Storage andBatch Analysis

StreamAnalysis

IngestionCollectionEvent production

Event hubs

Cloud gateways(web APIs)

Field gateways

Applications

Legacy IOT (custom protocols)

Devices

IP-capable devices(Windows/Linux)

Low-power devices (RTOS)

Search and query

Data analytics(Power BI)

Web/thick client dashboards

SQL DB

DocumentDBPower BI

Storage

Stream Analytics

Devices to take action

MachineLearning

DataFactory

HDInsight

IoT Hub

Page 27: Building IoT and Big Data Solutions on Azure

Resources• Azure IoT Dev center

– http://www.azure.com/iotdev

• Azure Services– https://azure.microsoft.com/en-us/services/event-hubs– http://azure.microsoft.com/en-us/services/iot-hub– https://azure.microsoft.com/en-us/services/stream-analytics– https://azure.microsoft.com/en-us/services/data-factory– https://azure.microsoft.com/en-us/services/documentdb– https://azure.microsoft.com/en-us/services/hdinsight

• Microsoft and IoT– https://www.microsoft.com/en-us/cloud-platform/internet-of-things– https://blogs.microsoft.com/iot/

• My Info– @IdoFlatow // [email protected] // http://www.idoflatow.net/downloads