using service bus to connect to the internet of things (iot)
TRANSCRIPT
Using Service Bus to Connect to the Internet of Things (IoT) Markus Horstmann, MicrosoftShawn C. Cutter, Fielding Systems
DEV-B379
Why IoT?Data-Driven Insight (“Telemetry”)More efficient use of resources (reduce cost, environmental impact)More targeted products and services (increase revenue, social impact)
=> “Things” = a rapidly expanding source of raw material for the Insight pipeline
Enable Action at a Distance (“Command & Control”)Engage with customers beyond the point of salePreventive maintenance, Best practices guidance, Proactive sales, Remote servicing etc.
=> From CRM to PRM – “Product Relationship Management”
That’s what this talk is all about…
Why Service Bus for IoT?
Service Bus: The Azure Cloud Device GatewayIoT in the real world: Fielding SystemsTechnology drill down:
In-vehicle diagnostics recorder: .Net Micro Framework and AMQPFielding Systems: Beagleboard – Embedded Linux and AMQP“Reykjavik” accelerator: scalable device gateway, HDInsight analytics and more
New in and around Service BusExpress Queues and Topics: trade off latency vs. durability guaranteesAMQP.Net Lite: client library for .NET Micro Framework, Windows Runtime etc.
http://amqpnetlite.codeplex.com/
What we’ll cover:
1.Enter details of an upcoming trip into your calendar applicationCalendar is configured to share data with the D2D navigation service
2.Before your trip, you get departure time notifications on your phoneService calculates when you should leave based on traffic and weatherIt also considers when you need to allow time to refuel your carService sends notifications to your smart phone as departure time draws near
3.When you start your car, map and directions are already on displayService recommends a different path due to construction or congestionGuides you to open parking spots near your destinationAutomatically pays parking with pre-configured account
4.When you exit your car, the navigation experience transfers to your phone
Service delivers walking directions to complete your journeyHelps you find your car again when finished
Example: Door-to-Door Navigation
100,000s of Vehicles100,000s of Users/Drivers 100,000s of Users/Drivers
Cloud Platform Enablers for Smart Mobility
Iden
tity
Ser
vice
s (W
AAD) Command RouterTelemetry Router
Service Bus + BizTalk Services/Server + Virtual Networks
Notification Hubs
Mobile Services
Web SitesService Bus SignalR
Service BusNotification
Hubs
Custom Protocol GatewayMobile Platform
Push Messaging
Mobile Solution Backend
Web PortalMobile
PlatformPush
Messaging
Vehicle Information
System
Web Sites
Navigation System
Calendar Management
System
Mobile Experience Portal Experience
3rd Party Data & Services (Maps, Weather, Traffic, Events)Corporate and Divisional Line of Business and Information Systems
Service Bus & Web APITelemetry Adapters
Azure Database, Blob and Table StorageReal-Time Analytics, HDInsightVirtual Machines (IaaS)
Cloud Services and Web Sites (PaaS)
Media Services
Service Bus & Web APIService Bus & Web API
IoT Cloud Platform “Stack”
Non-IPCapable (or
willing) Devices
IPCapable Devices Cl
oud
Dev
ice
Gat
eway
Custom
Code
Clou
d Pl
atfo
rm
Serv
ices
EnterpriseSystems
Third-Party Data
and Services
Fiel
d G
atew
ay
IoT Cloud Platform “Stack”: Cloud Device Gateway
Non-IPCapable (or
willing) Devices
IPCapable Devices Cl
oud
Dev
ice
Gat
eway
Custom
Code
Clou
d Pl
atfo
rm
Serv
ices
EnterpriseSystems
Third-Party Data
and Services
Fiel
d G
atew
ay
Cust
omCl
oud
Gat
eway
Pattern 1: Device Direct
Pattern 2: Custom Cloud Gateway
Fiel
d G
atew
ay
Fiel
d G
atew
ay
Service Bus Service Bus
ConnectivityAddressability (non-IP devices, firewalls/NATs, online/offline, roaming)Heterogeneity (OS/firmware, power/network constraints, protocols)Security (identity, authorization, privacy, data integrity)Efficient Scale (millions of devices per tenant, at a reasonable cost)
MessagingTelemetry (collection, filtering, routing, throughput, per-message QoS)Notifications (targeting devices/device groups within large populations)Command/Query and Inquiries (correlation, sessions/batching)
Cloud Device Gateway Challenges
IPv6 + Virtual Private NetworksGive every device a routable IP addressDevices expose services for control/query operationsRemote access enabled within the VPN’s routing domain
IssuesAddressability: network-layer intervention required, loosely connected devices
Security: Devices not protected from each other, control via IP addresses/ports not app-level
Scale: Many devices not VPN-capable, infrastructure costly and hard to maintain
The Default IoT Connectivity Model
Service-Assisted Communication
Connections are device-initiated and outbound
NAT/Firewall Device
(Router)
IP NAT
Cloud Gateway
Command Source
Port mapping is automatic, outbound
Device does not listen for
unsolicited traffic
No inbound ports open, attack
surface is minimized
Access-controlled command API
Secure, managed hosting platform
DNSmyapp.cloudapp.
net
Devices connect via open standard protocolsAMQP 1.0 and HTTP: natively in Service BusMQTT, CoAP and others: implement via custom gateway/adapter modelSockets secured via TLS (or a lightweight variant)
Each device gets a dedicated Inbox/Outbox “queue” on the GatewayOutbox: Telemetry, alerts and service invocationsInbox: commands and queriesRequest/reply: correlated messageon top of inbox/outbox messaging channelsSecurity: device can only access its own specific inbox/outbox endpoints (URI’s)
Service Bus as a Cloud Gateway
Backend Components
Cloud Gateway
Outbox(Telemetry)
Inbox(Command & Control)
Prot
ocol
Hea
d
Com
man
d AP
I
Device
Telemetry Routing with the Azure Service Bus
Split the stream Enable parallel processing
Implement different QoS
levels Level and balance the
load
Topic SubsFilters
Service Bus
Device 2
Receiver 2b
Device 1
Device 3Receiver
2a
Alerts
Data
Receiver 1 Alert Processor
StoragePre-processor
Routing Commands with the Azure Service Bus
TopicSubs Filters
Service Bus
Device 2
Device 1
Device 3Sender 2
Model A Device
1
Model T Device
3
Sender 1Model TDevice
2Model T
Model A
Target individuals or groups
Set delivery timeouts (TTL)
Deal with spotty connectivity
Traverse NATs/firewalls securely
Microsoft Service with Cloud Device Gateways:
Azure Intelligent System Service (in Preview)
Offers a rich SaaS solution and higher-level PaaS components for IoTService Bus is the Cloud Device Gateway component inside ISS
Visual Studio 2013 Profile SyncService Bus terminates all client connection in the Cloud
Halo 4 Game statisticsService Bus fans-out to backend systems behind a Custom Gateway
More exist and many more to come…
The Azure Device Gateway: Service Bus
Fielding Systems: The Digital Oilfield on Azure
Data from peopleData from devices
Oil and Gas: Legacy Devices and NetworksSlave devices and Serial networksAzure Service Bus
Controlled distribution of workloadRedundancy and scalability
Oil and Gas: Smart Devices and IoTMake the devices in the field smarter
Fielding Systems Scenario & Demo
Azure Service Bus: Pull Architecture
Azure Service Bus: Pull Architecture
Demo
Fielding Systems: ScadaVisor
ScalabilityResiliencyLocal data processing
LoggingCompression
Enhanced device securityReduction in latency for alarm notificationsOperate on web standards
Problems addressed by the Push approach
Azure Service Bus: Push Architecture
Demo
Fielding Systems: Smart-device Push• Beaglebone with Qpid Proton Python
Digital Oilfield Big Picture
Technology Drill down: Vehicle Diagnostics
Blob Storage Adapter
HIVE DDL
HDInsight
Azure Table
Storage
Office 365 BI:Excel, Power Maps
HIVE ODBC
Adapter
In-Vehicle Device: .Net GadgeteerFEZ Spider: Arm7, 16MB Ram, 4MB Flash- Onboard Diagnostics Module (OBD2)- Temperature and Humidity Sensor- Ethernet module, tethered via WP (“GSM module”)
Service Bus
Outbox Topic(s)(Telemetry)
Inbox Topic(s)(Command & Control)
Table Storage Adapter
Orleans Adapter
Companion App:How is my car?
Where’s my car?Unlock my car
…Mobile Service
Custom Real-time alerting
Commands
Alerts
Web Site
Device / User PairingNotifications
Device Status Checks
Real-time updates /
Alerts
TelemetryAlerts
Commands
Blob Storage
Demo / Code walkthrough:
Device-to-Cloud (and back)- NetMF with AMQP.Net Lite
-
Use ISS if you want a rich, end-to-end IoT platform
Turn-key client agents, open framework and protocols to roll your ownBackend with rich device repository and device management portal/APIsEvent storage, alerting and analytics, per-device billing modelMore information: http://www.microsoft.com/intelligentsystems/services
Use Service Bus directly if you need to Front ISS with a Custom Gateway for custom protocols or device data modelsBuild your own IoT platform or solution
Service Bus is the Gateway inside ISSNo matter where you start, your backend and device investments accrue
Intelligent System Service and Service Bus
Service Bus Messaging
1. Custom Protocol Gateway
2. Telemetry Pump and Adapters
3. Command Gateway
4. Provisioning Service and Metadata Store
Custom Protocol Gateway Host
MQTT CoAP …
Telemetry/Request Router
Notification/CommandRouter
Adapters Command API Host
Provisioning Service
Device Metadata and Key
Store
HDIn
sight
BizT
alk
Orle
ans
Azur
e St
orag
eAz
ure
Dbs
Serv
ice B
us HTTP
HTTP
Devices
AMQP
1
2 3
4
Configuration
HTTP
Cloud Device Gateway – Reference Architecture
Device Gateway – Partition Topology
The “Partition” is a set of resources dedicated to a specific device population (or subset thereof).The “Master” role manages partition deployment and device provisioning into the partitions.
PartitionMaster
Partition Repo
Command Topics
Service Bus Standard Protocol Custom Protocol
Device Repo
AMQP HTTP MQTT Custom Protocol HostProtocol Adapters
Telemetry Pump/RouterN Instances
Telemetry Adapter Analytics (HDInsigh
t)
Telemetry
AdapterLastValue (Table)
Telemetry
AdapterRealTime (Orleans)
Deployment Runtime
out0000 outFFFF…out000
1out000
2
s000 1
s000 2
s03E 7 s000 1
s000 2
s03E 7
s000 1
s000 2
s03E 7
s000 1
s000 2
s03E 7
g0000/rte0000
g0000/rte0001
out0
out1
out2
n Groups of m Routers
out0
out1
out2
g0001/rte0000
g0001/rte0001
out0
out1
out2
out0
out1
out2
Provisioning Runtime
Ingestion Topics
CommandAPI Host
Ingestion TopicAnalytics
SubscriptionLastValue
SubscriptionRealTime
SubscriptionCommand Response
in0000 inFFFF…in0001 in0002
CA L R CA L R CA L R CA L R
…
Device Gateway – Customer Topology
Global coverage achieved by spreading partitions across multiple Azure regionsReference architecture supports up to 1000 distinct partitionsNumber and distribution of partitions driven by data volumes, business continuity, legal and proximity considerations
Device Gateway Demo
Device Gateway deployment & provisioning
HDInsight adapterTable Storage adapter
Express Queues & TopicsTrade off lower-latency and higher-throughput against possibility of message lossPer-message force-persistence flag lets you get durability backSend us your feedback!Scenarios:
Second screen experiences for gamingInteractive UX (SignalR backplane)
AMQP.Net Lite libraryhttps://amqpnetlite.codeplex.com/ AMQP.Net Lite is a lightweight AMQP client library for use on a broad range of .Net Framework platforms, including .Net Micro Framework, .Net Compact.Vehicle Diagnostics Demo will be available as a sample soon
What’s new in Service Bus
RoadmapFirst and third-party IoT services are being built today on Service BusIoT related enhancements to Service Bus are coming soonWe will simplify and enrich support for Device Gateway patterns
Learn moreClemens Vasters’ video blog: http://channel9.msdn.com/Blogs/SubscribeExplore new SB features: Express Queues and Topics, AMQP.Net Lite
We are looking for customers and partners!Connecting devices at scale to backing services on AzureWilling to work closely with the product team and share learnings/results
Contact:
Markus Horstmann – [email protected]
Shawn C. Cutter - [email protected]
Next Steps
Visit the Developer Platform & Tools BoothHaving a friend buy your coffee?Yea, it’s kind of like that.
MSDN Subscribers get up to $150/mo in Azure credits.
Stop by the Developer Platform and Tools booth and visit the MSDN Subscriptions station to activate your benefits and receive a gift!http://aka.ms/msdn_teched
3 Steps to New Gear! With Application Insights
1. Create a Visual Studio Online account http://visualstudio.com
2. Install
Application Insights Tools for Visual Studio Online http://aka.ms/aivsix
3. Come to our booth for a t-shirt and a chance to win!
VSIP QR Tag Contests Visit our booth to join the hunt for cool prizes!
ResourcesMicrosoft Engineering Stories
How Microsoft Builds Softwarehttp://aka.ms/EngineeringStories
Visual Studio Industry Partner Program
Meet Our New Visual Studio Online Partners or Join Now.http://vsipprogram.com
Visual Studio | Integrate
Create Your Own Dev Environmenthttp://integrate.visualstudio.com
Development tools & services for teams of all sizeshttp://www.visualstudio.com
Complete an evaluation and enter to win!
Evaluate this session
Scan this QR code to evaluate this session.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.