azure stream analytics - github pages€¦ · analytics on iot edge presentation & action...
TRANSCRIPT
![Page 1: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/1.jpg)
![Page 2: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/2.jpg)
![Page 3: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/3.jpg)
![Page 4: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/4.jpg)
Enriched Data Movement
AutomationDashboarding
![Page 5: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/5.jpg)
![Page 6: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/6.jpg)
Enables customer to get real-time Insights
when time to action is critical
![Page 7: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/7.jpg)
Real-time Fraud Detection Streaming ETL Predictive Maintenance Call Center Analytics
IT Infrastructure and Network Monitoring Customer Behavior Prediction Log Analytics Real-time Cross Sell Offers
Fleet monitoring and Connected Cars Real-time Patient Monitoring Smart Grid Real-time Marketing
![Page 8: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/8.jpg)
![Page 9: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/9.jpg)
Programmer Productivity:
![Page 10: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/10.jpg)
![Page 11: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/11.jpg)
Data ManipulationSELECT
FROM
WHERE
HAVING
GROUP BY
CASE WHEN THEN ELSE
INNER/LEFT OUTER JOIN
UNION
CROSS/OUTER APPLY
CAST INTO
ORDER BY ASC, DSC
Scaling ExtensionsWITHPARTITION BY
OVER
Date and TimeDateNameDatePart Day, Month, YearDateDiffDateTimeFromPartsDateAdd
Windowing ExtensionsTumblingWindowHoppingWindowSlidingWindow
AggregationSUMCOUNTAVGMINMAXSTDEVSTDEVPVARVARPTopOne
StringLenConcatCharIndexSubstringLower, UpperPatIndex
TemporalLagIsFirstLastCollectTop
MathematicalABSCEILINGEXPFLOORPOWERSIGNSQUARESQRT
Geospatial (preview)
CreatePointCreatePolygonCreateLineStringST_DISTANCEST_WITHINST_OVERLAPSST_INTERSECTS
Declarative SQL like language to
describe transformations
Filters (“Where”)
Projections (“Select”)
Time-window and property-based aggregates
(“Group By”)
Time-shifted joins (specifying time bounds within
which the joining events must occur)
and all combinations thereof
![Page 12: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/12.jpg)
1,915 lines of code with Apache Storm
@ApplicationAnnotation(name="WordCountDemo")public class Application implements StreamingApplication{
protected String fileName = "com/datatorrent/demos/wordcount/samplefile.txt";
private Locality locality = null;
@Override public void populateDAG(DAG dag, Configuration conf){
locality = Locality.CONTAINER_LOCAL;WordCountInputOperator input = dag.addOperator("wordinput", new WordCountInputOperator());input.setFileName(fileName);UniqueCounter<String> wordCount = dag.addOperator("count", new UniqueCounter<String>());dag.addStream("wordinput-count", input.outputPort, wordCount.data).setLocality(locality);ConsoleOutputOperator consoleOperator = dag.addOperator("console", new ConsoleOutputOperator());dag.addStream("count-console",wordCount.count, consoleOperator.input);
}}
3 lines of SQL in Azure Stream Analytics
SELECT Avg(Purchase), ScoreTollId, Count(*)
FROM GameDataStream
GROUP BY TumblingWindows(5, Minute), Score
![Page 13: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/13.jpg)
Intelligent Edge and Cloud
![Page 14: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/14.jpg)
Serverless and low TCO
![Page 15: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/15.jpg)
Easy to get started
![Page 16: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/16.jpg)
Stream Analytics on
IoT Edge
Presentation & Action
Storage &Batch Analysis
StreamAnalytics
Event Queuing & StreamIngestion
Event production
IoT Hub
Applications
Archiving for long term storage/ batch analytics
Real-time dashboard
AzureStream
Analytics
Automation to kick-off workflows
Machine Learning
Blob Storage(streaming ingress and reference data)
Event Hubs
Devices &
Gateways
![Page 17: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/17.jpg)
Enterprise grade SLA
![Page 18: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/18.jpg)
![Page 19: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/19.jpg)
![Page 20: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/20.jpg)
![Page 21: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/21.jpg)
There are two distinct types of Inputs
• Data Streams:
• IoT Hub
• Event Hub
• Azure Blob storage
• Reference data:
• Azure Blob storage
Data Outputs Supported
• Azure Data Lake Store
• SQL Database
• Blob storage
• Event Hub
• Power BI
• Table Storage
• Service Bus Queues
• Service Bus Topics
• Azure Cosmos DB
• Azure Functions
![Page 22: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/22.jpg)
![Page 23: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/23.jpg)
![Page 24: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/24.jpg)
![Page 25: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/25.jpg)
![Page 26: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/26.jpg)
![Page 27: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/27.jpg)
Every event that flows through the system has a timestamp
ASA supports:Arrival Time - Event timestamps based on arrival time (input adapter clock, e.g., Event Hubs)
App Time - Event timestamps based on a timestamp field in the actual event tuple
SELECT * FROM EntryStream TIMESTAMP BY EntryTime
SELECT * FROM EntryStream
![Page 28: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/28.jpg)
Output at the end of each window
Windows are fixed length
Used in a GROUP BY clause
1 5 4 26 8 6 4
t1 t2 t5 t6t3 t4
Time
Window 1 Window 2 Window 3
Aggregate
Function (Sum)
18 14Output Events
![Page 29: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/29.jpg)
SELECT TollId, Count(*)
FROM EntryStream TIMESTAMP BY EntryTime
GROUP BY TollId, TumblingWindow(second, 10)
1 5 4 26 8 6 5
0 5 2010 15
Time (s)
1 5 4 26
8 6
25
A 10-second Tumbling Window
30
3 6 1
5 3 6 1
Every 10 seconds give me the count
of vehicles entering each toll booth
over the last 10 seconds
![Page 30: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/30.jpg)
Every 5 seconds give me the
count of vehicles entering each
toll booth over the last 10
seconds
1 5 4 26 8 7
0 5 2010 15 Time
(s)
25
A 10 second Hopping Window with a 5 second hop
30
4 26
8 6
5 3 6 1
1 5 4 26
8 6 5 3
6 15 3
SELECT TollId, Count(*)
FROM EntryStream TIMESTAMP BY EntryTime
GROUP BY TollId, HoppingWindow(second, 10, 5)
![Page 31: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/31.jpg)
SELECT TollId, Count(*)
FROM EntryStream TIMESTAMP BY EntryTime
GROUP BY TollId, SlidingWindow(second, 20)
HAVING Count(*) > 10
1 5
0 10 4020 30 Time
(s)
50
A 20-second Sliding Window
51
5
1
Entry
Exit1 5
Find all toll booths that have
served more than 10 vehicles in
the last 20 seconds
An output is generated whenever an event
either enters/leaves the system
![Page 32: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/32.jpg)
![Page 33: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/33.jpg)
Perform real-time scoring on streaming dataAnomaly Detection and Sentiment Analysis are common use cases
Function calls from the queryAzure ML can publish web endpoints for operationalized ML models
Azure Stream Analytics binds custom function names to such web endpoints
SELECT text, sentiment(text) AS scoreFROM myStream
in public preview
![Page 34: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/34.jpg)
in public preview
![Page 35: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/35.jpg)
![Page 36: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/36.jpg)
in private preview
It is recommended to have at least 50 events in each window for best results.
![Page 37: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/37.jpg)
in private preview
It is recommended to have at least 50 events in each window for best results.
![Page 38: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/38.jpg)
IoT Hub Stream
Analytics
![Page 39: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/39.jpg)
![Page 40: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/40.jpg)
Scenarios where you might find JavaScript user-defined functions useful:
• Parsing and manipulating strings that have regular expression functions, for example, Regexp_Replace() and Regexp_Extract()
• Decoding and encoding data, for example, binary-to-hex conversion
• Performing mathematic computations with JavaScript Math functions
• Performing array operations like sort, join, find, and fill
Here are some things that you cannot do with a JavaScript user-defined function in Stream Analytics:
• Call out external REST endpoints, for example, performing reverse IP lookup or pulling reference data from an external source
• Perform custom event format serialization or deserialization on inputs/outputs
• Create custom aggregates
![Page 41: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/41.jpg)
Provided by Rob Klause
![Page 42: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/42.jpg)
The ASA query
Provided by Rob Klause
![Page 43: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/43.jpg)
![Page 44: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/44.jpg)
![Page 45: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/45.jpg)
![Page 46: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/46.jpg)
IoT Edge
Factory / Customer Site
sqlFunction
AvgtoCloud
Alert
sql
tempSensor
{"routes": {
"TelemetryTosqlFunction": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/sqlFunction/inputs/input1\")","TelemetryToAsa": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/Alert/inputs/temperature\")","AlertsToReset": "FROM /messages/modules/Alert/* INTO BrokeredEndpoint(\"/modules/tempSensor/inputs/control\")","AlertsToCloud": "FROM /messages/modules/Alert/* INTO $upstream","TelemetryToAsaAvg": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/AvgtoCloud/inputs/EdgeStream\")","AvgToCloud": "FROM /messages/modules/AvgtoCloud/* INTO $upstream"
}}
![Page 47: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/47.jpg)
IoT Edge
Factory / Customer Site
sqlFunction
AvgtoCloud
Alert
sql
tempSensor
{"routes": {
"TelemetryTosqlFunction": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/sqlFunction/inputs/input1\")","TelemetryToAsa": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/Alert/inputs/temperature\")","AlertsToReset": "FROM /messages/modules/Alert/* INTO BrokeredEndpoint(\"/modules/tempSensor/inputs/control\")","AlertsToCloud": "FROM /messages/modules/Alert/* INTO $upstream","TelemetryToAsaAvg": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/AvgtoCloud/inputs/EdgeStream\")","AvgToCloud": "FROM /messages/modules/AvgtoCloud/* INTO $upstream"
}}
IoT Hub
![Page 48: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/48.jpg)
IoT Edge
Factory / Customer Site
sqlFunction
AvgtoCloud
Alert
sql
tempSensor
{"routes": {
"TelemetryTosqlFunction": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/sqlFunction/inputs/input1\")","TelemetryToAsa": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/Alert/inputs/temperature\")","AlertsToReset": "FROM /messages/modules/Alert/* INTO BrokeredEndpoint(\"/modules/tempSensor/inputs/control\")","AlertsToCloud": "FROM /messages/modules/Alert/* INTO $upstream","TelemetryToAsaAvg": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/AvgtoCloud/inputs/EdgeStream\")","AvgToCloud": "FROM /messages/modules/AvgtoCloud/* INTO $upstream"
}}
![Page 49: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/49.jpg)
IoT Edge
Factory / Customer Site
sqlFunction
AvgtoCloud
Alert
sql
tempSensor
{"routes": {
"TelemetryTosqlFunction": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/sqlFunction/inputs/input1\")","TelemetryToAsa": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/Alert/inputs/temperature\")","AlertsToReset": "FROM /messages/modules/Alert/* INTO BrokeredEndpoint(\"/modules/tempSensor/inputs/control\")","AlertsToCloud": "FROM /messages/modules/Alert/* INTO $upstream","TelemetryToAsaAvg": "FROM /messages/modules/tempSensor/* INTO BrokeredEndpoint(\"/modules/AvgtoCloud/inputs/EdgeStream\")","AvgToCloud": "FROM /messages/modules/AvgtoCloud/* INTO $upstream"
}}
IoT Hub
Blob Storage
![Page 50: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/50.jpg)
![Page 51: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/51.jpg)
Stream
AnalyticsData
Factory
Blob
StorageEvent Hub Time
Series
Insights
![Page 52: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/52.jpg)
![Page 53: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/53.jpg)
Feature Status Remarks
SQL Parallel Write GA WW rollout in 1 week
Blob O/P partitioning by custom date-time Public preview WW rollout in 1 week
C# UDF on IoT Edge Public preview Available now
Live testing in Visual Studio Public preview Available now
User defined custom repartition count Public preview Available now
New built-in ML models for A/D – Edge and Cloud Private Preview Access granted upon sign-up
Custom de-serializers on IoT Edge Private Preview Access granted upon sign-up
MSI Authentication for egress to ADLS Gen1 Private Preview Access granted upon sign-up
![Page 54: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/54.jpg)
Supports inline learning and real-time scoring
Easily invoked with simple function calls within query language
Types of Anomalies Detected:Spikes
Dips
Slow positive trend
Slow negative trend
Bi-Level change
![Page 55: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/55.jpg)
• Faster iterative testing
• Show results in real time
• View Job metrics
• Time policies support
Public
Preview
![Page 56: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/56.jpg)
Partition egress to Blob storage by
1) Any input field
2) Custom date and time formats
Gain more fine grain control over data written to Blob storage for dashboarding and reporting
Better alignment with Hive conventions for blob output to be consumed by HDInsight and Azure Databricks.
![Page 57: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/57.jpg)
To achieve fully parallel topologies, ASA will transition SQL ‘writes’ from Serial to Parallel operations for SQL DB and SQL Data Warehouse
4x-5x improvement in write throughput
Allows for batch size customization to achieve higher throughput
For e.g., this feature enabled our customer building a connected car scenario to scale up from 150K events/min to 500K events/min
![Page 58: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/58.jpg)
MSI based authentication will enable egress to Azure Data Lake Storage.
Key benefits over existing AAD (Azure Active Directory) based authentication:
• Job deployment automation (thru Power Shell etc.)
• Long running production jobs
• Consistency with other services
![Page 59: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/59.jpg)
Enables better performance tuning
Key Scenarios
• When upstream partition count can’t be changed
• Partitioned processing is needed to scale out to larger processing load
• Fixed number of output partitions
SELECT *
INTO
[output]
FROM
[input]
PARTITION BY
DeviceID INTO 10
![Page 60: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/60.jpg)
Public
Preview
![Page 61: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/61.jpg)
![Page 62: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/62.jpg)
![Page 63: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/63.jpg)
![Page 64: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/64.jpg)
![Page 65: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/65.jpg)
![Page 66: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/66.jpg)
Thank You!
![Page 67: Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production](https://reader034.vdocuments.net/reader034/viewer/2022042222/5ec8038c43fef409d772d5c8/html5/thumbnails/67.jpg)
© 2018 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to
changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the
date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.