microsoft azure in autonomous driving · microsoft antimalware for azure. sensor data fusion tools...

Microsoft Azure in

Autonomous Driving

We ensure your data is always under your control

We guarantee that the brand and customer experience belongs to you

We complement OEMs and suppliers – not compete

Microsoft’s approach to automotive

OEM System Engineering Process

Regional

Architecture (s)

Feasibility

Study/concept

Exploration

Changes

and Upgrades

Retirement/

replacement

Concepts of

Operations

System

Requirements

High-Level

Design

Detailed Design

Software/Hardware

Development Field

Installation

Unit/Device

Testing

Subsystem

Verification

System

Verification &

Deployment

System

Validation

Operations

and

Maintenance

Engineering Development Product Validation Manufacturing & Service

System Validation Plan

System Verification Plan(System Acceptance)

Subsystem Verification Plan (Subsystem Acceptance)

Unit/Device Test

Plan

Document/Approval

Implementation

Development Processes

Time Line

Lifecycle Processes

Sensor/ Algorithm

Testing (Open Loop)

Process,

Sample, Reduce

Test Vehicle Ingest/ Store Integrate Build

Train

Simulate

ReplayTag

Render/Convert

Control Logic

Validation

Performance

Simulation

Generate

Code

Software

in the loop

Hardware

in the loop

F(x)

Test-Drive

DATA INGEST & CURATE TEST | TRAIN | SIMULATE BUILD | VALIDATE

• Cloud and Analytics partner for

ACM, an autonomous and smart

mobility test facility

• Leveraged by all OEMs, tier ones and

technology start ups

• Engaged with ACM to influence

standards

Industry

Academia Government

• Open source AD Platform

• 200+ Member Consortium

• Microsoft is the cloud provider for

Project Apollo worldwide with the

exception of China

• OpenADx

• Interoperable Eclipse Framework

LG’s autonomous vehicle program had unique requirements,

including portability, security, and fast turnaround time; Data

Box Disk was the perfect solution.

“We needed a way to transfer massive amounts of data for

our autonomous vehicle projects, based all around the

world. The solution needed to be portable, simple to use,

cost-effective and, of course, very secure. The Azure Data

Box Disk met all of those criteria.”

– Hyoyuel (Andy) Kim, Senior Manager, LG Electronics.

Products and services

Azure Data Box Disk

Azure Blob Storage

Organization size

91,000

Industry

Automotive

electronics

Country

Worldwide

Azure Express Route

Azure Storage, Data Lake Gen2

Azure Data Box

Azure Compute

Azure Batch AI

Cycle Cloud

Databricks & HDInsight

Azure Machine Learning

Cognitive Services

Azure IoT Edge

Container and Kubernetes , Visual

Studio Team Services

Azure Key Vault, AAD

Test Vehicle Ingest/ Store

Test-Drive

Integrate Build

Train

Simulate

ReplayTag

Sensor/ Algorithm

Testing (Open Loop)

Process,

Sample, Reduce

Render/Convert

Control Logic

Validation

Performance

Simulation

Generate

Code

Software

in the loop

Hardware

in the loop


F(x)

In-car

data

storage

Data

reduction,

redaction

Azure blob

storage (cool,

archive)

ML

Advanced

Analytics

Azure

Container

Service

Container

RegistrySource

Control

CI/CD flows

HDInsight,

Azure

Databricks

ADLS Gen 2 and Azure blob

storage Hot Tier

Async

upload

Fast

upload

DATA INGEST

IoT Edge

Cycle Cloud

PowerBICosmosDB

SOFTWARE TOOLS AND DEVOPS PLATFORM

Stream Analytics

Machine Learning

Batch

Logic App

Machine Learning

Data Factory

Azure GPU

VM

Kubernetes

ServiceSimplygon

HDInsight

Service Bus Databricks

Virtual machine

Cosmos DB

Sensor/ Algorithm

Testing (Open Loop)

Process,

Sample, Reduce


Train

Simulate

ReplayTag

Render/Convert

Control Logic

Validation

Performance

Simulation

Generate

Code

Software

in the loop

Hardware

in the loop

F(x)

Test-Drive


Offline Data Transfer Online Data Transfer

Data Box Heavy

• Capacity: 1000TB

• Weight 500+ lbs

• Secure, ruggedized

appliance

• Currently in Preview

• Same service as Data Box,

but targeted to petabyte-

sized datasets.

Data Box Gateway

• Virtual device provisioned in

your hypervisor

• Supports storage gateway,

SMB, NFS, Azure blob, files

• Currently in preview

• Virtual network transfer

appliance (VM), runs on your

choice of hardware.

Data Box Edge

• Local Cache Capacity: ~12 TB

• Includes Data Box Gateway

and Azure IoT Edge.

• Currently in preview

• Data Box Edge manages

uploads to Azure and can

pre-process data prior to

upload.

Data Box

• Capacity: 100 TB

• Weight: ~50 lbs

• Secure, ruggedized

appliance

• Data Box enables bulk

migration to Azure when

network isn’t an option.

Data Box Disk

• Capacity: 8TB ea.; 40TB/order

• Secure, ruggedized USB

drives orderable in packs of

5 (up to 40TB).

• Currently in Preview

• Perfect for projects that

require a smaller form factor,

e.g., autonomous vehicles.

Order Fill UploadSend Return Cloud to Edge Edge to Cloud Pre-processing ML Inferencing

Network Data Transfer Edge Compute

Connecting your data center and Azure

Predictable performance and Secure

100 + Express Route partners

Ingest up to PB of data daily with Azure Networking Express Route Service

Private, high speed network connections

Massive Data Ingest – ExpressRoute

Store up to Exabytes of data in massively scalable object storage for

unstructured data

Frequently accessed data

Lowest transaction cost

Immediate (ms)

Less frequently accessed data

Lower capacity

cost

Immediate (ms)

Rarely accessed data

Lowest capacity cost

Hours

Data Storage – Azure Blob Storage

• Foundational service for Microsoft

• >40 million transactions per second

• Multi-PB accounts

• 50+Gbps account throughput

• Continued enhancements under

development

• Client & Service Encryption

• AAD Integration + ACLs

• Broad & deep compliance portfolio

Scalable

Performant

Secure &

Compliant

• Multiple redundancy options

• Strong consistency, data integrity

• Policy: Versioning & WORM locks

Durable

• Integrated storage tiers

• Lifecycle management

• Rich metrics

Cost

Effective

Security management

Data protection

Network security

Threat protection

Identity & access management

Azure Active Directory

Multi-Factor

Authentication

Role Based Access

Control

Encryption

(Disks, Storage, SQL)

Azure Key Vault

VNET, VPN, NSG

Application Gateway

(WAF), Azure Firewall

DDoS Protection

Standard

ExpressRoute

Azure Security Center

Azure Log Analytics

+ Partner Solutions

Azure Active Directory

(Identity Protection)

Microsoft Antimalware

for Azure

Sensor data fusion tools

Data extraction

Labeling and annotation

Data analysis

Data anonymization

Workflow orchestration and

managed services

Partner-based Solutions

AnnotationData

Preparation

Data

Extraction

Microsoft Services

Blob Cognitive Services Services Logic HDInsight Databricks Cosmos DB BatchSearchApps Functions

Data selection and transformation, PII data

redaction, annotation and training data

preparation

Analyze your Data in one place

Azure HDInsight

IoT Hub

Event Hubs

Data Warehouse

Data Factory

App Insights Log Analytics

Monitor

CDN

Search Power BI

Blob Storage Pillars

Open &

Interoperable

Manageable &

Cost Efficient

Scalable &

Performant

Secure &

Compliant

Durable &

Available

Machine Learning

Stream Analytics

Batch

Functions

Data Lake Analytics

Big Data Use Cases

Ingest & ETL Streaming Analytics & Machine Learning Data Aggregation Presentation

Manageable &

Cost Efficient

Scalable &

Performant

Secure &

Compliant

Semantic Segmentation

Each Pixel of the image is assigned a category

Object Detection & Classification

Bounding box drawn around each

object of interest

3D Point Cloud Labeling

Objects of interest as assigned a category in 3D LIDAR point cloud

Deploy AI models in Azure for auto-

labeling images to optimize for speed

of labeling and possibly cost

Sensor/ Algorithm

Testing (Open Loop)

Process,

Sample, Reduce


Train

Simulate

ReplayTag

Render/Convert

Control Logic

Validation

Performance

Simulation

Generate

Code

Software

in the loop

Hardware

in the loop


F(x)

Test-Drive

Open loop testing (e.g. ADTF)

Partnership Opportunities

Scene Reconstruction

and Replay

Microsoft Services

Blob Cycle Cloud CPUCompute

Batch Service BusCosmos

DBPowerBIKey VaultLogic Apps

Verification and validation of training

algorithms and sensors via open loop testing

Simulation tools for sensor and algorithm

validation Algorithm scoring Results AnalysisTest

repository

Workflow Management Services

Comprehensive Validation framework

Build and deploy deep learning models

Azure ML Services

Scale out clusters

Azure

databricks

Notebooks

Azure

databricks

Scale out clusters

Batch AI

Caffe2

MS cognitive

toolkit

Keras

TensorFlow

Choose VMs for your modeling needs

Process video using GPU-based VMs

Run experiments in parallel

Provision resources automatically

Leverage popular deep learning toolkits

Develop your language of choice

Scale compute

resources in any environment

Quickly evaluate

and identify the right model

Streamline

AI development efforts

Leverage your favorite deep learning frameworks

TensorFlow MS Cognitive Toolkit PyTorch Scikit-Learn ONNX Caffe2 MXNet Chainer

Operationalize and manage models with ease

Identify and promote your best models

Capture model telemetry

Retrain models with APIs

Deploy models anywhere

Scale out to containers

Infuse intelligence into the IoT edge

Build and deploy models in minutes

Iterate quickly on serverless infrastructure

Easily change environments

Proactively manage

model performance

Deploy models

closer to your data

Bring models

to life quickly

Train and evaluate models

Azure

databricks

Model MGMT, experimentation, and run history

Azure

ML services

Containers

AKS ACI

IoT edge

Simulation is the fastest way to “drive” the billions of miles needed to discover the breadth of real-world

anomalies required to develop AD solutions.

Scenario Simulation

Scenario definition and simulation for path

planning and navigational maneuvers

(overtake, parallel parking, etc) using post

perception object level simulation.

Run 7200 simulated scenarios covering ~5000

miles per day/per node

Sensor Simulation

Photo-realistic rendering of modeled sensors

(Lidar, radar, RGB cameras) for the training

and validation of Perception AI algorithms.

Render once, test many times

Mathematical based simulation for

Structural CAD design, Fluid dynamics,

Thermo dynamics, material properties

Design sensor properties and placement

on vehicle options at scale before

hardware prototyping is needed. Simulate

accidents and AD navigation to a safe

stop

Engineering Simulation

Build

Test-Drive

Integrate

Test Vehicle Fleet

Ingest/StoreTrain

Sensor Fusion,

Process, Sample,

Reduce, Tag, QA

Performance

Simulation

Control Logic

Validation

Generate

Code

Algorithm

Validation Engine

Simulation

Engine

Production Vehicle Fleet

Training

Dataset

Render/convert

Hardware

in the loop

F(x)

Software

in the loop

Simulation/Reinforcement Learning Centric Model

“Drive Millions, Train on Billions”

Sensor/ Algorithm

Testing (Open Loop)

Process,

Sample, Reduce

Test Vehicle Ingest/ Store

Test-Drive

Integrate Build

Train

Simulate

ReplayTag

Render/Convert

Control Logic

Validation

Performance

Simulation

Generate

Code

Software

in the loop

Hardware

in the loop

F(x)


Test-Drive

Control Logic

Validation

Performance

Simulation

Generate

Code

Software

in the loop

Hardware

in the loop

F(x)

Microsoft Services

Batch Active Directory Container service Blob storage GPU VM

Comprehensive test management

framework

Partner-based Solutions

HIL solutions

Embedded system validation via hardware-in-

loop and software-in-loop

System validation tools

Workflow management services

Managed services

microsoft azure in autonomous driving · microsoft antimalware for azure. sensor data fusion tools...

Documents