value drivers in a changing landscape of modeling &...

1

Value Drivers in a Changing Landscape of

Modeling & Simulation

Pieter J. Mosterman

Chief Research Scientist, Director

Advanced Research & Technology Office

MathWorks

Adjunct Professor

School of Computer Science

McGill University

2

$1,052

3

$11,221

Moore’s Law

4

“When one car learns something, the whole

fleet learns it”

$35,000

Moore’s Law Metcalfe’s Law Elon Musk

5

A smart emergency response system

A feature classification

Evolving architectures

Future value drivers

Machines are connecting

and collaborating

Where can we have

impact, which solutions

are needed, what

challenges these

solutions, and how can we

overcome the challenges?

A smart emergency

response system

6

John R. Boyd

Michael A. Jackson





7

John R. Boyd

Michael A. Jackson





Physics

Information

Electronics

Network

8

Opportunities for

CPS ensembles

John R. Boyd

Michael A. Jackson





Physics

Information

Electronics

Network

9

Kun Zhang

University of Arizona

Enes Bilgin

Boston University

David Escobar Sanabria

University of Minnesota

2013 MathWorks Summer Research Internship

10

İzmit, Turkey, 1999

11

Humanitarian needs

12

Humanitarian needs

13

Humanitarian needs

Where can we help make a

difference?

14

Deploy a heterogeneous fleet …

15

1002

… to serve many (changing!) requests …

16

1004

… across an uncertain infrastructure …

17

1004

… all in a time optimal manner!… all in a time optimal manner!

?

19

Internet

Protocol (IP)

network

backbone

(TCP, UDP)

Geo location

Video stream

Gas sensor stream

Geo location

Geo location

Video stream

Request [‘predefined list’]

Request type [supply, pickup]

Video stream

Audio commands

Geo locations

Video streams



Waypoints

Waypoints

Geo locations

Audio commands

IP traffic

Waypoints

Waypoints

Waypoints

Geo location

Geo location

20

Bluetooth



Internet

Protocol (IP)

network

backbone

(TCP, UDP)

Android OS

Smartphone

Opportunistic

network

Android OS

Smartphone

Opportunistic

network

MATLAB

PC

Face detection

Face recognition

Google Earth

MATLAB

PC

Mission virtualization

MATLAB

PC

Command and

control

Mission planning

User interface

Visualization

BeagleBone

Search and rescue dog

Video streaming

Gas detection

Arduino

Quadrotor drone

Directional antenna

WiFi access point

Ad hoc network

Arduino

Quadrotor drone

Directional antenna

WiFi access point

Ad hoc network

Geo location

Video stream

Gas sensor stream

ATLAS robot

Humanoid operations

KUKA mobile robot

Robotic arm

operation

Omni Phantom

PC

Haptic control

Quadrotor drone

Video stream

Self-driving ground

vehicle

Supply depot

Geo location

Geo location

Video stream



Video stream

Audio commands

Geo locations

Video streams



Waypoints

Waypoints

Geo locations

Audio commands

IP traffic

Waypoints

Waypoints

Waypoints

Geo location

Geo location

Reconnaissance

Fixed-wing UAV

TCP traffic

UDP traffic

Technology

Functionality

Bluetooth traffic

21

2013 MathWorks Summer Research Internship: A Spectacular Challenge (Get cyber real!)

https://youtu.be/MxrySx1m8VQ?t=2m42s

https://youtu.be/MxrySx1m8VQ?t=2m42s

22

hover transfer control control by responder

Man-machine control transfer

https://youtu.be/M3vq1ywbe10?t=41s

2013 MathWorks Summer Research Internship: A Self-flying Drone (Take cyber control!)

https://youtu.be/M3vq1ywbe10?t=41s

https://youtu.be/YytL8-zLE6E

RIVeR Lab has participated in the Global City

Teams Challenge in collaboration with Austin

Texas Fire Department, MathWorks, University

of North Texas, and Worcester Polytechnic

Institute.

https://youtu.be/YytL8-zLE6E

24

TCP traffic

UDP traffic

Technology

Functionality

Bluetooth traffic

MATLAB

PC

Face detection

Face recognitionVideo streams

Internet

Protocol (IP)

network

backbone

(TCP, UDP)

Android OS

Smartphone

Opportunistic

network

BluetoothAndroid OS

Smartphone

Opportunistic

network



Google Earth

MATLAB

PC

Mission virtualization

MATLAB

PC

Command and

control

Mission planning

User interface

Visualization

BeagleBone

Search and rescue

dog

Video streaming

Gas detection

Arduino

Quadrotor drone

Directional antenna

WiFi access point

Ad hoc network

Arduino

Quadrotor drone

Directional antenna

WiFi access point

Ad hoc network

Geo location

Video stream

Gas sensor stream

ATLAS robot

Humanoid operations

KUKA mobile robot

Robotic arm

operation

Omni Phantom

PC

Haptic control

Quadrotor drone

Video stream

Self-driving ground

vehicle

Supply depot

Geo location

Geo location

Video stream



Video stream

Audio commands

Geo locations



Waypoints

Waypoints

Geo locations

Audio commands

IP traffic

Waypoints

Waypoints

Waypoints

Geo location

Geo location

Reconnaissance

Fixed-wing UAV

25





26

Colonel John Richard Boyd

27

The Observe-Orient-Decide-Act (OODA) loop

28

Orient

Decide

Observe Act

OODA and the stages of cognition

Perception Interpretation Cognition

Environment

29

Process (video)

Analyze for validity (filter, reject)

Automatic

Orient

Decide

Observe Act

Environment

Adaptive

Orient

Decide

Observe Act

Environment

Autonomous

Orient

Decide

Observe Act

Environment

Compute control signals

Map to semantic concepts

Fuse sensor data


Reason based on knowledge

Plan for objectives and constraints

Assess alternatives

Determine course of action

Adjust control

Reconfigure behavior

Engineered systems and the stages of cognition

30

Automatic Adaptive Autonomous

Engineered systems and the stages of cognition

Process (video)

Analyze for validity (filter, reject)

Compute control signals

Map to semantic concepts

Fuse sensor data


Reason based on knowledge

Plan for objectives and constraints

Assess alternatives

Determine course of action

Adjust control

Reconfigure behavior

31

Michael Anthony Jackson

32

specificationrequirements &

domain knowledgeprogram

Machine (mh) Environment (eh)

O

(mv)

I (ev)

Requirements engineering

▪ A requirement is a desired relationship

among the phenomena

(e.g., actions/events, states) of the

environment

▪ Phenomena are categorized as

– eh: controlled (or initiated) by the

environment and hidden from (i.e.,

invisible to, not shared with) the machine

– ev: controlled by the environment but

visible to (i.e., shared with) the machine

– mv: controlled by the machine but visible

to (shared with) the environment

– mh: controlled by the machine and hidden

from (i.e., not shared with) the

environment

33

A behavioral view

Program

Open loop possible behavior

Closed loop possible behavior

Property satisfying behavior

Closed loop designed behavior

Machine EnvironmentOut

In

WorldOut

In

||

34

EnvironmentMachine

Out

In

A behavioral view

Program





||

WorldOut

In

35

Machine Environment

Out

In

A behavioral view

Program





World

||

Out

In

36https://youtu.be/STlt48wXsyY?t=17s

For fully-automated container handling in large terminals

and terminal environments, Terex Port Solutions supplies

solutions with outstanding performance.

https://youtu.be/STlt48wXsyY?t=17s

37

ReasonPerceive Interpret

Ensemble

Individual

Connected

Autonomous

Collaborative

Adaptive Automatic

Distributed

Social-cognitive features in control systems

38

Perceive Interpret

Ensemble

Individual

Connected

Reason

Autonomous

Collaborative

Adaptive Automatic

Distributed


39

Perceive Interpret

Ensemble

Individual

Connected

Reason

Autonomous

Collaborative

Adaptive Automatic

Distributed


40





41

system Water clock (in about 270BC the Greek Ktesibios invented a

float regulator for a water clock)

42

physics

H(s) G(s)

control Centrifugal governor(James Watt designed his first governor in 1788

following a suggestion from his business partner

Matthew Boulton)

43

physics

H(s) G(0)

ADC

DACECU

feature

Engine control unit(from a 1996 Chevrolet Beretta)

require

mentstests

44

App

physics

H(s) C

ADC

DACμP

G(z)

G(s)

feature

Freescale MPC561 MCU(32-bit PowerPC embedded microprocessors that

operate between 40 and 66 MHz, used in engine

controllers for General Motors)

RTOS

require

mentstests

45

App App

physics

H(s) C

ADC

DACμP

G(z)

G(s)

comms

network

comms physicsμP

G(z)

H(s)

G(s)

C

ADC

DAC

feature feature

RTOS RTOS

require

mentstests

require

mentstests

46

App App

physics

H(s)

ADC

DACμP comms

network

comms physicsμP

H(s)

ADC

DAC

G(z)

G(s)

C

feature

G(z)

G(s)

C

feature

require

mentstests

OS OS

C++

feature

view1 view2

classes

C++

feature

view1 view2

classes

require

mentstests

VM VM

47

μP μPphysics

H(s)

ADC

DACμP comms

network

comms physicsμP

H(s)

ADC

DAC

OS

VM

App

OS

VM

App

G(z)

G(s)

C

feature

G(z)

G(s)

C

feature

require

mentstests

C++

feature

view1 view2

classes

C++

feature

view1 view2

classes

require

mentstests

48

physics μP

G(z)

H(s)

OS

comms

VM

μP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

VM

App

49

physics μP

G(z)

H(s)

OS

comms

VM

μP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

VM

App

50

physics μP

G(z)

H(s)

OS

commsμP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

App

middleware

51

physics μP

G(z)

H(s)

OS

commsμP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

App

middleware

52

physics μP

G(z)

H(s)

OS

commsμP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

App

middleware

53

Ab

str

ac

tio

nP

hys

ics

(s

tate

)P

hys

ics

(s

tru

ctu

re)

physics μP

G(z)

H(s)

OS

commsμP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

App

middleware

physics

54

Ab

str

ac

tio

nP

hys

ics

(s

tate

)P

hys

ics

(s

tru

ctu

re)

physics μP

G(z)

H(s)

OS

commsμP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

App

middleware

physics

• Time to market

• Cost (design, implementation)

Product

• Robotics, industrial automation

• Medical

• Wearables

• Automotive, Aerospace

Applications

• Speech processing

• Computer vision

• Battery control

Features

• Correctness, consistency

• Heterogeneity

Formalisms

• Feature interaction

• Mixed quality of service (QoS)

• Mixed safety integrity levels (SiL)

Functionality

55

Ab

str

ac

tio

nP

hys

ics

(s

tate

)P

hys

ics

(s

tru

ctu

re)

physics μP

G(z)

H(s)

OS

commsμP

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACFPGA

network

HDL

physicsμP

G(z)

H(s)

comms FPGA

G(s)

C

feature

C++

feature

view1 view2

classes

ADC

DACμP

HDL

require

mentstests

require

mentstests

bit

stream

bit

stream

App

OS

App

middleware

physics

• Models of computation

• Runtime reconfiguration

Implementation

• Performance characterization

Platforms

• Concurrency

• Heterogeneous

• Mixed signal, optical

• FPGA, GPU

• Networks

Technology

• Power, area, speed

• Heat

• Dimensions

Parafunctional

56

physics

μP

OS

commsμPADC

DACFPGA

network

physics

μPcomms FPGAADC

DACμP

bit

stream

bit

stream

App

OS

App

middleware

physics

Information

Physics

Electronics

Network

57

Information

Physics

Electronics

Network

58

Physics

Information Information

Physics

ElectronicsElectronics

Network Network

59

Physics


Physics

Electronics

Network Network

Electronics

App

Resource

60

Electronics

Network

Electronics

Network

Physics Physics


App

Resource

61

Electronics

Network


Electronics

Network

Physics Physics

App

Resource

62

Physics

Electronics

Network


Physics

Electronics

Network

App

Resource

63

Physics

Electronics

Network


Physics

Electronics

Network

App

App

Resource

Resource

Resource

Resource

Open but tightly coupled!

64

Physics

Electronics

Network


Physics

Electronics

Network

•Internet of Things (IoT)

•Machine to Machine (M2M)

•Industry 4.0

•Cyber-Physical Systems

Paradigms

•Intelligent roadway intersection

•Connected operating room

•Swarms

•Emergency response

•Smart grid, micro grid

Applications

•Open (collaborative, configurable, secure)

•Robust (dependable, resilient)

•Natural (intuitive, safe, communicative)

Needs

•Humanoid (speech, vision, dexterity)

•Intelligence (data analytics)

•Longevity (battery control)

Features

•Emerging behavior (certifiable)

•Mixed quality of service (QoS)

•Mixed safety integrity levels (SiL)

Functionality

65

Argentinian director Fernando Livschitz of Black Sheep

Films transforms a busy intersection into a choreographed

dance by cloning cars, bikes, and people.

https://youtu.be/ufK2XRGUjuc

https://youtu.be/ufK2XRGUjuc

66





67

Perceive Interpret

Ensemble

Individual

Connected

Reason

Autonomous

Collaborative

Adaptive Automatic

Distributed


68

Perceive Interpret

Ensemble

Individual

Connected

Reason

Autonomous

Collaborative

Adaptive Automatic

Distributed


69

Wireless communication Service utilization

Exploit distributed information

resources

Reliably configure features with

varying quality of service

Assemble available functionality

into features after deployment

Data sharing

Connected

70


resources





Multirate architectures

Extracting and deriving specific

value from general information

Wireless communication Service utilizationData sharing

Connected

Megamodeling and

metamodeling

Technology challenges in

CPS

71


resources








Physically aware configurable

protocol stack that is IP

compatible

Precise timing and

synchronization in a distributed

environment


Connected

IEEE 802.15.4e low cost

communication

IEEE 1588 precise timing Distributed real-time

systems task scheduling

Processor and

network scheduling

72


resources








Physically aware configurable

protocol stack that is IP

compatible

Precise timing and

synchronization in a distributed

environment

Real-time embedded services

operating in a physical

environment

Smart services discovery

Information sharing in a

heterogeneous system ensemble


Connected

Real-time discovery

services

Real-time middleware Semantic middleware

73

Hardware resource sharing Functionality sharing

Reliably configure an ensemble

online to exploit exogenous

functionality

Contract out endogenous

resources and balance use of

exogenous resources

Purpose functionality to create

novel system features post

deployment

Runtime system adaptation

Collaborative

74



functionality



exogenous resources



deployment

Reasoning and planning

adaptation of an ensemble of

systems

Hardware resource sharing Functionality sharingRuntime system adaptation

Collaborative

Models @ runtime Automated model

calibration

75



functionality



exogenous resources



deployment


Collaborative



systems

Flexible and transferable

embedded functionality dispatch

Performance characterization

from abstract functionality

Real-time virtualizationOpen Services Gateway

Initiative (OSGi)

Platform-based design

76



functionality



exogenous resources



deployment


Collaborative



systems

Multi-use functionality post-

deployment

Feature interaction

Flexible and transferable

embedded functionality dispatch

Performance characterization

from abstract functionality

Data distribution

service (DDS)

Online calibrationRequirements mining Multi-rate double

buffering

77

Design artifact sharing

Collaborate between

stakeholders throughout the

system life cycle

Confidently design systems as

part of a reliable ensemble

Virtual system integration

Design

Emerging behavior design

Systematically design optimal

behavior of system ensembles

78

Collaborate between


system life cycle





Proper models in design

System-level design and analysis

by using models

Connectivity among models,

software, and hardware

Design artifact sharingVirtual system integration

Design


Model Building

Automation SystemCounterexample

guided abstraction

refinement

Hybrid dynamic

systems

Multiparadigm modeling Real-time simulationHyperdense time domain

for hybrid bond graphs

79

Collaborate between


system life cycle







by using models



Collaborative planning, guidance,

and control


Design


Service orchestrationService oriented sensor

programming

80

Collaborate between


system life cycle





Tool coupling among disparate

organizations

Support manifold views and tools

in design



by using models



Collaborative planning, guidance,

and control


Design


Graph transformationsSingle underlying modelOpen services for

lifecycle collaboration

81

Pieter J. Mosterman and Justyna Zander, “Cyber-physical

systems challenges: a needs analysis for collaborating

embedded software systems,” in Software & Systems

Modeling, Springer Berlin/Heidelberg, ISSN 1619-1366, vol.

15, nr. 1, pp. 5-16, 2016

82





83

Conclusions

▪ World is becoming one of

machines that

– Adapt to the environment

– Make decisions autonomously

– Are connected

– Work together

▪ Metcalfe is supplanting Moore as

value driver

▪ Key M&S application areas

– Performance

– Concurrency

– Physics

▪ We need a stupendous range of

technologies combined

– Do not be an individualist

– Let’s discuss!

▪ Opportunities

▪ Solution needs

▪ Starting points

value drivers in a changing landscape of modeling &...

Documents