Cognitive Management of Objects and

Applications for the Internet of Things

Raffaele Giaffreda (CREATE-NET)Raffaele Giaffreda (CREATE-NET)

Keynote at GIIS conference

Trento, 31 Oct 2013

Outline

• Introduction, IoT vs. the Internet

• object virtualisation – separation between object data and object mgmt concerns

• overview of IoT standardisation activities

• interoperability and objects as services – IoT reliability and resilienceresilience

• a top-down perspective on the IoT – user friendliness, wide adoption

• Real World Knowledge modelling and use of cognitive technologies in IoT

• examples of ongoing trials

• conclusions

transistor density / space efficiency

Turing’s Pilot ACE: Automatic

Computing Engine

bandwidth / spectral efficiency

a bit of IoT infographics...

BOSCH

7 bln connected devices by 2015

SAP

24 bln connected devices by 2020

INTEL

31 bln connected devices by 2020

CISCO

37-50 bln connected devices by 2020

others...

Source: IDATE

some initial considerations

• IoT will be BIG

• problems

• human in the loop

• configuring, using, maintaining• configuring, using, maintaining

• handling huge amounts of data produced

the Internet parallel

• imagine the Internet with no browser, no

plugins

• collection of bespoke, non interoperable

content specific applications enabling access content specific applications enabling access

and visualisation of connected files

The Internet parallel

HTTP/WWW

search engines

connect your info

TCP/IP

HTML

represent info / aggregate info

WWWpersonalised knowledge

collections, blogs...

The Semantic Web

find info

VALUE!

The Internet parallelearly stages for the IoT...

HTTP/WWW

The Semantic Web

VALUE!

personalised knowledge

collections, blogs...

represent info / aggregate info

search engines

connect your info

TCP/IP

HTML

WWW

find infoobject

today

Internet vs. Internet of Things

• files vs. objects

• static memory cells vs. energy standalone

units

• need to separate data source from data mgmt • need to separate data source from data mgmt

and operations

• objects virtualisation

Introducing Virtual Objects

VO SW agent

VO APIs

(remote) proprietary API calls

VO SW agent host

the VO concept

• VO exposes several APIs to the upper

layers

– Features, functionalities and resources

can be re-used

• Cognitive control enabled by exposing

APIs which can be used to optimize the

behaviour of the ICT object

VO SW agent may or may not be

Exposed APIs

ICT

object processes

ICT APIs

API calls

ICT object

A simple example: VO SW agent host = laptop with Zigbee dongle

ICT object = Zigbee temperature meternon-ICT object = room

• VO SW agent may or may not be

installed on the ICT object

– Depends on ICT object capabilities

• Association management between ICT

and non-ICT is a real challenge!

• RESILIENCE ASPECTS

Association

non-ICT object17

what do VOs achieve: logical level

iCore FW

Application: pure function

VO Front VO Front

End

VO Front VO Front

End

VO Front VO Front

End

18

Gateway

VO Back

End:

Net Driver

VO Back

End:

Net Driver

VO Back

End:

RWO Driver

VO Back

End:

Net Driver

RWO1

fostering automation - discovery

• description associated with an IoT Object, it better be

machine readable

• i.e. semantic enrichment based on info model for semantic-

based selection

• what is this good for? • what is this good for?

– selection “by relevance”: performance and “selection quality” is

dependent upon combination of enrichment + algorithm that exploits

it...

– assessment of “proximity” is a prerequisite in achieving more

automatic and scalable solutions

VO Information Model – semantic

search

Examples (energy efficiency for

sensors)

• besides discovering and selecting

• virtual representative “takes the heat off” real sensors becoming their actual “manager”– energy efficiency

– reuse– reuse

– resilience

– self-x for constrained resource devices

• conflict resolution (actuators)

• Examples– compression algorithms, data caching, pub/sub

schemes, rules for self-x management

HUMANMACHINE

added value besides sensing efficiency

what happens when it becomes

easier and easier to tap into

object produced data?

cars increasingly more

complex

increasing competition

for owner’s attention

OBD

On Board Diagnostics

HUMANMACHINE

added value besides sensing

efficiency – Innovation potential

“Innovation”: one

can focus on apps!!!

we make “machines” step-in, assisting us!

MACHINEHUMAN

OBD

On Board Diagnostics

the story so far...

• increasing number of objects

• discovery and self-management of objects

• connect and virtualise your objects, unlock

valuevalue

• no mention of application domains...

siloed and bespoke IoT applications

APPS

HO

US

E

APPS

FR

IDG

E

APPS

PA

TIE

NT

APPS

PA

TIE

NT

APPSAPPS

APPSAPPS

APPSAPPS

DATA / INFORMATION OVERLOAD, BUT...

CA

R

SENSORS

HO

US

E

SENSORSF

RID

GE

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

APPS

PA

TIE

NT

SENSORS

APPS

TR

UC

K

SENSORS

APPS

IF A WELL-DEFINED INTERFACE INTO CAR

SENSORS BRINGS SUCH POTENTIAL...

APPS

HO

US

E

APPS

FR

IDG

E

APPS

PA

TIE

NT

APPS

PA

TIE

NT

APPSAPPS

APPSAPPS

APPSAPPS

CA

R

HO

US

E

FR

IDG

E

PA

TIE

NT

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

PA

TIE

NT

SENSORS

APPS

PA

TIE

NT

SENSORS

APPS

TR

UC

K

SENSORS

APPS

SENSORS SENSORS SENSORS SENSORS

of course that’s a dream far from

becoming true...

http://readwrite.com/2013/06/14/whats-holding-up-the-internet-of-things

the IoT standardisation jungle

Real-World Information DB

Au

then

tica

tio

n

Au

then

tica

tio

n

Real-World Knowledge Model (RDF Concepts & Facts)

GUI

Service Requester (Technology Agnostic)

Situation Awareness

Situation Projection

Situation Classification

Situation Recognition

Situation Detection

Queried Fact Collector

Natural Language Processing

(NLP)

Service Request

(SPARQL)

API

Service Request Analysis

RDF Rules

Inference

Engine

Semantic

Query

MatcherLearning

Statistics

Service Analysis

User Characterisation

Learning

Mechanisms

Intent Recognition

Service Templates

Repository

CVO FactoryAP

I

Domain

Expert /

Knowledge

Engineer

AP

I

Ad

min

istr

atio

n &

Man

agem

ent

I/F

Authentication

Authentication

CVO Management Unit

Service Execution Request

M2M

M2MM2M

M2M

M2M

CLOUD

P1723

EPCGlobal

SES

W3C PROV

PROV-DM /PROV-O /

PROV-AQ / PROV-XML

W3C PROV

PROV-DM / PROV-O /

PROV-AQ / PROV-LINKW3C PROV

PROV-DM /PROV-O /

PROV-AQ / PROV-XML

W3C PROV

PROV-DM / PROV-O / PROV-

AQ / PROV-CONSTRAINT

ITU-T

IoT-GSI

ITU-T

FG M2M

ITU-T

FG DistractionISO/IEC

JTC1 WG7

CVO Container (Execution)

Au

then

tica

tio

n

Au

then

tica

tio

nA

uth

enti

cati

on

VO Factory

VO Templates Repository

CVOCVO

CVOCVO

CVO

CVO

Situation ObserverCVO

Situation ObserverCVO

Situation ObserverCVO

Situation Observer

System Knowledge Model

CVO Registry

CVO Templates Repository

VO Registry

VO Management Unit

VO Lifecycle

ManagerCoordination

Data Manipulation

/ Reconciliation

Resource

OptimisationVO Container (WS host)

VOVOVOVOVOVOVOVOVOVO

VO Front End

VO Back End: RWO Driver

Approximation &

Reuse

Opportunity

Detection

CVO Composition

Engine

Orchestration /

Workflow

Management

Learning

Mechanisms

Device

manufactu

rer

API

Resource SensorActuator

GTW/Controller

Data

Processing

Domain

Expert /

Developer

Access

Control

Access

Control

Ad

min

istr

atio

n &

Man

agem

ent

I/F

Installer/

User

Installs

Sensor/Ac

tuator

Devices

Resource SensorActuator

GTW/Controller …………..

….. …..

CVO Lifecycle

ManagerCoordination

Quality

Assurance

Performance

Management

CVO Execution Request

M2M

M2M

M2M

CLOUD

CLOUD

CoAP

CoAP

SSN-XG

MQTT

MQTT

3GPP

3GPP

3GPP 3GPP

EPCGlobal

SPS

SOSSES SAS

SPS

SOS

SPS

SASWNS

SPSSOS

SOS

SOR SIR

CSW

O&M

SensorML

W3C PROV

PROV-DM /PROV-O /

PROV-AQ / PROV-XML

W3C PROV

PROV-DM / PROV-O / PROV-

AQ / PROV-CONSTRAINT

W3C PROV

PROV-DM /PROV-O /

PROV-AQ / PROV-XML

EPCGlobal

ITU-T

IoT-GSI

ITU-T

IoT-GSI

ITU-T

IoT-GSI

ITU-T

IoT-GSI

ITU-T

FG M2M

ITU-T

FG M2M

ITU-T

FG Distraction

ISO/IEC

JTC1 WG7

ISO/IEC

JTC1 WG7

ISO/IEC

JTC1 WG7

LWM2M

LWM2M

LWM2M

LWM2M

LWM2M

LWM2M

LWM2M

courtesy of Panagiotis Vlacheas and Vera Stavroulaki (Piraeus University )

some (good) candidates

• imagine the Internet with no browser, no plugins

• collection of bespoke, non interoperable content

specific applications enabling access and

visualisation of connected filesvisualisation of connected files

an IP based web services view from Sensinode

http://www.iot-week.eu/presentations/thursday/02_Shelby-IoT-Smart-Cities.pdf

Courtesy of Zach Shelby (Sensinode)

fostering interoperability

• at service level (ESBs)

• at communication level (PUB/SUB MQTT bus)

• at device level (GSN)

• no silver bullet...a lot of it will depend on

application context...

useful ingredients?

• common interfaces to interact with

objects (i.e. REST)

• + extra containers for metadata

• let the systems know what the object • let the systems know what the object

is good for, its location (“I am a Temp

sensor in Room A”), its accuracy, its

energy levels etc.

take inspiration from HTML and the Semantic Web

“I am a webpage and I talk about Paris (city of France) history”

Integration at “application level” with all pros and cons associated with it

the story so far...

• increasing number of objects

• discovery and self-management of objects

• connect and virtualise your objects, unlock

valuevalue

• interoperability across application domains

and reliability still big issues...

once achieved the means to access

an objects as a service...

• object redundancy would allow me to cope with resource constraint nature of objects as well as with the diversity of interfaces– if I had a bunch of VO temp objects to chose from I would be much

more likely to tell you what the temperature is...

• semantic enrichment allows me to find alternatives, to foster object reuse and achieve service approximation conceptsobject reuse and achieve service approximation concepts

• here we start entering more the “cognitive-inside” IoT object management territory

• having a logic for choosing the appropriate Virtual Objects according to the application expectations

• having the means to easily connect objects together in a more or less complex graph (CEPs, PUB/SUB channels)

• features of Composite Virtual Objects and associated “CVO Templates”

cognitive mash-ups of semantically interoperable VOs (and their offered

services) which render services matching the application requirements

Introducing the CVO

CVO concept allows for approximate services...

CA

R

APPS

HO

US

E

APPS

FR

IDG

E

APPS

PA

TIE

NT

APPS

HO

US

E

FR

IDG

E

PA

TIE

NT

SENSORS SENSORS SENSORS SENSORS

PATIENT is near the FRIDGE

CAR is near the HOUSE

PATIENT is driving the CAR

objects reuse

across domains

KitchenPresDetect PatientStatusDetect

CVOs allow Automatic Composition

CVOType 1

VOType :: Temp sensor

getTemp()

Subject to constraints:

- Dist (Pos, myPos) < 10m

CVO 1

VOx

FIND

getTemp()

VOType :: Press sensor

getPressure()

- Dist (Pos, myPos) < 10m

- Not already allocated

Logic:

If getTemp() > 20° and getPressure() > 2bar

then NiceWeather

VOy

CP Solver to find VO

allocations that satisfies

all constraints and

minimizes network traffic

USE

leveraging on System Knowledge (i.e. VOx is good and fully

charged) to maintain IoT-based services...

CVO templates

• factoring “smart logic algorithms” out of users /

developers concerns

– IF “crash” THEN “alertRSA”

– “crash” (IF VO_x = TRUE THEN crash := TRUE)

– (IF VO_x = TRUE AND VO_y = TRUE THEN crash := TRUE)

TAG:

crash

detect

VO_x

TAG:

crash

detect

VO_yIF (VO_x = TRUE) AND (VO_y = TRUE)

THEN crash := TRUE

IF VO_x = TRUE

THEN crash := TRUE

IF (VO_x > TH_x) AND (VO_y > TH_y)

THEN crash := TRUE

factor out cognitive technologies

• “ready meals” for IoT apps

workflow-based SEP for CVOs

Car’s sensors/actuators

Open Data (Web)

courtesy of Michele Stecca (M3S)

more info: http://www.slideshare.net/steccami/ieee-icin-2011

Event based CVO execution

CVO ContainerCVO Container

CVOCVO CVO CVO

CEP engineCEP engine

Observer

Observer

Machine Learning

extensions

Machine Learning

extensions

Event / (C)VO Bus (pub/sub based on MQTT)

Event / (C)VO Bus (pub/sub based on MQTT)

CEP engineCEP engine

VO ContainerVO Container

Sensor

VO

Sensor

VO

Sensor

VO

Sensor

VO

Sensor

VO

Sensor

VO

Actuator

VO

Actuator

VO

Actuator

VO

Actuator

VO

courtesy of Walter Waterfeld (Software AG)

more info: http://terracotta.org/downloads/universal-messaging

Internet vs. IoT

• a page + a page + a page...connect info

• represent info – HTML

• aggregate info – hyperlink

• a (sensor) feed + a feed + a feed...

• represent feeds – VO

• aggregate feeds – CVO

the story so far...bottom-up

what’s in here?

user friendliness and user friendliness and

wide adoption...

the story so far...

• increasing number of objects

• discovery and self-management of objects

• connect and virtualise your objects, unlock value

• exploit redundancy pick the most suitable / interoperable / reliable objects interoperable / reliable objects

• VO / CVO services like Lego bricks fostering innovation from IoT makers

• cognitive inside? so far only application-driven match-making

• ultimate goal: user-friendly IoT services fostering wide adoption

a ‘top-down’ view

• routine jobs: water the plants, feed the fish, take my pills, track sent items etc.

• there are objects, sensors, actuators

• there are people (busy lives, forgetful patients, green fingers vs. fingers that “kill every plant they look after”)every plant they look after”)

• objects can be connected

• objects can be mashed-up

• create your own IoT apps (this is what IoTmakers do) vs. provide some input and have this interpreted so the right actions are set to achieve your goals

• make the IoT easy to use and rely upon...

unlocking a huge potential

data

data datada

ta

da

ta

datapresence

patterns exist ...

CVOs

derive patterns of ...

interpret

data

da

ta

data data

data

da

ta data

data

da

ta

datadata

data

data

H/WSE

NS

ING

Real World Objects

(RWO)

VOs

data goldmine

and lots of

siloed

applications

presence

derive patterns of ...

it’s a complex IoT world...

the need for cognitive technologies

• rather than for the selection of appropriate templates,

here focus is on refinement of selected one according

to observed system-reality matching

• Real-World-Knowledge “growing”

• Learning and adaptation to the users preferences• Learning and adaptation to the users preferences

TAG:

crash

detect

VO_x

TAG:

crash

detect

VO_yIF (VO_x > TH_x) AND (VO_y > TH_y)

THEN crash := TRUE

assess

QUALITY of

PREDICTION

REFINE

TH_x, and TH_y

Real World and System Knowledge

models

Real World Knowledge

(RWK)

Models

derive patterns of ...

interpret

data

System Knowledge

(SK)

Models

presence

derive patterns of ...

What are these

good for?

Cognitive Inside where and why...

• Service Level: gather data relate to actions /

situations

• support users (OBSERVE – LEARN – REPLACE)

– routine jobs (watering plants, feeding the fish, taking – routine jobs (watering plants, feeding the fish, taking

pills, switch on/off lights)

– non-expert alerts (a fire, a leak, a fault)

• provide feedback

– improvement of system performance

Some examples please?

• tracking cars in a smart city

• medical equipment tracking and asset

management

tracking cars in a smart city

Best demo

award at

FuNeMS 2013

courtesy of Marc Roelands (Bell Labs – Alcatel Lucent)

more info: http://www.iot-icore.eu/attachments/article/66/iCore_FuNeMS%2713_ALU.pdf

tracking medical equipment

Validate

Train

Execute

RWO parameter reconfiguration

recommendations to improve energy

efficiency of location sensorsIn the demo implementation,

Database of location

information(spatial &

temporal) of objects

3

4

5

Train efficiency of location sensorsIn the demo implementation,

location data of objects is

simulated

1

2

2a4a

6

7

Trento Hospital S. Chiara

Trilogis + ZIGPOS

IoT, Cloud and Big Data

the challenges ahead...

• Big data: “big” relates to the huge number of data sources – have data, patterns exist

– Need to purposefully aggregate data

– scaling-up use of machine learning is a challenge...

• Cloud: constrained devices and limited scope for data • Cloud: constrained devices and limited scope for data processing– dynamic deployment of data-processing resources on the

data-source � data-consumer path is a challenge...

• IoT Networking: delivery of “object-produced” data– M2M traffic and dynamic deployment of connectivity

resources is a challenge...

Conclusions

• increasing number of objects

• discovery and self-management of objects

• connect and virtualise your objects, unlock value

• exploit redundancy pick the most suitable / interoperable /

reliable objects reliable objects

• VO / CVO services like Lego bricks fostering innovation from

IoT makers

• cognitive inside: the importance of modelling the Real World

• Cognitive IoT: user-friendly services fostering wide adoption

• implementations exploiting iCore project results in real trial

settings

• challenging times ahead!

Further info / links

[REF2] P. Vlacheas, R. Giaffreda et al. "Enabling Smart Cities Through

a Cognitive Management Framework for the Internet of Things“,

IEEE Communications Magazine - Special Issue on Smart Cities (June

2013)

[REF1] IERC April 2013 Newsletter – Foreword (see THIS LINK)

[REF3] iCore website (www.iot-icore.eu/latest-news) [REF3] iCore website (www.iot-icore.eu/latest-news)

Best Demo Award at FuNeMS 2013

Thank you!

Raffaele GiaffredaSmart IoT (RIoT) Research Area Head

(CREATE-NET)

EU FP7 iCore Project CoordinatorEU FP7 iCore Project Coordinator

raffaele.giaffreda@create-net.org

Websites:

www.create-net.org/research/research-areas/riot

www.iot-icore.eu

Backup slides

CVO Management Unit

Real World Knowledge/Model

API

Service Request Analysis

iCore User

Preferences

Service Templates

Repository

System Knowledge/Model

Service Execution Request

CVO

Domain Expert

/ Knowledge

Engineer

AP

I

iCore User

Service Request

RW

K

De

sig

n

& S

tore

AP

I

CV

O T

em

pla

te

De

sig

n &

Sto

reUser Profiling

Sys

tem

Ad

min

istr

ati

on

&

Ma

na

ge

me

nt

RWK Update

Situation

Modelling

People

modelling

Natural Language

Processing

the iCore Architecture

VO FactoryVO Template

Repository

CVO

Registry

CVO Templates

Repository

VO

Registry

VO

Container VO OVOVO VO VO Front End

VO Back End: RWO Driver

CVO Factory

Device

manufac

turer

AP

I

Data

Processing

Domain

Expert /

Developer

AP

I

Device

Install

er

VO Management Unit

AP

I

VO Data Session

iCore

System

Operator

CV

O T

em

pla

te

De

sig

n &

Sto

re

AP

IA

PI

Sys

tem

Ad

min

istr

ati

on

&

Ma

na

ge

me

ntVO Execution Request

CVO Container (Execution)CVOCVOCVOCVOCVO CVO

Situation Observer

Real World Objects

(RWO)

Cognitive Inside – take-away messages

(RWK)

Models

(SK)

Models

more dependable IoT

support users of future

Smart Cities applications

(routine + alerts)

more reliable and

interoperable IoT

Models

more resilient IoT

IoT reliability and durability

through VOs

IoT resilience and fault

tolerance

Dublinked initiative

build models and predict!

mash-up data across domains

IBM Research Ireland

personalised journey tips

throughout the execution

iCore ID

3 yrs EU FP7 Integrated Project

(started 1st Oct 2011)

20 Partners with strong industrial

representation

8.7mEur EU Funding

ID Card

Japan

8.7mEur EU Funding

EU + China and Japan