FRUCT 17, Conference Opening

Workshop on Cyber-Physical-Social Systems

Yaroslavl, Russia 22.04.2015

Cyber-Physical-Social Systems

Alexey Kashevnik, e-mail: alexey@iias.spb.su

PhD, Senior Researcher

Laboratory of Computer Aided Integrated Systems

St.Petersburg Institute for Informatics and Automation of the

Russian Academy of Sciences (SPIIRAS), St.Petersburg, Russia

International Research Laboratory

«Intelligent Technologies for Socio-Cyberphysical Systems»,

ITMO University, St.Petersburg, Russia

Table of Contents

Motivation

Cyber-Physical and Cyber-Physical-Social Systems

Development Technologies

Smart Space

Context for CPSS

Data, Information, Knowledge

Semantic Web, Ontologies

Ontology Matching Method

Smart Spaces

Smart-M3 Platform

Access Control Model for Smart Space

Application Examples

2

Motivation

Device interaction technologies

which allows to achieve maximum

profit: Internet of Things (IoT), World

Wide Sensor Network, Smart

Building Environment, and etc.

Limitation of cognitive human

abilities, which progress slowly than

computers (usually at the moment

human can not keep in mind all

information he/she need for making

decisions). 3

Increasing number of devices with built-in

processors and data storage possibilities

(more than 50 billions of devices, more than 5 billions of smartphones will be available in 10 years).

Top 12 Technologies by McKinsey Global

Institute (May 2013)

4

Source: McKinsey Global Institute, Report MGI “Disruptive technologies: Advances that will

transform life, business, and the global economy” (May 2013).

http://www.mckinsey.com/insights/business_technology/disruptive_technologies

Modern Technologies

5

Mobile Internet Increasingly inexpensive and capable mobile

computing devices and Internet connectivity

Automation of knowledge

work

Intelligent software systems that can perform

knowledge work tasks involving unstructured

commands and subtle judgments

The Internet of Things Networks of low-cost sensors and actuators for

data collection, monitoring, decision making, and

process optimization

Cloud computing Use of computer hardware and software

resources delivered over a network or the

Internet, often as a service

Advanced robotics Increasingly capable robots with enhanced

senses, dexterity, and intelligence used to

automate tasks or augment humans

From Industry 1.0 to Industry 4.0

6

Source: German Research Center for Artificial Intelligence (2011)

Cyber-Physical Systems (CPS)

7

“Cyber-physical systems are physical and engineered

systems whose operations are integrated, monitored,

and/or controlled by a computational core.

Components are networked at every scale.

Computing is deeply embedded into every physical

component, possibly even into materials. The

computational core is an embedded system, usually

demands real-time response, and is most often

distributed. ”

Helen Gill, NSF, USA

Cyber-Physical Systems or “smart” systems are co-engineered interacting

networks of physical and computational components. These systems will

provide the foundation of our critical infrastructure, form the basis of emerging

and future smart services, and improve our quality of life in many areas.

National Institute of Standards and Technology, USA

Difference Between CPS and CPSS

8

People

Interacted devices

Physical devices

Cyber-Physical-Social Systems (CPSS)

CPSS in contrast with CPS consist of not only cyberspace and physical

space, but also human knowledge, mental capabilities, and

sociocultural elements.

9

Physical world IT world

Social Networks

Cyber-

Physical-

Social

Systems

Example of CPSS

10

Communication harvester – back-office

o Cut tree; measure length, diameter, wood quality.

o Check markets prices and customer orders.

o Decide on usage of the tree (which product, what customer).

o Cut according to back-office decision and mark product.

Smart Space for CPSS

11

Semantic Integration

CPSS Smart Space Social Space Physical World

Semantic Integration

Smart Space – computational environment consisting of

multiple heterogeneous resources (electronic and

computational devices, Internet pages, data based, etc.) which

has intelligent behavior and can proactively provide services

taking into account current situation.

Types of Resources in CPSS

12

Cyber-Physical Social System

Acting

Resources

Computational

Resources

Social Network

Information

Resources

Physical

Environment Smart Space

Social Space

Crowdsourcing

13

*) Jeff Howe “Crowdsourcing: A Definition” http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html **) Yano Research Institute: http://www.yanoresearch.com/press/press.php/001161

Crowdsourcing market size in Japan**

“The act of a company or institution taking a function once performed

by employees and outsourcing it to an undefined (and generally large)

network of people in the form of an open call” (Jeff Howe*)

Crowd Computing

14

Crowd computing* – “an umbrella term to define a

myriad of tools that allow human interaction to exchange

ideas, non-hierarchical decision making and full use of

mental space of the globe”.

Characteristics**:

A crowd of humans

Computer-mediated interaction

Purposive crowd activity

Task utilize human capabilities

(optional) Harnessing collective intelligence

*) Schneider D, de Souza J, Moraes K. Multidões: a nova onda do CSCW? In Brazilian

Symposium on Collaborative Systems, 2011.

**) Parshotam K. Crowd computing: a literature review and definition. In Proceedings of the

South African Institute for Computer Scientists and Information Technologists Conference,

2013.

Example of Crowdsourcing-Based CPSS

15

Source: http://www.ecouterre.com/ca-debuts-clothes-hangers-that-display-facebook-likes-in-real-time/

Can’t decide which shirt to get? Let Facebook be the judge. C&A introduced a

high-tech hanger that tallies the number of Facebook “likes” an item of clothing

on its racks receives.

Introduction: Context in CPSSs

16

CPSSs are expected to be context-aware.

Context is any information that can be used to characterize the situation of an entity. An entity is a person, place or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves [Dey et al. 2000].

Context is described as an

ontology-based model specified

for actual settings. Multiple

sources of

data/information/knowledge

provide information about the

actual settings.

Fundamental categories for context information Zimmermann, A., Lorenz, A., Oppermann, R.: An Operational Definition of

Context. In: Kokinov, B. et al. (eds.) CONTEXT 2007. LNAI, vol. 4635, pp.

558–571. Springer-Verlag, Berlin, Heidelberg (2007)

Data, Information, Knowledge, Wisdom

17

Knowledge

Information

Data Symbols, words, numbers, …

Processed data in a context

Information Interpretation

Wisdom

«I know one thing: that I know nothing», Socrates

Applied

Context

Meaning

Raw 4, red

4 wheels

Red Car

Passenger car

has 4 wheels

It is better to stop

the car in this

situation

Adapted from: http://blog.falkayn.com/2011/03/is-knowledge-all-there-is.html

Examples: Data, Information, Knowledge,

Wisdom

18

Source: http://okcon.org/2013/08/14/okcon-2013-guest-post-open-data-and-increasing-the-impact-of-research-its-a-piece-of-cake/

Semantic Web

19

• Increasing number of Internet

pages and services.

• A lot of information is accessible

for the user.

• Different Internet Services has to

understand each other.

Semantic web allows to make possibility of

processing and “understandable” a web data

by machines.

Berners-Lee, Tim; James Hendler; Ora Lassila

(May 17, 2001)

Web Evolution

20

Years

Inte

llig

ence

Настоящее время 1990 гг

Adapted from: https://blog.law.cornell.edu/voxpop/category/legal-semantic-web/

Possible Scenarios of Services and Devices

Interaction in the Internet

21

• Please, order this book for me in the nearest

library.

• Taking into account public transport schedule and

my calendar, please book tickets for football

match.

• Please, advice and order a vine for these dishes in

vine shop (take into account that my wife does not

like Muscat).

• Microwave, please, download from manufacturer

web page heating parameters for these dishes.

Semantic Web Models. Ontology.

22

• XML (eXtensible Markup Language) determines syntax

and semantics of information in the Internet.

• RDF(S) (Resource Description Framework) – language

for recourse description.

• OWL (Ontology Web Language) – ontology language

based on RDF.

An ontology, is a domain vocabulary complete with a set

of precise definitions, or axioms, that constrain the

meanings of the terms sufficiently to enable consistent

interpretation of the data that use that vocabulary.

Графический Пример RDF(S) модели

23

http://www.foo.com/W.Smith http://www.foo.com/S.Smith

rdf: object

William Smith

First Name Last Name

Smith Susan

First Name Last Name

vat: Maried_With

rdf: subject

rdf: predicate

rdf: Statement rdf: type

vat: Holy_Father

vat: confirmedBy

vat: - http://vatican.va/

OWL: Main Concepts

24

Individuals is an elements of classes; properties link them

with each other. E.g., individual Smith can be described

as element of class person.

Two types of properties:

owl:ObjectProperty – object property, that links individuals

owl:DatatypeProperty – data property that links individuals

with values.

OWL Example

25

Megafone

company

Moskow

St.Petersburg

William Smith

Susan Smith

Organization

City

Person

hasMainOffice

hasAdditionalOffice

employ

workTogether

employ

Ontology Matching: Generic Scheme

26

Comparison of elements of two

ontologies using similarity-

based method

Comparison of elements of two

ontologies using semantic-based

distances search method

Graph-based distance

improvement

Comparison of elements of two

ontologies using synonyms

Linguistic

Contextual

Combined

Method class Matching model

S. Balandin, S. Boldyrev, I. J. Oliver, T. Turenko, A.V. Smirnov, N. G. Shilov, A. M. Kashevnik “Method and

apparatus for ontology matching”, US 2012/0078595 A1, March 2012.

Ontology Matching: Similarity-Based

Method

27

Improved FSC

Stemming Conventional Fuzzy string

comparison (FSC)

Normalization

“looking” “look”,

“device” “devic”,

“vertical” “vertic”

Normalization of

estimations Classical Algorithm:

Calculation of substrings of first string

entries to the second string:

“DTSTART” и “START_D”

Count of substrings of first string found in

the second: 16

Overall Count of substrings: 28

Similarity: 57%.

Улучшенный:

FC1 = FuzzyCompare(Element1, Element2)

FC2 = FuzzyCompare(Element2, Element1)

Re’=n*FC1+(1-n)*FC2

Ontology Matching: Semantic-Based

Distance Search Method

Empirically defined weights:

Semantic distance:

28 23.08.2010

Graph-Based Distance Improvement: Idea

Set X=(x1, x2, ..., xn) is the set of subjects and objects in the ontology of two knowledge processors

Set Dx = (d(xi, xj), ...) is a degree of similarity between xi and xj

Set R = (r1, r2, ..., rn) is a set of predicates in the ontology of two knowledge processors

Set Dr = (d(ri, rj), ...) is a set of degrees of similarity between ri and rj

Constant Tr is a threshold value that determines whether two ontology elements mapped to each other or not

29 23.08.2010

Graph-Based Distance Improvement:

Algorithms

For RDF Subjects and Objects:

For RDF Predicates

30 23.08.2010

Smart Space

31

Smart Space is service-oriented infrastructure which

provides possibilities of information sharing by different

devices and meet the following properties:

- Devices have to be integrated in the space or

dynamically join and leave it.

- Devices have to provide personalized user support.

- Devices have to take into account current situation in

smart space.

- Devices have to be adaptive (they have to respond to

other devices and users actions).

- Devices have to provide proactive behavior (provide

the user usefull at the moment services without

explicit query).

Smart-M3 Platform for Smart Space-Based

Application Development

Smart-M3 includes: SIB: Devices and software entities (applications) can publish their

embedded information for other devices and software entities through simple, shared Service Information Brokers.

The interface for managing information in the SIB is provided by Knowledge Processors (KP)

The understandability of information is based on the usage of the common RDF ontology models and common data formats.

September, 2010 32

Smart Space

SIB

Knowledge

base

…

Application

Application

KP

Ontology

KP

Ontology

Smart-M3 allows

user KP to: add,

remove,

change, and

subscribe,

on information in SIB.

Access Control Model For Smart Space

33

Access Control Service

Access Control Rules

Roles

Trust Values Permissions

Public Smart

Space

Private Smart

Space Smart Space Service

User

Context

Information

Request

Information

Request

Private Information

Flow

Private

Information

Flow

Access

Response

Access

Request

Access

Request

Access

Response

Cyber-Physical-Social System Example:

Tourist Assistant – TAIS

34 34

Tourist Attraction

Recommendation

Service

Location

Preferences

Geo2Tag

Search

https://play.google.com/store/apps/details?id=ru.nw.spiiras.tais

Recommended Attractions

Context Service Recommendation

Service

Attraction Information

Service

TAIS: UML Sequence Diagram of Services

Interaction

35

Client App.

Sharing tourist contextinformation (location,

preferences, ...)

Notification aboutchanges in the tourist

context

List of attractions nearby the tourist

Query internal identifier for an attraction and insert new one

Attraction identifier

Query default image

Default image

SS AIS RS Go2TagExternal Sources

Images Database

Context Service

Query attraction list nearby the tourist in radius R

Loop: for each external source

Loop: for each attraction

List of attractions withidentifiers and default

images from internal DB

Notification about accessible list of attractions and region context

Notification about changes in the tourist context

Context of the tourist location region

List of attractions ordered by recommendation service

Notification aboutordered attractions withdefault images accessible

Notification aboutordered attraction

accessible

Query I default images nearby the tourist

Default ImagesList of attractions ordered by recommendation service

with default images

Cyber-Physical-Social System Example:

Home Cleaning Scenario

Home cleaning scenario.

Devices:

User Mobile Device (schedule, preferences)

Robot vacuum cleaner (e.g. Yujin

Robot iClebo Arte or iRobot Roomba)

“Smart home” systems (illumination control, information

network, grid network, etc.)

Manipulating robot (e.g.

FESTO Robotino XT)

36

Home Cleaning:

Two Simplified Scenarios

Robots Interaction Scenario

Two or more robots receive a task to execute actions, e.g. find an

object and bring it to a storage.

Only one robot should handle this task.

Robots should interact to find the one who will bring the object to the

storage.

Pick-and-place scenario

The system solves the task of pick-and-place an object from one

point to another

Two types of robots participate in the system scenario:

- Pipeline robot (can scan object’s characteristics, can provide the

object to the end of pipeline)

- Manipulating robot (can take the object and move it to another

place based on object’s characteristics)

37

38

Lego® Mindstorms EV3 Kit

Large motor

Large

motor

Main block

Medium

motor

Ultrasonic

sensor

Touch

sensor Gyroscopic

sensor

Color

sensor

Lego Mindstorms Robot Example

39

Control block

Gyroscopic sensor

Wi-Fi USB-adapter

Ultrasonic sensor

Large motor

40

Robot’s Interaction in Smart Space Based

on Ontology Matching

Service2

Ontology

Matching Service Smart Space

Ontology Library

Ontology2

Service1

Ontology1

Smart Space

Robot1

Robot2

Smart

Space

Robots Interaction Scenario Live Demo

41

40 cm

30 cm

50 cm

Robot 1:

30 cm = 30 cm => It’s another robot.

40 cm – distance to object.

40 cm < 50 cm => I have to go to object 30 cm

Robot 2:

30 cm = 30 cm => It’s another robot.

50 cm – distance to object.

50 cm > 40 cm => I have to stay here

Cyber-Physical Systems Example: Self-

Organization in Industrial Systems

42

Service Linear Drive

Service Gripper

Interaction / Self-Organization

Ethernet

Ethernet

Unix

Android

Physical Space

Controller

Controller

Smart Space

Wi-Fi

Wi-Fi

Industrial Robots Interaction: Screenshots

43

Industrial Robots Interaction in the Cyber-

Physical System

44

Controller 2 Controller 1

Physical Interaction

Service 1 Service 2

Smart Space

Smart Space

Ontology Library

SSAP SSAP

REST API REST API

Dispatcher 1 Dispatcher 2

Executive

module 2

Pick-and-place robot Manipulating robot

Ontology 2 Ontology 1

STL STL Executive

module 1

Thank you for Attention

Questions are Welcome

45

Alexey Kashevnik, PhD

St. Petersburg, Russia, E-mail: alexey@iias.spb.su