semantic web services for smart devices in a “global understanding environment” () semantic web...

Semantic Web ServicesSemantic Web Servicesfor Smart Devicesfor Smart Devices

in a “Global Understanding Environment”in a “Global Understanding Environment”

((SmartResource))

Semantic Web ServicesSemantic Web Servicesfor Smart Devicesfor Smart Devices

in a “Global Understanding Environment”in a “Global Understanding Environment”

((SmartResource))

Vagan Terziyan

Industrial Ontologies GroupAgora Center, University of Jyväskylä

HCISWWA , November 7, 2003, Catania (Sicily), Italy

http://www.cs.jyu.fi/ai/OntoGroup/index.html

SmartResource: HCISWWA -2003 Presentation 2 of 40

ContentContentContentContent

• Resources in Semantic Web and Beyond

• Global Understanding Environment

• Resource Adaptation

• Remote Diagnostics of Resources

• Resource Maintenance and Networking

MAIN RESEARCH OBJECTIVEMAIN RESEARCH OBJECTIVEMAIN RESEARCH OBJECTIVEMAIN RESEARCH OBJECTIVE

Our intention is to make “resources” (Web documents and services, industrial devices, human experts, etc.) active in a sense that they can analyze their state independently from other systems and applications, initiate and control own maintenance proactively. Resource state can provide knowledge about resource condition, whereas both resource condition and goal of the resource will result in certain behavior of active resource towards effective and predictive maintenance.


Self-maintenanceSelf-maintenanceSelf-maintenanceSelf-maintenance

• Do not expect that someone cares about you, take care yourself even if you are just an industrial device !

• You should be proactiveproactive enough to “realize” that you exist and want to be in a “good shape”;

• You should be sensitivesensitive enough to “feel” your own state and condition;

• You should be smartsmart enough to “understand” that you need some maintenance.


Resource AgentsResource AgentsResource AgentsResource Agents

2. “Yeah, your condition is not good. You need urgent help”

1. “I feel bad, temperature 40, pain in stomach, … Who can advise what to do ? “

3. “Hey, I have some pills for you”

Resource agentsResource agents are intelligent “supplements” of various resources. They represent these resources in Semantic Web-enabled environment and interoperate, realizing resource’s (pro-)active behavior


Industrial ResourcesIndustrial ResourcesIndustrial ResourcesIndustrial Resources

Classes of resources in maintenance systems:

• Device - machines, equipment, etc. • Processing Unit – embedded, local and remote systems,

for monitoring, diagnostics and control over devices• Human (Expert) – users of the system, operators,

maintenance experts


Research ChallengesResearch Challenges

• Resource Adaptation and Interoperability (Semantic Web) Unify data representation for heterogeneous environment Provide basis for communication

• Resource Proactivity (Agent Technology) Design of framework for delivering self-maintained resources to

industrial systems• Resource Interaction (Peer-to-Peer, Web Services technologies)

Design of goal-driven co-operating resources Resource-to-Resource communication models in distributed

environment (in the context of industrial maintenance) Design of communication infrastructure


GUN ConceptGUN ConceptGUN ConceptGUN Concept

Semantic Web: Before GUNSemantic Web: Before GUN

Semantic Web Resources

Semantic Web Applications

Semantic Web applications “understand”, (re)use, share, integrate,

etc. Semantic Web resources

Global Understanding eNvironment

GUN Concept:GUN Concept: All GUN resources “understand” each otherAll GUN resources “understand” each other

Real World objects

OntoAdapters

Real World Object ++ OntoAdapter +

+ OntoShell == GUN ResourceGUN Resource

GUNGUN

OntoShells

Real World objects of new generation(OntoAdapter inside)

First Slice of Gun Architecture

RESOURCE ADAPTATIONRESOURCE ADAPTATIONRESOURCE ADAPTATIONRESOURCE ADAPTATION


TargetsTargetsTargetsTargets

A generic resource-access mechanism (semantic adapter) for devices, diagnostic services and humans

An environment for remote access and resource browsing via semantic-based communication interface

http://www.phpbb.com/mods/

http://images.google.com/imgres?imgurl=www.mincom.cz/images/maintenance.jpg&imgrefurl=http://www.mincom.cz/maintenance.html&h=189&w=200&prev=/images%3Fq%3Dmaintenance%26start%3D40%26svnum%3D100%26hl%3Dru%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26newwindow%3D1%26sa%3DN


Diversity of ResourcesDiversity of ResourcesDiversity of ResourcesDiversity of Resources

GUN (Global Understanding

eNvironment) concept considers

notion of resource in a very general

sense. Types of resources that can

be integrated into GUN are not

limited only to digital documents and

database content. Real-world objects

can be also represented as resources

capable, for example, to accept and

respond to queries, interact with

other resources in order to achieve

own goals.

Generic GUN-resource


Generic Resource AdapterGeneric Resource AdapterGeneric Resource AdapterGeneric Resource Adapter

The integration requires development of the Generic Resource Adapter, which

will provide basic tools for adaptation of the resource to Semantic Environment.

It should have open modular architecture, extendable for support of variety

low- and high-level protocols of the resources and semantic translation modules

specific for every resource (e.g. human, device, database).

Generic Resource Adapter must be configurable for individual resource.

Configuration includes setting up of communication specific parameters,

choosing messaging mechanism, establishing messaging rules for the resource

and providing a semantic description of the resource interface.

GUN-resource

Communication-specific connector of a resource

Resource-specific messaging

Semantic “wrapping” of resource actions; translation of external messages into resource-native formats

Connectivity Layer

Semantic Layer

GUN environment

Generic Adapter

configuration

Messaging Layer


Semantic adapter for DevicesSemantic adapter for DevicesSemantic adapter for DevicesSemantic adapter for Devices

API

Semantic environment

If to consider field devices as data

sources, then information to be

annotated is data from sensors,

control parameters and other data

that presents relevant state of the

device for the maintenance process.

Special piece of device-specific

software (Semantic Adapter) is used

for translation of raw diagnostic data

into standardized maintenance data

based on shared ontology.

Shared ontology

Adapter

Semantic message

Device-specific calls


Semantic adapters for ServicesSemantic adapters for ServicesSemantic adapters for ServicesSemantic adapters for Services

Semantic environment

The purpose of Service Semantic

Adapter is to make service

component semantic web enabled,

allowing communication with service

on semantic level regardless of the

incompatibility on protocol levels,

both low-level (data communication

protocol) and high-level (messaging

rules, message syntax, data

encoding, etc.).

Shared ontology

Adapter

Semantic message

Service-specific calls

http://images.google.com/imgres?imgurl=www.usenix.org/publications/library/proceedings/es99/full_papers/lee/lee_html/img44.gif&imgrefurl=http://www.usenix.org/publications/library/proceedings/es99/full_papers/lee/lee_html/node11.html&h=134&w=207&prev=/images%3Fq%3Dlearning%2Balgorithm%26start%3D160%26svnum%3D10%26hl%3Dfi%26lr%3D%26ie%3DUTF-8%26sa%3DN


Semantic Adapters for Human-expertsSemantic Adapters for Human-expertsSemantic Adapters for Human-expertsSemantic Adapters for Human-experts

Human in the system is an initiator

and coordinator of the resource

maintenance process.

The significant challenge is

development of effective and handy

tools for human interaction with

Semantic Web-based environment.

Human will interact with the

environment via special

communication and semantic adapter. User

interface

Human

GUN-resource

Action translated into

semantic message

Semantic message that

will be visualized

Shared ontology

Second Slice of Gun Architecture

REMOTE DIAGNOSTICSREMOTE DIAGNOSTICSREMOTE DIAGNOSTICSREMOTE DIAGNOSTICS


GoalsGoalsGoalsGoals

Development of remote diagnostic model with

semantic-based communicationexpert (human) and diagnostic (Web) service

with learning capabilities


Device: local platformDevice: local platformDevice: local platformDevice: local platform

Device is a sample of a device, which state is to be automatically annotated with “diagnosis”. It is supplied with Local Platform, which contains Local Alarm Service and History Data Storage.

“History Data Storage”

““Device”Device”

““Local Alarm Local Alarm Service”Service”

Local PlatformLocal Platform

Device state data

Device state data

Remote Diagnostic

Local Alarm Service is a local device-specific algorithm capable to detect alarm states of the Device

History Data is collected by Device via the maintenance ontology for history data representation

http://images.google.com/imgres?imgurl=www.worldcom.com/us/resources/library/dig_library.jpg&imgrefurl=http://www.worldcom.com/us/resources/library/&h=189&w=210&prev=/images%3Fq%3Dlibrary%26start%3D40%26svnum%3D10%26hl%3Dfi%26lr%3D%26ie%3DUTF-8%26sa%3DN

http://images.google.com/imgres?imgurl=www.pvtech.org/images/hyperprimergraph.gif&imgrefurl=http://www.pvtech.org/technology2.htm&h=153&w=216&prev=/images%3Fq%3Ddata%2Bgraph%26start%3D40%26svnum%3D10%26hl%3Dfi%26lr%3D%26ie%3DUTF-8%26sa%3DN


Services (are able to learn)Services (are able to learn)Services (are able to learn)Services (are able to learn)

Learning sample

Learning sample Diagnostic modelLabelled

history data

““Service”Service”

Service is a standalone diagnostic algorithm capable to learn Diagnostic (Classification, Prediction) Model of an expert based on labelled history data about the device state.


http://images.google.com/imgres?imgurl=www.acornstairlifts.com/images/tick.jpg&imgrefurl=http://www.acornstairlifts.com/de/features.htm&h=144&w=144&prev=/images%3Fq%3Dtick%26start%3D60%26svnum%3D10%26hl%3Dfi%26lr%3D%26ie%3DUTF-8%26sa%3DN






Device – Expert : interactionsDevice – Expert : interactionsDevice – Expert : interactionsDevice – Expert : interactions

““Expert”Expert”


Querying diagnostic

Querying diagnostic resultsresults

Labelled data

Labelled data

Watching and querying

diagnostic data

Labelled data

History data

Accepts semantic description of device state and can respond with classification label (semantic description of diagnosis)

Can make semantic query to request device-state data (also labeled history data), get response from Device and provide own label for observed device state

Expert:


Device – Service : interactionsDevice – Service : interactionsDevice – Service : interactionsDevice – Service : interactions

Service presents to a Device possibility to use it as a tool for self-diagnostics.

If classification model has to be built first (no model yet) than perform learning:

Service accepts semantic description of device state from a Device and responds with classification label obtained using existing learned classification model

Request data required for learning using semantic query

Build (via a machine learning technique) a classification model

Notify Device about readiness to perform diagnostics


Device – Service, learningDevice – Service, learningDevice – Service, learningDevice – Service, learning



Querying data for learning

Diagnostic model

Learning sample

Learning sample

Labelled data

History data

Learning process: creation of the Diagnostic

Model


Device – Service, servicingDevice – Service, servicingDevice – Service, servicingDevice – Service, servicing


Querying diagnostic

Querying diagnostic resultsresults

Labelled data

Labelled data


Diagnostic model

History data

Labelled data

Labelled data


System structureSystem structureSystem structureSystem structure

““Expert”Expert”


Labelled data

Labelled data

Diagnostic model

Que

ryin

g di

agno

stic

Que

ryin

g di

agno

stic

resu

ltsre

sults

Labelled data

Labelled data

Wat

chin

g a

nd

qu

eryi

ng

dia

gn

ost

ic d

ataLa

belle

d da

ta

Labe

lled

data

History data


Querying data for

learning

Learning sample and

Learning sample and

Querying diagnostic results

Querying diagnostic results

Simple remote diagnostic model with semantic-based communication, expert and diagnostic service with learning capabilities.

Third Slice of Gun Architecture

MAINTENANCE NETWORKINGMAINTENANCE NETWORKINGMAINTENANCE NETWORKINGMAINTENANCE NETWORKING

http://images.google.com/imgres?imgurl=www.infracast.com/architecture_concept.png&imgrefurl=http://www.infracast.com/products_imb.asp&h=206&w=179&prev=/images%3Fq%3Darchitecture%2B%2Bintegration%26start%3D160%26svnum%3D10%26hl%3Dfi%26lr%3D%26ie%3DUTF-8%26sa%3DN


NetworkingNetworkingNetworkingNetworking



GoalsGoalsGoalsGoals

• Develop network infrastructure for resource maintenance system;

• Support global experience reuse;• Support automated search of potential

partners for services and resources (devices);

• Support collaborative resource diagnostics by multiple services and servicing multiple resources by one service.


P2P networkingP2P networkingP2P networkingP2P networking

- highly scalable

- fault-tolerable

- supports dynamic changes of network structure

- does not need administration Why to interact?

resource summarizes opinions from multiple services

service learns from multiple ”teachers”

one service for multiple similar clients

resources exchange lists of services

services exchange lists of clients


Notice boardsNotice boardsNotice boardsNotice boards

Service 1

Service 2

Service 3

Client 1

Client 2Client 3

Component advertisement solution

Allows search for new partners

Source of new entry points into P2P network

Allows automated search based on semantic profiles


P2P semantic resource discoveryP2P semantic resource discoveryP2P semantic resource discoveryP2P semantic resource discovery

• P2P network formation through Notice Boards;

• Search for necessary partners in P2P network according to their semantic descriptions;

• Establishment of additional P2P links via exchanging addresses between partners;


Discovery: sample scenarioDiscovery: sample scenarioDiscovery: sample scenarioDiscovery: sample scenario

Number of queried peers is restricted due to:• superhub based structure;• query forwarding mechanism based on

analysis of semantic profile;

Resource Service

Matched service

Wrong service

Response

Query propagation


Lear

ning

and

tes

t sa

mpl

e.

Lear

ning

and

tes

t sa

mpl

e.

Que

ryin

g di

agno

stic

res

ults

.

Que

ryin

g di

agno

stic

res

ults

.

Devices: multiple servicesDevices: multiple servicesDevices: multiple servicesDevices: multiple services


““Device”Device”Labelled data

Learning sample

Test sample


Diagnostic model

Diagnostic model

ww11

ww22

ww33

ww44

ww55

Evaluation and Result integration

mechanism

…

Labelled data

Labelled data

Lab

elled d

ataL

abelled

data

Device will support service composition in form of ensembles using own models of service quality estimation. Service composition is made with goal of increasing diagnostic performance.


Services: multiple devicesServices: multiple devicesServices: multiple devicesServices: multiple devices


Diagnostic model

Diagnostic model

…


Labelled data

““Device”Device”““Device”Device”

Labelled data

…


““Device”Device”““Device”Device”

Labelled data

Labelled data

Labelled data

Labelled data

1n

Device-specific diagnostic model

Device Class-specific diagnostic model

Service builds classification model; many techniques are possible, e.g.:

own model for each device

one model from several devices of same type (provide device experience exchange)


Results of NetworkingResults of NetworkingResults of NetworkingResults of Networking

Decentralized environment that integrates • many devices,• many services,• many human experts

and supports :

Establishment of new peer-to-peer links through NoticeBoards, advertisement mechanism

Semantic based discovery of necessary network components

Service

Interaction ”One service – many devices”

Interaction ”One device – many services”

Exchange of contact listsbetween neigbor peers








Device-to-Device “opinion” exchangeDevice-to-Device “opinion” exchangeDevice-to-Device “opinion” exchangeDevice-to-Device “opinion” exchange

Device

Device 1Device 2

Service 1

Service 2

trust =

100

trus

t = 2

6

1

?

?

4

8

Device will be able to derive service

quality estimates basing on analysis

of ”opinions” of other devices and

trust to them.

Service quality

evaluations




Service-to- Service “model” Service-to- Service “model” exchange and integrationexchange and integration

Service-to- Service “model” Service-to- Service “model” exchange and integrationexchange and integration

Diagnostic models exchange

Diagnostic models integration entails creation of a more complex model extension or a service with new diagnostic model


CertificationCertificationCertificationCertification

53

4

Certifying party

Device

Service 1

Service 2

Service 3

612

Own evaluations

Support for certification authorities

in the network. Certificates gained by

services will be used by devices for

optimal service search and selection.

Device makes its decision taking into

account also its own service quality

evaluations.

trust

http://images.google.com/imgres?imgurl=library.med.nyu.edu/library/libinfo/classes/house/Certificate.jpg&imgrefurl=http://library.med.nyu.edu/library/libinfo/classes/house/HouseStaffFinal_index.html&h=565&w=616&prev=/images%3Fq%3Dcertificate%26start%3D40%26svnum%3D50%26hl%3Den%26lr%3Dlang_en%7Clang_fi%7Clang_ru%26ie%3DUTF-8%26oe%3DUTF-8%26newwindow%3D1%26sa%3DN





http://images.google.com/imgres?imgurl=www.state.ma.us/itd/spg/publications/bulletins/ITBulletinSummer2002/html/certificate.jpg&imgrefurl=http://www.state.ma.us/itd/spg/publications/bulletins/ITBulletinSummer2002/html/corpdiv.html&h=211&w=190&prev=/images%3Fq%3Dcertificate%26start%3D120%26svnum%3D50%26hl%3Den%26lr%3Dlang_en%7Clang_fi%7Clang_ru%26ie%3DUTF-8%26oe%3DUTF-8%26newwindow%3D1%26sa%3DN


Maintenance “executive” servicesMaintenance “executive” servicesMaintenance “executive” servicesMaintenance “executive” services

Device

Service

Control

Support for maintenance services that

can influence on device state and

perform maintenance actions upon it

(automated control system, maintenance

personnel).

They complete the minimal working set

of maintenance system components.

datadia

gn

osi

s

control



Business ModelsBusiness ModelsBusiness ModelsBusiness Models

Certifying party

Device

Service

Noticeboard owner

?New players are

possible

1-day advertisement =

300 €

certification = 3000 €

service cost =

10€/hour

1000 new service

addresses = 40€

opinion cost = 80€

expert support = 40€/hour

service teaching =

45€/min

search service = 80€/item

platform

hosting =

5€/day

platform

package =

3000€


Concluding RemarkConcluding RemarkConcluding RemarkConcluding Remark

• Among recent initiatives aimed at development of adoption of open information standards for operations and maintenance and implementation of interoperable cooperative industrial environments are:

• MIMOSA (Machinery Information Management Open System Alliance)[1]. The project consortium pretends to build an open, industry-built, robust Enterprise Application Integration and condition-based maintenance specifications.

• PROTEUS[2], funded by industrial companies and led with a goal to develop a generic maintenance-oriented platform for industry.

• These initiatives are very expensive, labor and resource consuming, and still does not attempt to apply and benefit from the Semantic Web technology. We believe however that without comprehensive metadata description framework, ontologies and open knowledge/semantics representation standards their results will be just next consortium-wide standards, rather than comprehensive, flexible and extensible framework.

•[1] http://www.mimosa.org/

• [2] http://www.proteus-iteaproject.com/

semantic web services for smart devices in a “global understanding environment” () semantic web...

Documents

resource condition

resource state

resource communication

resources web documents

maintenance systems

contentcontent resources

networking slide

industrial devices