DomoML - an Integrating Devices Framework for Ambient Intelligence Solutions

Lorenzo SommarugaSemantic and Multimedia Systems

Laboratory (LSMS)

University of Applied Sciences and Arts of Southern Switzerland (SUPSI)

6928 Manno, Switzerland

+41 (0)58 666 6531

lorenzo.sommaruga@supsi.ch

Tiziana FormilliSemantic and Multimedia Systems

Laboratory (LSMS)

University of Applied Sciences and Arts of Southern Switzerland (SUPSI)

6928 Manno, Switzerland

+41 (0)58 666 6534

tiziana.formilli@supsi.ch

Nicola RizzoSemantic and Multimedia Systems

Laboratory (LSMS)

University of Applied Sciences and Arts of Southern Switzerland (SUPSI)

6928 Manno, Switzerland

+41 (0)58 666 6534

nicola.rizzo@supsi.ch

ABSTRACTWithin an ambient intelligence context, this paper presents an approach to Human Home Interaction through the adoption of the DomoML web service based framework. This framework consists of a suite of XML based languages aiming to describe and interconnect home resources in order to obtain a smart home appliance of heterogeneous integrated devices. The DomoML framework is integrating and managing different devices, by abstracting them as homogeneous resources. It is based on the use of semantic web technologies, on one hand, which provide interoperability and allow the automated processing and reasoning; and, on the other hand, on the use of RESTful web-services as a mechanism of integration and communication between heterogeneous physical devices. This platform is going to be applied in elderly people's home to enhance quality of life regarding independence of daily living and safety. The application of the framework to this domain aims to provide innovative solutions and ICT services to answer the social and health-care needs of elderly people at sustainable costs.

Categories and Subject DescriptorsD.2.12 [Interoperability]: Distributed objects, Interface definition languages.

General TermsAlgorithms, Experimentation, Standardization, Languages.

KeywordsWeb Services, REST, devices as services, integration, DomoML, ontology, Human Home Interaction, Ambient Intelligence.

1. INTRODUCTIONService Oriented Architectures and Ambient Intelligence are complementary areas of research that can bring intelligence to our

everyday environments, making those environments sensitive and responsive to people by exposing functionalities as interconnectable services. This work focuses on the application of web service technologies to a smart home domain. Ambient Intelligence [1] is an interdisciplinary paradigm that is based on a number of pre-existing research areas, including: pervasive computing and smart objects, i.e., the growing trend towards embedding microprocessors in everyday objects; Artificial Intelligence, including learning algorithms, pattern matching, speech recognition, gesture classification, etc.; context awareness, i.e., to track and position objects and people and represent their interactions with their environments; domotics, i.e., home and building automation; human computer interaction with Human Centric interfaces, Multi-modal interaction, etc.

One basic requirement of an Ambient Intelligence system is the smart integration of many different and heterogeneous devices, which provide sensing and actuating operations on the environment.

Within a specific domain, such as a domestic environment, technology has become more and more pervasive in a great number of household appliances. These have achieved an important role both in the ordinary and in the entertaining daily activities: assisting people during cooking, washing, phone calls, house supervising, etc. On the other hand, their operations are in most cases individual and not connectable actions. Different control systems operate under the human direction, and can hardly “speak” to each other. The smart integration of functionalities is considered as an intelligent task, which can provide intelligent behavior to objects and to the environment.

People expect smart behaviors from their electronic household appliances, questioning: “Can these household appliances interact each other and provide some benefit?” Or, more specifically, for example: “Could I easily delay the dish washer starting after the washing machine ending, when I am watching my favorite movie at TV?”

Unfortunately there is no universal solution able to exchange information and connect household appliances. The communication between them is sometimes possible for devices built by the same constructor, or through a compatible bus technology (e.g. KNX).

More in general, another solution could be considering device operations as interoperable web services, where the orchestration (e.g. WS-BPEL) or choreography (e.g. WS-CDL) of individual

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.WEWST '11, September 14 2011, Lugano, SwitzerlandCopyright 2011 ACM 978-1-4503-0748-2/11/09…$10.00.

capabilities is sometimes accomplished in a crispy way, by symbolic sequencing of service operations, usually described in an XML vocabulary.

A number of attempts concerning the development of a device integration middleware, in the past, have been [2][3][4][10], also applied to smart home [5]. There exist some similarities with our approach, such as the flow executor described in [3], the application and device elements of the Hydra SOAP-based software architecture [4], and the Complex Event Processing in [10].

Our solution is focusing more on semantic integrability and provides a full seamless simple semantic integration of any device, both real and virtual ones (see Section 2.2). Another distinctive feature of our approach is that it fully exposes in REST and RDF/XML not only operational device functions, but also scenario and device management functions, which allows a dynamic behaviour setting to be handled. In fact, the system configuration can be changed on the fly by adding new devices and or rules through REST messages. Moreover, it is an applied research software system which has been realized for an ambient intelligent solution in a real environment.

Another interesting project based on the idea of making a universal language to unify the needs and understanding of the home environment was the first DomoML work [6]. In this project, XML and semantic web technologies were used to achieve an infrastructural support for resource interoperability and integration.

The new DomoML is a revised version of the previous human home interaction framework consisting of a suite of XML based languages, which are aiming to describe and interconnect home resources. These include household appliances, walls, ceiling, doors, etc. and their functions, such as switch-on/off, dimmering, get-temperature, etc. An extensible OWL ontology has been defined to represent these elements and their properties, and is completed by a standard communication message protocol among resources [7] (see also Section 2.). The DomoML framework covers structural, functional and communication aspects. Therefore it not only deals with the interoperability issues, but captures also semantic aspects of the environment and its changes, through properties that can be used to create intelligent behaviors. A controlling mechanism is necessary to govern the interaction among devices and to direct their choreography.

The rest of the paper introduces the new DomoML framework and the revised DomoML languages (Section 2.). It shortly presents the resource-oriented architectural style of the DomoML framework architecture, consisting of a core (Section 2.1.), and clients (Section 2.2.). It summarizes how to obtain a smart home appliance of heterogeneous integrated devices, with the description of some scenarios for elderly people (Section 2.1.1.); the adopted RESTful approach is presented in Section 3. It finally summarizes the current implementation status (Section 4.).

2. DOMOML: A FRAMEWORK FOR INTEGRATING DEVICES

The DomoML framework is a software tool consisting of three main components: the DomoML Core, the DomoML Client and the Reasoner. Figure 1 outlines a simplified schema of how

the DomoML framework is integrating and managing different devices, by abstracting them as homogeneous resources.At the bottom of the picture some different real devices of a home environment are represented. Each one of them needs to be described using the RDF/XML formalism on the basis of the OWL DomoML ontology; hence, specific software drivers can be developed for interfacing the corresponding real world sensing and acting commands to the standard DomoML abstraction. This is the basic step for enabling integration and interoperability among them. Once integrated, any heterogeneous device (dev1, dev2, devN) is represented within the framework as an homogeneous resource (res1, res2, resN). The next step is to control and manage their interconnections providing intelligent behaviours. This task is carried out by the DomoML core by means of a scenario manager.

In the currently implemented DomoML suite, the scenarios are described in XML through simple conditional rules with antecedents and consequents, as presented in the example of section 2.1.

Figure 1. DomoML framework architecture

2.1 The DomoML Core and ReasonerDomoML is a software framework which provides an

abstraction layer on top of devices and permits their interoperability. It considers three XML-based languages which capture the description, communication and interaction of home resources:

Device Manager

Controller

Scenario Manager

View

Reasoner

DomoMLClient

Driver

DomoMLClient

Driver

dev1 dev2 devn...

REST ws

res1 res2 resn...

DomoML Core

Model

Ontology

DomoML Framework

layer

Middleware layer

Real world layer

2 – The DomoML Core activates its Scenario Manager that sends a switch-on DomoML-com message to the DomoML LonWorks Client. This message is:

<Lamp rdf:ID="tableLamp"> <Function> <Name>state</Name> <Value>on</Value> </Function></Lamp>

Scenario Manager has a reasoning mechanism that triggers the execution of scenarios when some conditions are verified. This is accomplished by a Reasoner, which is evaluating simple XML conditional rules. Currently, for performance reasons, due to the real application environment based on limited hardware resources for the DomoML Core (an all-in-one panel PC), semantic web inference mechanisms and rules standard formalism, have not been adopted.

In the case of the example, the scenario is described by rules with the following syntax:

<Rule ID="LightOnWii"> <label>Turn lamp with wii remote</label> <Conditions> <hasCondition> <Button rdf:ID="wiiId"> <State> <Name>toggle</Name> <Value>true</Value> </State> </Button> </hasCondition> <hasCondition> <Lamp rdf:ID="tableLamp"> <State> <Name>state</Name> <Value>off</Value> </State> </Lamp> </hasCondition> </Conditions> <Actions> <hasAction> <Lamp rdf:ID="tableLamp"> <Function> <Name>state</Name> <Value>on</Value> </Function> </Lamp> </hasAction> </Actions></Rule>

In general, useful scenarios can be easily described through XML conditional rules with antecedents (<Conditions>) and consequents (<Actions>), allowing a controlled interconnection among devices.

2.1.1 Current developments of DomoML and its ApplicationsAfter a first prototype development aiming at concept proofing [9], another more complete framework prototype has been recently developed to test the effectiveness of the DomoML approach in a real case. The initial ontology has been revised and

extended in order to cover new device types, such as remote control, or other domain specific devices etc. Simple scenarios have been initially defined and tested in a real environment. to simulate situations of device interoperability. Within the domain of health-care needs of elderly people, a number of possible useful scenario examples are: “a lamp in the bedroom should light at low brightness during night if a presence sensor detects a person has woken up, enlightening the path to the toilet or the kitchen”, “a voice reminder could alert elderly person about routine activities, as taking medicine”, “a sms could be automatically sent to caregivers in case of emergency, e.g. if the elderly person has fallen.”

The prototype architecture developed for this system is composed of two main elements: the DomoML Core (server) and the DomoML clients, i.e., the various devices installed into the environment. The main server role is to register and coordinate resources (household appliances, etc.) of the domestic environment using their RDF/XML descriptions. Another important job of the server is to dispatch messages coming from the devices. In this architecture server and clients interaction is based on the DomoML core communication mechanism, which is open to interface any environment resource through middleware layers with specific software driver implementation.

The DomoML Framework and, in particular, its ontology is demonstrating its usefulness in many different contexts. In particular, it is currently used in two projects.

One of them is SUPSI's Granno on-going applied research project, carried on in collaboration with Lugano Municipality and Tessin Society for Blind Assistance (STAC - Società Ticinese per l'assistenza dei Ciechi). This project aims to provide innovative solutions and ICT services to answer the social and health-care needs of elderly people at sustainable costs. ICT and domotic devices are installed at elderly nursing homes and people houses to improve their quality of life. In this context the DomoML ontology is used to allow the different devices to interoperate, monitor and alert critical situations or states both of the environment and of the human living in it. The ontology has been extended to include new devices and structural object typologies to meet the specific project needs. This project represents a test-case for the ontology in a real-application context that is useful to consolidate the DomoML Framework. An example scenario in this case could be sending an alarm message to assisting organisations or to parents alerting that the person has fall down and is inactive. This condition is detected by monitoring accelerations, physiological parameters (e.g. heart rate), and movements.

Another application of DomoML is carried on in the area of Ambient Intelligence within the Sensing Kit project of the interaction design laboratory (IDLab) at SUPSI. This project focuses on the design and development of non-obtrusive interface solutions for human home interactions. The aim of the project is to provide interface solutions able to support and facilitate the sharing of human activities by using physical and/or digital feedbacks. In this project the DomoML ontology will be extended to include the end user in the ontology classification in addition to the devices.

2.2 The DomoML ClientThe DomoML Client handles the interactions between a

device and the Core, by wrapping the device into a DomoML client.

It is the only module knowing about the low-level driver (figures 1 and 4) and its details; it also exposes the device functions as URIs, executes these functions after an invocation from the Core and notifies about state changes, following a RESTful approach.Real and virtual devicesIn the DomoML Framework there are two device types: real and virtual.

• A real device is an object in the real world. It can be a sensor, an actuator or both: a switch, a lamp, a computer, an accelerometer, a led on a more complex device, etc.

• A virtual device is essentially an abstract entity acting as a real object, such as a html button, a slider, etc. It can also be a software program running anywhere (e.g. Slideshare) or a complex widget showing graphs and charts.

Figure 4. DomoML Clients

The Core doesn't have to know if a device is real or virtual because the DomoML Client “wraps” it, hiding its nature: so virtual and real devices can seamlessly interact and are fully interchangeable in rules and scenarios: we could turn on the dish washer using a real switch or a touch interface on a mobile phone

through a remote connection or even with a voice command and receive alerts, notifications and feedback in the same way.

Writing a new Client has been made easy, through a Client interface which implements the description of a new device for the framework; it is also possible to make a “lightweight” client for a virtual device using only a web browser and some scripts: for example we made virtual buttons and, just combining these buttons, a complete home automation control panel using html and javascript: this interface is usable in any device capable of doing ajax calls to a server to send commands and receive notifications.A lightweight client (a virtual device) can be quickly written by the average web developer without the knowledge of other technologies as just a few lines of code in a web page which can run everywhere.

In the near future we expect to create an Open Source community where developers can make their own Clients to enrich the Framework capabilities and the range of devices DomoML can support.

3. The RESTful integration of devicesIn the current DomoML architecture, the communication

between device drivers and DomoML Core is based on RESTful web-services. Initially DomoML was a SOAP based architecture, and it is has been now extended to support REST calls, similarly to other middleware platforms for heterogenous device integration [10]. This approach has the advantage to represent each device as an independent resource in a very flexible way. DomoML Framework not only supports RDF/XML format but also JSON especially for visualization purpose via javascript and web pages.Any device can be represented via a RDF/XML file, which contains the description of its functionalities exposed as web service operations: so it's easy to bind heterogeneous objects with direct connections or with a complex set of rules executed by the Reasoner.

There are various possible connections among the real and virtual devices: real to real, virtual to real, real to virtual, or virtual to virtual. Some examples of them are outlined in figure 5.

The simplest case is a binding between real devices: the black lines are connecting a remote controller, e.g., using bluetooth for the data transfer, and a lamp: there are different protocols, drivers and DomoML Clients, which can communicate via REST.

In addition, it is straightforward to expose a device state or even call a device function with a html/js client: in this way any device capable of displaying an html page – laptop/desktop, smartphones, tablets, etc. - could contain a virtual device to control a real one, as shown in figure 5 (red connectors): a “virtual controller” running on a touch-enabled device triggers a simple rule that turns on a real lamp.In the same manner the opposite connection becomes possible too: a real sensor embedded in a t-shirt sends its data to the DomoML Core and the Core itself dispatches that data to a virtual device showing the user's ECG in real-time (green connectors).Another example is the use of a wiimote to monitor accelerations and to provide handy command buttons: indeed this device contains a 3 axis accelerometer which provides data to determine if the user (the elderly person in our case study) has fallen, or it can be used as a remote command to switch on light. The

accelerometer should be substituted in the future with another low obtrusive device, like a watch or a wearable object.Again, we used a balance board to show the user's weight on a on-screen widget; it could also be possible to observe the barycenter position and analyze the user's posture with this device.In the fourth case, there is a connection between two virtual devices (blue connectors), for example from a virtual controller to a software, such as a presentation on SlideShare: the controller can be a touch interface running in a mobile phone, the actuator is a web application using the SlideShare APIs to change the current slide.

It would be simple to modify this scenario in order to substitute the remote controller with a real one, because – as mentioned in section 2.2 – the DomoML Core and the Reasoner do not know anything about the devices.

It is enough to modify the rule block

<Conditions><hasCondition>...</hasCondition></Conditions>

leaving the action part unchanged

<Actions>...</Actions>

This shows the powerful feature of the DomoML framework for easily and simply combine the integration of various devices in order to coordinate ambient intelligent solutions.

Figure 5. Integration of heterogeneous devices

4. Implementation

A first prototype of DomoML Framework has been developed in Java (jdk 1.6.0_24) on Linux platform. It has been tested both on Tomcat 7 and on Jetty 8. Some tests have been done on Windows 7 too.

By now, a number of devices typically useful for a elderly people apartment (above 30 devices, with about 100 state/function operations) have been successfully integrated and tested in the system. They are fully interconnected and can interact each other, including: LonWorks power line communication light switches and blind actuators, ZigBee presence sensor, Wii Remote, Wii Balance Board, an ecg t-shirt (measuring heart rate, breathing and body accelerations), a compact Android smartphone (xperia X10 mini, for voice commands). Further implementation is in progress in order to include other domotics system (e.g. BT MyHome) and to consolidate the framework performances and stability.

5. ConclusionsThis paper introduces the DomoML framework, which offers a web service based ambient intelligence solution for the integration of heterogeneous devices and more generally of any resource involved in human home interaction. Main benefits are the easiness and flexibility in defining and integrating new devices into a coordinated world of other devices. This is obtained thanks to ontology based RDF/XML representations, a non-proprietary open protocol and a simple RESTful operation exposure. The new REST approach provides lighter clients compared to the previous SOAP implementation. In addition, the reasoning mechanism adopted for scenarios and rules execution is also simplified, and it is envisaged to support semantic web rules representations or other inference mechanisms. Some testings have already been carried out in order to prove the beneficial usage in real world use cases, such as in the above mentioned ongoing applied research Granno project, where monitoring and controlling home environment and elderly people under critical situations is considered. Further evaluation will be carried out after the complete installation in all the 20 apartments foreseen for elderly people.

6. REFERENCES[1] H. Aghajan, R. Lopez-Cozar Delgado, J. C. Augusto

“Human-Centric Interfaces for Ambient Intelligence” Elsevier, 2010.

[2] Qixin Wang, Wook Shin, Xue Liu, Zheng Zeng, Cham Oh, Bedoor K. AlShebli, Marco Caccamo, Carl A. Gunter, Elsa Gunter, Jennifer Hou, Karrie Karahalios, Lui Sha, “I-Living: An Open System Architecture for Assisted Living”, in IEEE International Conference on Systems, Man, and Cybernetics, 2006.

[3] Fahim Kawsar, Gerd Kortuem, Bashar Altakrouri, “Supporting Interaction with the Internet of Things across Objects, Time and Space”, Internet of Things (IOT), 1-8, DOI: 10.1109/IOT.2010.5678441, 2010.

[4] The Hydra project, http://www.hydramiddleware.eu

[5] Kim, Jin; Choi, Hyeok-soo; Wang, Hui; Agoulmine, Nazim; Deerv, M. Jamal; Hong, James Won-Ki; "POSTECH's U-Health Smart Home for elderly monitoring and support",

World of Wireless Mobile and Multimedia Networks (WoWMoM), IEEE International Symposium, 2010.

[6] F. Furfari, L. Sommaruga, C. Soria, and R. Fresco, “DomoML: the definition of a standard markup for interoperability of Human Home Interactions,” EUSAI 2004, Second European Symposium on Ambient Intelligence, Eindhoven - NL, November 2004.

[7] L. Sommaruga, A. Perri, F. Furfari, “DomoML-env: an ontology for Human Home Interaction”, In: Proceedings of SWAP 2005, the 2nd Italian Semantic Web Workshop, Trento, Italy, December 14-16, 2005, CEUR Workshop Proceedings (2005).

[8] "LonBridge Server User's Guide", Echelon Corporationhttp://www.echelon.com/support/documentation/manuals/devtools/078-0386-01A_LonBridge_Server_UG.pdf

[9] A. Perri “Ricerca e definizione di un linguaggio standard per l’interazione fra Uomo e Casa”, Thesis of Master in Advanced Computer Science (MACS), SUPSI-DTI, 2005.

[10] Daniel Romero, Gabriel Hermosillo, Amirhosein Taherkordi, Russel Nzekwa, Romain Rouvoy, Frank Eliassen, “RESTful Integration of Heterogeneous Devices in Pervasive Environments”, 10th IFIP International Conference on Distributed Applications and Interoperable Systems (DAIS'10), Amsterdam, Netherlands, 2010.