Editor: James HendlerUniversity of Marylandhendler@cs.umd.edu

T h e S e m a n t i c W e b

68 1094-7167/03/$17.00 © 2003 IEEE IEEE INTELLIGENT SYSTEMS

application-rich environments where we live and work.Let’s consider the ritual of visiting a lab or a business unitto deliver a presentation and collaborate with colleagues.We exchange printed business cards that end up in adesk drawer because we never have the time to enterthem into our personal information manager. We mustdeal with capricious projectors that might not work asintended while we make a presentation. We promise toemail a soft copy of our presentation when we get back toour office or follow up with a new scheduled meeting. Wesearch for addresses and directions to a restaurant or theairport and print them on paper to take with us. Perform-ing each of these simple tasks entails a complex choreog-raphy of interactions with a variety of devices and applica-tions, and manual cutting and pasting, resulting in time-consuming, error-prone, frustrating processes.

Fujitsu Laboratories of America and the MINDSWAP

research group at the University of Maryland have createdan environment operating in a conference room where allthe tasks just described and more can be done with just afew simple point-and-click operations. More than 30 typesof services such as “view on projector,” “print,” “map,” and“play (video)” are in the conference room and on the localnetwork—services that you can use and manipulate inmany ways. For example, you can easily view and controlyour local presentation file on any viewing service in theroom (for example, “view on projector”) with only mouseclicks through your Web browser. If you install our soft-ware environment on your computer, you can do even morepowerful things with your computer’s local services com-bined with services available pervasively and remotely.

Our research aims to empower nonexpert users with the ability to perform complex tasks in information-rich,device-rich, and service-rich environments using devices(limited in Graphical User Interface real estate, computa-

tional power, and so on). Our approach is to expose thefunctionality in such environments (device functionalityor third-party functionality) as Semantic Web services,which in turn the user can discover and arbitrarily com-pose. We call this approach task computing, which wedefine as computation to fill the gap between the tasks thatusers want to perform and the services that constituteavailable actionable functionality.1

STEER in TCETo support task computing, we have implemented a Task

Computing Environment including client environment,service discovery mechanism, and Semantic Web servicesand tools. TCE is composed of several components includ-ing STEER (Semantic Task Execution EditoR), White Hole,and PIPE (Pervasive Instance Provision Environment)shown in Figure 1.

STEER is a Task Computing client, which has a Webpage interface (shown here) as well as a Java interface.STEER’S Compose page is shown in the Web browser. Thepossible service compositions are categorized based ontheir semantic inputs and outputs. In each pair, any servicein the right-side drop-down menu can be combined withany service on the left-side drop-down menu. The userchooses the service from the drop-down menu and exe-cutes the composition. The user clicks the construct buttonto create a more complex composition starting from thetwo-service composition in this Compose page. The swirlon the upper right corner is the “White Hole.” The usercan drag and drop objects such as files, URLs, contacts,and schedules from PIM to create services that providesemantic versions of the objects dropped into it. On thebottom is a service management GUI by PIPE. Throughthis GUI, the user can make the services local or pervasiveor temporarily hold the services for possible future use.The user can also control how services are provided suchas expiration time, and so on.

STEER lets fairly unsophisticated users find, select, com-pose, and execute Web services to achieve common tasks.While STEER typically has several built-in or local servicesplus a relevant set of remote services on the Web, it

Information technology’s rapid evolution has made

tremendous amounts of information and services

available at our fingertips. However, we still face the frus-

tration of trying to do simple things in the device- and

Ontology-Enabled PervasiveComputing Applications

Ryusuke Masuoka and Yannis Labrou, Fujitsu Laboratories of AmericaBijan Parsia and Evren Sirin, MIND Lab, University of Maryland

Published by the IEEE Computer Society

becomes especially interesting when it canwork with a rich, varied set of pervasiveservices (many of them on devices andperipherals).

By exposing the various functionalitiesas Web services and advertising them viaUniversal Plug and Play (UPnP, www.upnp.org), STEER clients on portable devicesdynamically discover the services availablein the room where they are physically pres-ent. Tying service discovery to physicalpresence both filters the set of services to amanageable level (after all, one generallydoesn’t care about projectors in rooms onanother floor) and suggests some orga-nizing principles for those services. Theroom’s purpose largely determines the sortsof tasks facing users when in the room. Theactual devices installed further constrain theset of possible and probable tasks. Finally,the Web services’ semantic descriptionsprovided by the devices permit STEER toautomatically combine services into likelycompositions for the end user to consider,select, extend, and perform. STEER bridgesthe gap between what users want to do andwhat they can do by organizing the unwieldymass of services into likely courses of usefulaction. Moreover, STEER helps users moreclearly understand what they need and wantto do by making likely and common compo-sitions obvious to users.

So you want to do what …with Web services?

The type of Web service compositionthat STEER supports is a simple form of pro-gramming, and programming, even a sim-ple form, is not commonly done nor donewell by end users. While a simple macrorecorder can follow users’ actions in per-forming a task and play those actions backat a later time, such recording requiresusers to already know how to perform thetask. Task computing presumes that userstend to not know in advance how to achievetheir goals or even what those goals are.STEER gives users some starter composi-tions and helps guide them in extendingthem. This guidance takes two fundamentalforms: documentation for end users andsuggested compositions and compositionsteps from STEER. When building a compo-sition, relatively few services are compati-ble with the prior steps. STEER shows usersonly the sensible next steps, plus the asso-ciated documentation. Faced with limitedoptions plus more detailed explanation as

desired, users generally have an easy timechoosing what to do. To support user-guided composition, STEER must find suffi-ciently rich descriptions of the services.

Most Web services are described by WebService Description Language documents.2

WSDL is a nice, simple Interface Descrip-tion Language well suited for developersand developer-class tools. WSDL is easy tointegrate with popular programming lan-guages such as C#, Java, and Perl, and is nomore difficult to read, understand, or usethan the typical module system. Indeed,given its simplicity, it’s far easier than most.If you have a clear idea of what you aretrying to accomplish and a good grasp ofprogramming, WSDL makes composingand invoking Web services trivial. WhatWSDL doesn’t provide is extensive, struc-tured documentation of the service, nordoes it easily support automated composi-tion. Universal Description, Discovery andIntegration of Web Services (www.uddi.org)tries to cover some of this gap, but it targetsdevelopers, not end users. DAML-S3 and its

successor, OWL-S, supply ontologies fordescribing Web services so that they mightbe discovered, explained, composed, and exe-cuted. OWL, the Web Ontology Language,4

extends the Resource Description Languagewith powerful modeling constructs suffi-cient for naturally describing many subjectdomains. Built on the metadata facilities ofResource Description Framework (RDF),OWL effectively describes all manner ofWeb resources for both human beings andprograms. This is, of course, exactly whatwe needed for STEER.

Although OWL-S provides the meansfor describing Web services, it doesn’t dis-pense with WSDL. WSDL remains the basicdescription of Web services. OWL-S con-nects to WSDL by grounding its descrip-tions in the lower-level WSDL descriptions.Roughly, WSDL provides information onhow to invoke a Web service and OWL-Slets us say what the service does, why wemight want to use it, and what to expect ifwe use it, among other things. Even thoughthere is an expressive gap, OWL-S makes

SEPTEMBER/OCTOBER 2003 computer.org/intelligent 69

Figure 1. Screenshot of the Task Computing Environment client desktop.

good use of what WSDL offers. OWL-Sdescriptions include a grounding specifica-tion that defines how an OWL-S service isbound to a WSDL service so the informa-tion in WSDL can be used for invocation.

(This relationship should get even smootherwith WSDL 1.2, which will contain a canon-ical mapping into RDF and OWL.)

But there’s still a gap between a WSDLdescription and corresponding OWL-S

descriptions. We often exploit that gap topresent the base Web service functionalityin many ways. But the gap still needs filling.In particular, WSDL documents tend todescribe input and output parameter types interms of XML Schema. While certainlyadequate for programmers, and althoughOWL is perfectly capable of handling XMLSchema types, they aren’t the best things fornaïve end users and they aren’t the easiestthings to automatically compose. As anexample, a WSDL description states that aservice accepts a floating number—not veryuseful information. But mapping this type toa price concept, which is defined in an OWLontology and linked to other concepts suchas currency, dollars, and so on, lets users andsoftware agents better interpret the meaning.

WSDL’s extraordinary tool support is oneof its compelling features. In many IDEs,WSDL generation and consumption is asimple touch of a button. For the STEER user,OWL-S descriptions are even more trans-parent. But if we are to expect Web serviceproviders or third-party Semantic Web ser-vice description providers to add the extramarkup that OWL-S permits, we must givethem tools that make the job feasible.

OntoLinkOntoLink lets developers generate OWL-

S descriptions from WSDL annotations(see Figure 2).The OntoLink tool’s mainfunction is to ground the semantic servicedescribed in an OWL-S description to itscorresponding Web Service implementa-tion. The mapping between XML Schematypes and OWL ontologies is defined andsaved in OWL-S’s grounding information.

When a WSDL file is loaded, the pro-gram automatically generates a simplesemantic description template by using theinformation in WSDL, such as text descrip-tions, parameter names and types, and soon. A simple point-and-click interface letsusers extend this template by definingmappings between the XML Schema typesand OWL concepts. For example, a WSDLdescription could define a parameter to be a string of five digits. Users select theAddress concept from the class hierarchy,link the parameter to the ZipCode property,and state that the Country property ofAddress should equal USA (italic wordsrepresent concepts defined in OWL ontolo-gies).The transformation functions gener-ated here are then stored as XSLT scriptsand put inside the grounding specification

70 computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Figure 3. OntoLink for semantic object mapping. OntoLink defines a transformationfunction between two concepts from different ontologies. This mapping informationis described as a Web service and used in compositions to connect two otherwiseincompatible services.

Figure 2. OntoLink is used to generate an OWL-S description from a WSDL annotationfor grounding.

of the OWL-S description. Easily creatingsemantic descriptions for WSDL servicesmakes it possible for STEER to access andinvoke Web services on the Internet. As aresult, STEER provides a richer environmentwhere users can combine pervasive ser-vices with the ones available on the Web.

So far, we’ve glossed over the problem ofservice inputs and outputs belonging todifferent ontologies. In fact, we definitelydo not expect the world of specific perva-sive environments to be accounted for by asingle universal ontology—we consider thisidea unrealistic. One large, common ontol-ogy is difficult to create, agree on, and man-age. Our idea is to start with small ontolo-gies for specific domains and purposes:Only when interoperability between thosedomains and purposes becomes necessaryshould we create mappings between (only)necessary parts of them. Mapping doesn’thave to be complete or exact; it is goodenough as long as the mapping serves thepurpose reasonably.

You can also use the OntoLink tool todefine these mappings between ontologyelements (see Figure 3). Using the sameinterface as in the XML Schema-OWLmapping, we can define mappings such asa Name property of a Person conceptdefined in ontology A is the concatenationof FirstName and LastName properties thatare defined in ontology B. Even arithmeticcomputations between ontology elements,such as converting the value of the Lengthproperty from Inches to Meters, can beintegrated into these transformations, asXSLT provides quite expressive functions.

In our framework, the semantic objectmapping or transformation functionsdefined between ontology elements arealso published as Web services. We cancompose two services that use disjointontologies by simply adding the appropri-ate transformation service in between.This approach lets us handle the ontologymapping in a simple and consistent sys-tem. STEER integrates the transformation

services with the compositions automati-cally as needed so the user is presentedwith a set of services that can be seamlesslycomposed together.

Figure 4 shows how STEER actually usessemantic object mappings and ground-ings by OWL-S descriptions generated by OntoLink. The “Business Address of”OWL-S file provides a semantic objectmapping via its XSLT script, thus making it possible to connect contact-producingservices with address-consuming services.XSLT scripts in OWL-S files for address-consuming services marshal semanticaddress instances into appropriate SOAPmessages for invocations of Web and UPnPservices. Therefore, a “Contact Providing”service can be composed with, for exam-ple, “View Map of” service through “Busi-ness Address of” (semantic object map-ping) service, and the composition can beexecuted using only XSLT scripts in theseOWL-S files, to show the area map aroundthe business address of the contact.

SEPTEMBER/OCTOBER 2003 computer.org/intelligent 71

<soapenv:Envelope ... ><soapenv:Body>

<GetWeather ... ><zip type="xsd:string"... > 20740</zip>

</GetWeather></soapenv:Body>

</soapenv:Envelope>

“Weather info of”Web service

<soapenv:Envelope ... ><soapenv:Body>

<InitService ... ><address type=xsd:string" ... >8400 Baltimore Ave, College Park, MD 20740

</address></InitService>

</soapenv:Body></soapenv:Envelope>

“View map of”UPnP service

</soapenv:Body></soapenv:Envelope>

<soapenv:Envelope ... ><soapenv:Body>

<GetPhoto ... ><street type="xsd:string"... > 8400 Baltimore Ave</street><city type="xsd:string" ... > College Park</city>

<state type="xsd:string" ... >MD</state> <zip type="xsd:string" ... >20740</zip>

</GetPhoto>

“Aerial photo of”Web service

<rdf:RDF... > ... <Contact <rdf:ID="Contact_Ryusuke_Masuoka"><hasBusinessAddress><Address>

<Country>USA</Country><StreetAddress>8400 Baltimore Ave</StreetAddress><City>College Park</City><State>MD</State><ZipCode>20740<ZipCode></Address>

<hasBusinessAddress><hasHomeAddress>... </hasHomeAddress>

<FirstName>Ryusuke</co:FirstName><LastName>Masuoka<LastName>

... </Contact ... > ... </rdf:RDF >

Semantic contact instance Semantic address instance

“Aerial photo of”OWL-S

“Viewmap of”OWL-S

<rdf:RDF ... ><Address ... >

<Country>USA</Country><StreetAddress>8400 Baltimore Ave</StreetAddress><City>College Park</City><State>MD</State><ZipCode>20740</ZipCode></Address>

</rdf:RDF>

“Weather info of”OWL-S

“Business Address of”

OWL-S

Figure 4. Semantic object mapping and groundings by OWL-S used in STEER. All the XML data is taken from actual executions of service compositions and edited for readability.

Why ontology and pervasivecomputing … at all?

The ad hoc, spontaneous, and dynamicnature of the pervasive computing envi-ronment requires the systems to be late-binding. The user interface, availablewhile on the go, is usually limited in modal-ities, bandwidth between users, and so on.Ontologies describing services give thesystem the necessary semantics to provideusers with enough functionality in limiteduser interfaces without really accessingservices themselves.

Ontologies in the pervasive computingenvironment are more manageable com-pared to, for example, those for the Inter-net. Ontologies for devices will be createdby device manufacturers, which can putresources into their creation. They can evencreate an oligopoly of ontologies by form-ing consortia. Embodiments of deviceswith physical representations related to theparticular location lead to simpler ontolo-gies. You can have the same device in the next room or downstairs, and there isreal reuse of ontologies enabled by naturalboundaries in physical environments.

On the other hand, people and companieson the Internet are under the constant pres-sure of differentiating from others becauseof the Internet’s universal connectivity (thevery reason for its success). Even end usersare pressed to present views of the worldthat are different from others. These differ-ences are very difficult for ontologies tocapture.

Our implementation of a rich, user-friendly task computing environmentdepends crucially on Web ontologies writ-ten in the OWL language. In our observa-tion, OWL provided a reasonable balancebetween power and accessibility. Althoughwe were happy that both OWL and OWL-Shave their advanced capabilities, we didn’treally use them to the fullest possible degree.It turns out that a little semantics goes along way.

Or so we believe. While very loosely cou-pled, the components of our task-computingenvironment are based on a set of technolo-gies designed to work well together. Eachpart has its exciting moment, but it’s theway they mesh together that makes workingwith the system natural and compelling. Itseems possible that the system’s overall feelcould be preserved even if we relied less onontologies.

In a sense, we do rely less on ontologies

as traditionally conceived and used. Theuser’s understanding of the functionalitiesavailable in the environment were as muchdetermined by extra-ontological metadataas by the formal relationships betweenterms. For example, we found commentsand term labels embedded in the ontologyinvaluable. The formal description allowedSTEER to put the service in the right place,but it’s the human-centered descriptions thatgive the user confidence to make the finalselection. Similarly, connecting high-levelontological descriptions with lower-levelXML Schema and WSDL descriptions notonly allowed STEER to automatically invokecomposed services but made those servicescomprehensible to nonprogrammers.

So, while ontologies are key, how theyfit in with the wide variety of representa-tional schemes present in our increasinglyheterogeneous but interconnected informa-tion systems will make or break them. Bytaking to heart the Web’s lessons, ontolo-gies stand poised to break free from theirhistorical niche to become an important andvalued part of our everyday computingexperience. Both ontologies and our com-puting experience will be the better for it.

References

1. R. Masuoka, B. Parsia and Y. Labrou, “TaskComputing—The Semantic Web Meets Per-vasive Computing,” Proc. 2nd Int’l SemanticWeb Conf. 2003 (ISWC 03), Springer-Verlag,2003, to be published.

2. W3C Web Services Description WorkingGroup, www.w3.org/2002/ws/desc.

3. S. McIlraith and D. Martin, “Bringing Seman-tics to Web Services,” IEEE Intelligent Sys-tems, vol. 18, no. 1, Jan./Feb. 2003, pp. 90–93.

4. W3C Web-Ontology (WebOnt) WorkingGroup, www.w3.org/2001/sw/WebOnt.

72 computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Ryusuke Masuokais a senior researcherwith Fujitsu Labora-tories of America.Contact him at rmasuoka@fla.fujitsu.com.

Yannis Labrou is aresearcher with Fu-jitsu Laboratories ofAmerica. Contacthim at yannis@fla.fujitsu.com.

Bijan Parsia is aresearch philosopherat the University ofMaryland’s MINDLab. Contact him atbparsia@isr.umd.edu.

Evren Sirin is aPhD student at theComputer ScienceDepartment of Uni-versity of Maryland,College Park. Con-tact him at evren@cs.umd.edu.

We’d liketo hearfrom youEMAIL

intelligent@computer.org

QUESTIONS?COMMENTS?

IEEE Intelligent Systems wants to hear from you!

QUESTIONS?COMMENTS?

IEEE Intelligent Systems wants to hear from you!