User Satisfaction of a Hybrid Ontology-engineering Tool

Christophe Debruyne and Ioana Ciuciu

13-09-2013 @ META4eS, OTM 2013

Overview

‣ Motivation

‣ Background

‣ Usability Test Design

‣ User Satisfaction

‣ Results and Recommendations

‣ Conclusions and Future Work

2

Motivation

‣ Semantic interoperability and vocabularies for Linked Data initiatives

‣ Ontology engineering is far from trivial

‣ Requires appropriate methods and tools

‣ Hybrid Ontology Engineering platform in the context of the Open Semantic Cloud for Brussels (OSCB) project

‣ Evaluate usability and user satisfaction of the hybrid ontology platform

3

Background

‣ IBM Post-study System Usability Questionnaire (PSSUQ)

4

Information Quality

System Use

Interface QualityOverall

Background

‣ Hybrid Ontology Engineering‣ Community promoted to first-class citizen‣ Social interactions are‣ formalized‣ lead to ontology evolution‣ supported by a glossary

‣ Concepts are described ‣ formally --> with binary fact-types called lexons

e.g., <Research Domain, Call, with, of, Deadline>‣ informally --> with glosses

5

GOSPL Method and Tool

6

GOSPL Method and Tool

7

GOSPL Method and Tool

‣ With respect to the previous study‣ System usefulness scored best‣ Information quality needed most improvement

‣ Changes made to the system to tackle problems‣ More (online) documentation‣ Changes to stylesheets‣ RSS to keep track of discussions

‣ Additional functionalities‣ Tool support for managing annotations‣ A reputation framework

8

Usability Test Design

‣ 23 MSc student volunteers part of an ontology engineering course

‣ Objective:‣ prototype ontology related to research domain‣ use ontology to annotate‣ “reverse engineered” existing application‣ existing research information system

‣ Evaluate formative and summative user satisfaction of the ontology engineering platform‣ Compare with results from a previous study with similar experiment

9

Usability Test Design

‣ Tasks on the platform‣ Propose changes to the hybrid ontology‣ Discuss and vote‣ Conclude discussions --> ontology evolution‣ Create and manage communities‣ Annotate information systems

‣ Participants‣ 36 students, 9 groups for the ontology engineering experiment‣ Background in computer science or similar‣ 23 students participated in the survey (of which 4 complete groups)

10

Summative User Satisfaction

‣ Summative evaluation‣ System usefulness and interface quality remained the same‣ Information quality and overall satisfaction improved

‣ Formative evaluation‣ Tracking “hottest” discussions‣ Correcting mistakes‣ Voting mechanism (!) and verbalization of constraints (!)

‣ Positive outcome given the problems encountered with application management and the reputation framework

11[2] Ioana Ciuciu, Christophe Debruyne: Assessing the User Satisfaction with an Ontology Engineering Tool Based on Social Processes. OTM Workshops 2012: 242-251

Recommendations

‣ Overview of discussions need improvement

‣ (Error-) management of application commitments

‣ Verbalization

‣ Removing voting mechanism

12

Conclusions and Future Work

‣ Conclusions‣ System usefulness performs best‣ Information quality improved a lot‣ Overall satisfaction remained stable‣ Given the fact that new functionalities posed problems, this can be

interpreted as a considerable improvement of the tool

‣ Future work‣ Third iteration planned in May 2014‣ Discussion overview + lightweight user participation

13

Questions?