evaluating wiki collaborative features in ontology authoring

1041-4347 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TKDE.2014.2312325, IEEE Transactions on Knowledge and Data Engineering

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 1

Evaluating Wiki Collaborative Features inOntology Authoring

Chiara Di Francescomarino, Chiara Ghidini, and Marco Rospocher

Abstract—It is nowadays well-established that the construc-tion of quality domain ontologies benefits from the involve-ment in the modelling process of more actors, possibly havingdifferent roles and skills. To be effective, the collaborationbetween these actors has to be fostered, enabling each of themto actively and readily participate to the development of theontology, favouring as much as possible the direct involvementof the domain experts in the authoring activities. Recent workshave shown that ontology modelling tools based on wikis’paradigm and technology could contribute in meeting thesecollaborative requirements.

This paper investigates, both at the theoretical and empir-ical level, the effectiveness of wiki features for collaborativeontology authoring in supporting teamworks composed of do-main experts and knowledge engineers, as well as their impacton the entire process of collaborative ontology modelling andentity lifecycle.

Index Terms—collaborative ontology modelling, ontologyengineering, wikis

I. INTRODUCTION

It is nowadays well-established that crafting ontologieshas become a teamwork activity, as it requires a rangeof knowledge and skills hardly findable all together ina single person. For this reason collaborative aspects inontology modelling have been investigated, and severalworks to support and enhance collaboration in this con-text have been presented (see e.g. [1], [2], [3], [4]). Therequirements and features that have emerged from thesestudies highlight the need to support collaboration in anarticulated way: from supporting the collaboration betweenwho understands the domain to be represented (the DomainExpert, or DE) and who has proper expertise in ontologymodelling (the Knowledge Engineer, or KE), to supportingcommunication, discussion, and decision making between(geographically) distributed teams of ontology contributors.

The requirement of effectively involving domain experts,making them able not only to provide domain knowledge toknowledge engineers but also to directly author ontologiestogether with knowledge engineers, is also recognised in anumber of works (see e.g., [4], [5], [3]) as a crucial step tomake the construction of ontologies more agile and apt tothe needs of e.g., a business enterprise. As argued in [4],traditional methodologies and tools are based on the ideathat knowledge engineers drive the modelling process. Thisoften creates an extra layer of indirectness which makes thetask of producing and revising conceptual models too rigidand complex, e.g., for the needs of business enterprises.In addition, the leading role of knowledge engineers can

The authors are with FBK—IRST, Trento, ItalyEmail: dfmchiara|ghidini|[email protected]

hamper the model construction as the domain experts (anddomain knowledge) may become secondary to the processof efficient knowledge modelling, especially when domainexperts have no understanding of the languages and toolsused to build the conceptual models. Furthermore, thelogical formalisms with which ontologies are encoded (e.g.OWL) may prevent domain experts from accessing thedomain knowledge encoded in the model.

The requirement of supporting the collaboration betweenthe individuals of the whole modelling team, independentlyof their role, is similarly important and well recognised(see e.g., [1], [2], [3]). Indeed, it is not rare the situationwhere the modelling team is geographically distributedand/or users may not be able to participate to physicalmeetings. Supporting collaboration requires enabling theawareness of the user on the evolution of the modellingartefacts, favouring the coordination of the modelling effortwithin the team, as well as fostering the communication ofmodelling choices and decisions among the modellers.

These different collaborative modelling aspects maybenefit from the availability of wiki tools for ontologyauthoring. Nowadays, wikis are among the most populartechnologies for collaborative and distributed content au-thoring: people from all over the world can access simulta-neously the same version of the content, and any change isimmediately available to all the users. Although wikis wereoriginally envisioned for editing of unstructured content,like text (e.g. Wikipedia), recent works ([6], [7], [8]) haveshown their applicability for the (collaborative) authoringof structured content, including ontologies. Indeed, wikistypically offer some collaborative features (to which werefer to as wiki collaborative features), which can beexploited to favour the kind of collaboration needed forontology authoring:

• wikis editing interfaces can be easily customized, evenat user (or group of users) level, so to provide mecha-nisms to access and author the ontology compatiblyto the skills and role of the team members. Thus,simplified and guiding interfaces may be provided toDEs to edit (a limited part of) the ontology, whilemore powerful interfaces may be offered to KEs.

• the collaborative nature of wikis makes them a sourceof inspiration for collaborative features able to increasethe DEs involvement in the ontology authoring. There-fore, functionalities for discussing ideas and choices,for commenting decisions, for notifying (and tracking)changes and revisions, are in the core-set of most wikisystems, or can be easily added to them. The advan-tages of having these functionalities tightly integratedin the modelling tool are many, from the possibility to




link discussions to specific entity of the ontology, tothe capability to avoid users learning yet another tool.

In this work we investigate, both at the theoreticaland empirical level, the effectiveness and impact of wikifeatures to support collaborative ontology authoring. Thetheoretical analysis, performed by using a reference frame-work for the analysis of collaboration features in modellingtools [9], has the aim of positioning the support for col-laboration provided by wiki features w.r.t. five levels ofsocial interaction which are at the basis of collaborationin modelling activities. The results of this analysis showthat wiki collaborative features address all the five levelsof social interaction contained in the framework.

The empirical evaluation, performed with real DEs andKEs according to the methodology proposed in [10], hasthe aim of understanding more in detail whether wikicollaborative features are effective in (i) making DEs moreactive in the authoring of ontologies, and (ii) supportingthe collaboration during modelling. This evaluation hasbeen performed using MoKi [8], a wiki-based ontology au-thoring tool employing several wiki collaborative features.The results of this evaluation show that wiki collaborativefeatures:

• are effective in encouraging and supporting the activeinvolvement of DEs in the modelling activities and inreducing the overall effort spent by team members ininteracting; and

• have an impact on the ontology development processadopted by the team members as well as on thelifecycle of the built ontology entities; in detail, thefeatures seem to make (i) the process of collaborativemodelling less regimented and rigid, enabling teammembers to more agile reactions and indirect inter-actions with others and (ii) the entity lifecycles morestructured and close to a common pattern.

The contribution of the paper, hence, substantially ex-tends the work presented in [11] with a wider perspectiveand a finer-grained analysis of the support provided by thewiki collaborative features to collaborative ontology author-ing (a more detailed analysis is reported in Section VI).

To the best of our knowledge, the evaluation performedin this work provides a first theoretically and empiricallyrigorous investigation of the support provided by wikicollaborative features to favour collaboration in ontologymodelling. Though dealing with wiki collaborative fea-tures, the results of this comprehensive evaluation goesbeyond the boundaries of wiki-based collaborative ontol-ogy tools: indeed, the features we investigated are easilyimplementable, if not already offered, also in collaborativemodelling tools not based on wikis. Therefore, the resultsobtained suggest that also these tools could deeply benefitby the adoption of wiki collaborative features.

The paper is organized as follows. In Section II, the wikicollaborative features are introduced and MoKi, a wiki sys-tem based on these features, is described (Subsection II-A).In Section III we report about the theoretical evaluation ofthe support provided by the wiki features to the collabo-rative modelling. We hence describe how the experimentalevaluation has been designed (Section IV), reporting and

discussing in detail the obtained results (Section V). Finally,in Sections VI and VII we present some related works andconcluding remarks.

II. WIKI COLLABORATIVE FEATURES

Following the huge success and popularity of Wikipedia,the free on-line encyclopaedia, the last ten years haveseen the massive proliferation of wikis, both on the web,for the collaborative authoring and sharing of communityknowledge1, as well as within companies’ intranets, as con-tent/knowledge management system of an organization2.The success of wiki systems is due to several factors [12],spanning from the easiness to create/edit content simplyvia a web-browser and the possibility to relate pages vialinks, to the involvement and support of the user in thecollaborative creation process. These factors have recentlyinspired the successful development of wiki systems tosupport the (collaborative) authoring also of structuredcontent, including ontologies ([6], [7], [8]).

Our contribution investigates the effectiveness and im-pact of some wiki features on the collaborative authoringof ontologies3. In details, we considered the followingcollaborative features, which either are among the core-set4 of wiki functionalities, or are motivated by some ofthe aspects which made wikis so popular for collaborativeauthoring.Multi-mode access to content: one of the success of wikislies in the easiness of customizing them for a given context,especially for what concerns the user interface and theway users interact with the content. For instance, tens ofextensions5 of the MediaWiki framework [14] deal withcustomizable user input methods, e.g. by means of formsor input-boxes. Indeed, content access/editing interfaces canbe easily customized even per user or user group, so toprovide mechanisms to access and author the wiki contenttailored to the preferences, skills, and roles of the teammembers, thus favouring and supporting their participationin the authoring process.Discussion mechanism: wikis offer discussion mechanisms(e.g. the Talk tab of a Wikipedia page) to enable usersto communicate/comment/debate the content of a wikipage. Comments may be organized in sections or threads,sometimes simply via text indentation, and are usuallylogged with user/date/time details.Watchlist: wikis provide watchlist mechanisms to allowusers to monitor the evolution of wiki pages they areinterested.Notification mechanism: notification services are usuallyoffered to update users about changes on the pages that arerelevant for them.Revision History: one of the key feature of wiki systems

1E.g., see http://community.wikia.com/wiki/Hub:Big wikis2See for instance some companies listed at http://socialmediatoday.com/

stewartmader/107010/7-effective-wiki-uses-and-companies-benefit-them3A more detailed description can be found at https://dkm.fbk.eu/index.

php/MoKi Evaluation Resources4Recent wiki tools have also proposed additional mechanisms to favour

collaboration effectively, such as automatic support for conflict resolution(see [13]).

5See http://www.mediawiki.org/wiki/Extension Matrix/AllExtensions

http://community.wikia.com/wiki/Hub:Big_wikis

http://socialmediatoday.com/stewartmader/107010/7-effective-wiki-uses-and-companies-benefit-them

http://socialmediatoday.com/stewartmader/107010/7-effective-wiki-uses-and-companies-benefit-them

https://dkm.fbk.eu/index.php/MoKi_Evaluation_Resources


http://www.mediawiki.org/wiki/Extension_Matrix/AllExtensions




is the tracking of any change and comment on the wikicontent (e.g. the View history tab of a Wikipedia page).

We introduce a wiki system for collaborative ontologyauthoring which implements all these features.

A. MoKiMoKi [8] is a collaborative MediaWiki-based [14] tool

for modelling ontological and procedural knowledge in anintegrated manner6. A running installation of MoKi canbe tested on-line at https://moki.fbk.eu/moki/tryitout2.0/.MoKi is grounded on three main pillars, which we brieflyillustrate with the help of Figure 1:

• each basic entity of the ontology (i.e., concepts, objectand datatype properties, and individuals) is associatedto a wiki page. For instance, the concept Mountain inFigure 1 is associated to a wiki page which containsits description;

• each wiki page describes an entity by means of bothunstructured (e.g., free text, images) and structured(e.g. OWL axioms) content;

• a multi-mode access to the page content is provided tosupport easy usage by users with different skills andcompetencies. Figure 1 shows three different accessmodes, for accessing the unstructured and structuredcontent of the wiki page.

A comprehensive description of MoKi is presented in [8].In what follows, we focus only on the collaborative featuresthat MoKi offers to the users of the modelling team.Multi-mode access to page content: Users can access theontological knowledge contained in MoKi using the threedifferent access modes:

• The unstructured access mode: This access modeallows the user to edit/view the content of the un-structured part of the MoKi page of an ontology entity.The editing/viewing of this part occurs in the standardMediaWiki way. The textual description of the conceptMountain in Figure 1 (top-right) is an example of thisaccess mode. This access mode is meant to be usedby any user of the modelling team.

• The fully-structured access mode: This access mode(middle-right box of Figure 1) allows the user toedit/view the content of the structured part of a MoKipage using the full OWL 2 expressivity, allowing toview/edit formal statements (axioms) describing theontology entity associated to the page. Axioms arewritten according to the latex2owl syntax, a latex-styleformat for writing ontologies using a text-editor, whichcan be automatically translated into (an RDF/XMLserialisation of) OWL. This access mode is meant tobe used by KEs only.

• The lightly-structured access mode: The purpose ofthis access mode is to allow users with limited knowl-edge engineering skills, to edit/view the content ofthe structured part of the MoKi page in a simpli-fied and less formal way. The lightly-structured ac-cess mode is provided through a form made of two

6Though MoKi allows to model both ontological and procedural knowl-edge, here we will limit our description only to the features for buildingontologies.

components, as depicted in the bottom-right box ofFigure 1. In the top half part the user can view andedit simple statements which can be easily convertedto/from OWL statements. For instance, in the caseof concepts the user can edit statements of the form“Every subject is a object”, “Every subject has aspart a object”, or, more generally, statements of theform (subject, property, object), which correspondto the latex2owl statements “subject \cisa object”,“subject \cisa \exists hasPart.object”, and “subject\cisa \forall property.(object)”. Analogous forms areprovided for properties and individuals. If the OWLversion of any of these statements is already containedin the structured part of the page, then the correspond-ing fields are pre-filled with the appropriate content.Similarly, when any of these simple statements ismodified in the lightly-structured access mode, thechanges are propagated to the content of the structuralpart of the page. The bottom half of the form providesa description of those OWL statements which cannotbe intuitively translated/edited as simple statements asthe ones in the top half of the page. In the current im-plementation, this part contains the translation of thosestatements in Attempto Controlled English, providedby the OWL 2 Verbalizer [15]. The purpose of thisbottom half of the form is to give the DEs a flavour ofthe complex statements that a knowledge engineer hasformalized. If a domain expert is doubtful about someof the statements, he/she can mark them and ask fora clarification using e.g., the Discussion mechanism.This access mode is meant to be used mainly by DEs.

Discussion mechanism: MoKi exploits the MediaWiki Dis-cussion mechanism to enable domain experts and knowl-edge engineers commenting on modelling choices, anddebating possible modelling options in order to converge toa shared formalization. Discussions are possible on a singleontology entity or on (part of) the whole model. Commentsin the discussion pages are organized in threads, with detailson the user and date/time associated to each comment.Watchlist: The MediaWiki watchlist functionality allowsMoKi users to be notified (with messages and email alerts)of changes performed on pages (and, thus, ontology en-tities) they are monitoring. The user can autonomouslydecide to add/remove ontology entity to their own watchlist.By default, pages created/edited/discussed by a user areautomatically added to the user’s watchlist.Notification mechanism: MoKi provides a notificationmechanism which allows users to be updated of changeson the ontology that are relevant for them. Notifications areautomatically sent in case changes to pages in the users’watchlist occur. Users can also explicitly notify (a subset of)other users of specific changes they performed, solicitinga confirmation or revision on some specific aspect of theontology. Users receive notification by email, or the firsttime they connect back to MoKi.Revision history: Any change and comment added onspecific ontology entities is tracked in MoKi, thanks to therevision mechanism provided by MediaWiki. Some specificfunctionalities are provided, like browsing the last changesperformed, newly created ontology entities, specific user

https://moki.fbk.eu/moki/tryitout2.0/




Fig. 1: An example of page and of the different access modes in MoKi.

contributions, recent/new discussions, most active users,and so on, to enable users being aware of the activitieshappening on the ontology.

III. THE THEORETICAL ANALYSIS

In this section we present a theoretical analysis of thesupport provided by the wiki features described in Sec-tion II to collaborative ontology authoring. To this purpose,we exploit a framework, recently proposed by Mendlinget al [9], for the analysis of the collaboration featuresin process modelling tools. They argue that collaborativemodelling is a form of social interaction among people: itrequires that different stakeholders work together in a team,sharing the common goal of creating a common model aswell as a common understanding of the process they arecarrying out and of their individual contribution. In otherterms social interaction among people in the modellingteam contains all the aspects on which collaboration isbased on.

Technology should be able to fill the gaps occurringamong team members (e.g., geographical distribution orasynchronous work) and support them in their social in-teraction. Starting from these assumptions the proposedframework investigates the technical support provided bymodelling tools to each of the five levels of social interac-tion identified by Malone et al. [16] and further investigatedin the field of CSCW (Computer-Supported CooperativeWork): awareness, communication, coordination, group de-cision making and team-building. Each of these levels isbased on the previous one(s) and provides support to thefollowing one(s).Awareness: Awareness consists in the mutual provision(and possession) of information about other team members,their role and the work they carried or are carrying out, thusallowing to reduce uncertainty and to improve collaborationin situations of mutual dependencies. It can be classified infour main types [17]:

• Informal awareness: information about the presence,activities, location and intentions of other members ofthe team.

• Group-structural awareness: information about the

composition, roles, responsibilities and positions in-side the team.

• Social awareness: information about the interests, theattention and emotional conditions (e.g., communi-cated by means of the facial expressions or eye con-tact) of the other members of the team.

• Workspace awareness: information about workspaceartefacts and users’ work and intentions on them.

Typical examples of technical support to awareness aresystems providing information about presence (e.g., instantmessaging, user status notification, logging information),those collecting information about activities and emotionsby users working on the same artefacts (e.g., task histories,gestures, emotions) and finally systems integrating infor-mation from other systems.Communication: Communication is usually seen as theprocess of exchanging information (in the form of messagesfrom a sender to a receiver) in a proactive and meaningfulway. This requires the encoding of the information bythe sender and the decoding (of the same information)by the receiver. Encoding and decoding require a com-monly agreed language that allows the sender and thereceiver to understand each other. Communication, relyingon the awareness, poses the basis for the coordinationthat requires mutual agreement. Communication can bedirect (the sender knows to whom the message is directed)or indirect (messages are stored and can be retrieved bypotential receivers in whatever successive time), as wellas synchronous (sender and receiver are both availableduring the message exchange) or asynchronous (messagesare encoded and decoded in different times). Under theumbrella of technologies supporting the communication itis possible to find the whole set of classic and recenttechnologies used for the synchronous (e.g., chat, phonecalls, audio and video conferences) or asynchronous (e.g.,emails, text messages, forums) communication.Coordination: Coordination is “the act of managing in-terdependencies between activities performed to achieve agoal” [16]. Malone et al. identified four components ofcoordination: (i) goal identification; (ii) mapping of thegoals to activities; (iii) selection of actors and assignmentof activities; and (iv) management of interdependencies




Social Aspect MoKi Functionalities

Awareness Revision history, Watchlist, Notification mechanismMulti-mode access to content, (Shared workspace)

Communication Multi-mode access to content,Discussion mechanism, (Shared workspace)

Coordination Revision history, Watchlist, Notification mechanismDecision Making Discussion mechanism, Notification mechanismTeam Building Multi-mode access to content, (Shared workspace)

TABLE I: Theoretical Framework applied to MoKi

between the activities to support goal relevant relationshipsbetween activities. The technological support to coordina-tion can be either explicit (i.e., systems fully automate thecoordination activities) or implicit (i.e., systems only pro-vide a semi-automated coordination together with aware-ness information to users that can in this way coordinatethe team). Among these technologies we can find workflow-based systems, but also systems guiding the coordinationby means of the communication or special forms based onsemi-structured messages.Group Decision Making: Group decision making dealswith the identification of solutions for complex problems incollaborative settings, where different people with differentknowledge work together using awareness and communi-cation to solve the problems. Decision making takes henceadvantage of awareness and communication to converge toa final choice. Technical support to the group decision mak-ing includes, besides systems enhancing the communication(and thus facilitating the decision making process), alsothose systems structuring and recording decision processesin order to reuse them, as well as the communication-driven decision support systems, i.e., groupware supportingcommunication, scheduling, document sharing.Team Building: Team building is the constitution ofsocial entities that work together by sharing a commonunderstanding of their work as well as the responsibilitiesof the produced results. Team building has as purposethe collaborative work and as prerequisites all the lowestlevels of social interaction. Examples of technologicalsupport for the team building are shared workspaces, groupeditors, mechanisms for role and authorization control.

As done by Mendling et al. for a critical analysis ofthe process modelling tools, we applied the framework tothe wiki collaborative features, investigating their supportto each of the interaction aspects. Table I summarizes theresults of our analysis: each of the features contributes tosupport one or more aspects.

In detail, the revision history and the watchlist func-tionalities together with the notification mechanism allowusers to be aware of the activities carried out by otherusers (collaborating to the ontology authoring) on theshared ontology and being notified in real time (informalawareness); moreover, the revision history and the multi-mode access to content functionalities, besides the sharedworkspace, make the ontology and its updates available tothe whole team (workspace awareness).

Besides the shared workspace that, per se, providessupport to the (implicit) communication among the users,the wiki discussion mechanism, that allows users to discussabout specific pages, as well as the multi-mode access to

content, that fosters the exchange of knowledge betweenusers with different competencies, explicitly and implicitlycontribute to the users’ communication.

All the functionalities providing information about theactivities carried out by collaborating users and aboutchanges occurring on entities of the ontology, togetherwith the associated notification mechanisms, contributeto support the coordination of activities. Moreover, thepossibility of explicitly notifying to specific users aboutperformed changes in order to solicit a confirmation orrevision also represents an important means to facilitate thecoordination.

Discussion and notification mechanisms, by allowingusers to debate and discuss about possible modellingchoices and to be notified about changes, are also of the ut-most importance for implicitly supporting the convergenceof decisions to be taken together.

Finally, the shared workspace and the multi-mode accessto content, by allowing to share the ontology in a wayunderstandable by actors with different skills and compe-tencies, also pose firm bases to the creation of a strongcollaborative team.

In summary, the analysis shows that wiki collaborationfunctionalities are able to support all the levels of socialinteraction.

IV. THE EMPIRICAL EVALUATION: EXPERIMENTDESIGN

This section presents the design of the empirical studycarried out to evaluate the support provided by the wikicollaborative features described in Section II to the processof ontology modelling. The study is conducted and reportedaccording the methodology proposed by Wohlin [10] for theevaluation of software engineering experimentations.

A. Goal of the Study and Research QuestionsIn the study we are interested in investigating whether

wiki collaborative features effectively support the collab-oration between domain experts and knowledge engineers(hereafter DEs and KEs, respectively) working together forthe ontology authoring, as well as whether the features havean impact on the overall process of collaborative modellingcarried out by DEs and KEs and on the lifecycle of thebuilt ontology entities.

The goal of the study is to consider two approaches(with and without the collaborative wiki features) withthe purpose of evaluating the resulting processes ofcollaborative ontology formalization. In detail, ourquality focus is on: (i) the effectiveness of the supportprovided by the collaborative wiki features to the processof collaborative modelling and (ii) the impact of thecollaborative features on the processes of collaborativeontology modelling and of the entity lifecycle. Hence, theresearch questions to investigate are the following:

Effectiveness of the wiki collaborative features for on-tology modelling

RQ1 Do the wiki collaborative features improve theinvolvement (and productivity) of DEs in editingactivities?




RQ2 Do the wiki collaborative features reduce theeffort required to team members to interact?

RQ3 Do the users perceive the support provided by thewiki collaborative features as effective?

RQ1 deals with the effectiveness of the wiki collaborativefeatures in augmenting the involvement of DEs inthe authoring of the ontology, while RQ2 regards theeffectiveness of these features in reducing the effortrequired for the interaction among the team members, akey factor to enable the collaboration. RQ3 deals with thesubjective perception of the users about the effectivenessof the support provided by the wiki collaborative features.Though subjective, real users’ opinion is absolutelyrelevant when evaluating features aiming at supportingthem in their work.

Process of collaborative ontology modelling: impact ofwiki collaborative features

RQ4 Is there any difference in the sequence of macro-typologies of activities (i.e., edit, view and inter-action) characterizing the process of collaborativeontology modelling carried out by team members,with and without the support of wiki collaborativefeatures?

RQ4 aims at investigating the impact of the collaborativeauthoring features on the sequence of activities (edit, viewand interaction) carried out for collaboratively building theontology, in order to understand whether differences existbetween the resulting processes (e.g., whether one of thetwo processes is more dynamic and agile than the other).

Entity lifecycle process: impact of wiki collaborative fea-tures

RQ5 Is there any difference in the sequence of (editing)activity types (i.e., entity creation, descriptionediting, axiom editing, renaming and deletion)characterizing the process lifecycle of the ontol-ogy entities, with and without the support of wikicollaborative features?

Similarly to RQ4, RQ5 evaluates the impact of the wikicollaborative features on the sequence of activities char-acterizing the lifecycle of the modelled ontology entities(entity creation, description editing, axiom editing, renam-ing and deletion). This way RQ5 investigates whetherdifferences exist in the quality of the final ontology in termsof entity lifecycles (e.g., whether the lifecycle of the entitiesbuilt with or without the collaborative features is overallmore structured than the lifcycle of the entities modelledwith the other approach).

Following the methodology in [10], for each of the aboveresearch questions, a null hypothesis Hx0 and an alternativeone Hxa have to be formulated. The rejection of the nullhypothesis determines the positive answer to the researchquestion. For example, the rejection of:

“(H20) The time spent by teams’ members to interactwith the support of collaborative wiki features is not lowerthan the one spent without the collaborative features.”would imply the acceptance of:

“(H2a) The time spent by teams’ members to interactwith the support of collaborative wiki features is lower thanthe one spent without the collaborative features.”

The rejection of the null hypothesis implies the accep-tance of the alternative one and hence the positive answerto the corresponding research question.

The study is conducted in a context in which the involvedsubjects are four teams, each composed of two DEs andone KE, and the objects to be formalized in ontologies aretwo domains from the pedagogical field.

Each team was asked to collaboratively model an on-tology for each of the two domains. To this purpose, allsubjects were given the possibility to interact through chatsand emails, and to use a modelling tool. In detail, each teamwas asked to develop the ontology for one domain with amodelling tool equipped with wiki collaborative features,and the ontology for the other domain with a modelling toolwithout these features: hence, in the latter case, membersof the team were able to communicate only through meansexternal to the modelling tool (chats and emails).

With the aim of achieving a more rigorous evaluation ofthe impact of the wiki collaborative features, reducing theeffect of external factors, we chose to rely on two versions(with and without the collaborative features) of the sametool to run the experiment, rather than using two differenttools (e.g., one equipped with collaborative features andone without them). The rationale of this choice is that usingtwo different tools, the results could have been influencedby further differences between them (e.g. user interface,performances), not related to collaborative features. Amongthe wiki-based collaborative ontology editing tools, weused MoKi7 (see Section II), which presents several wikicollaborative functionalities. Hence, we resorted to twoversions of MoKi: one with the collaborative features (CM)and one in which the collaborative features have beenremoved (NCM).

At the end of the experiment, we compared and analysedthe resulting processes and ontologies, by focusing on theabove-mentioned aspects.

B. Context

The study involved 12 subjects (8 DEs and 4 KEs),organized in 4 teams (TA, TB , TC and TD), each includingtwo DEs and one KE. In detail, the 8 DEs are pedagogistsand psychologists employed in a publishing house8 special-ized in educational books, while the 4 KEs are experts inknowledge engineering working at FBK9.

To match the background of the DEs, two domainsfrom the pedagogical field were used for the experiment:one related to two cognitive abilities, namely Attentionand Concentration (AC), and the other to motivationaland emotional aspects of the learning process, namelyMotivation and Emotion (ME). The AC domain dealswith concepts (and corresponding processes, functions andmodels) such as attention, concentration, memory, their

7We remark here that our goal was not evaluating MoKi but the wikicollaborative functionalities.

8Centro Edizioni Erickson (http://www.erickson.it/)9www.fbk.eu

http://www.erickson.it/

www.fbk.eu




Teams L1 (morning session) L2 (afternoon session)NCM CM NCM CM

TA AC METB AC METC ME ACTD ME AC

TABLE II: Balanced design applied to the study

interactions and (cause-effect) relationships, as well as withtheir relationships with behavioural aspects and the learningprocess. The ME domain, instead, is related to concepts asmotivation (and its different aspects), emotion as well asthe educational material and the interventions to be usedfor facing motivational or emotional difficulties. A bookchapter was provided to the domain experts for both thedomains, to be used as starting point for the ontologyconstruction. Both chapters, consisting of approximately 25pages each, were taken from the same book.

C. Design, Material and ProcedureSubjects carried out the ontology modelling tasks on the

two domains in two different laboratory sessions, hereafteridentified with L1 and L2, held one in the morning andone in the afternoon, using NCM and CM (hereafter alsocalled treatments) according to a balanced design [10].Such a design allows the same subject to perform theexperiment twice and hence, given the same set of subjects,to maximize the data available for the evaluation, whilecontrolling learning effects.

Table II reports the details of the balanced schemaadopted in the study. Each team worked with each of thetreatments and each of the domains exactly once (e.g., TA

formalized AC with NCM and ME with CM). The order inwhich treatments were used is balanced (for instance, NCMis used in the morning session by TA and TC , and in theafternoon session by TB and TD; the contrary happens forCM). This way, the learning effect that could arise due tothe formalization of the same domain with both treatmentsand the effect of the order in which treatments are used bythe teams are limited.

To carry out the experiment each participant was pro-vided with: (i) a pre-questionnaire collecting informationabout her background and high-level competencies; (ii) anemail and chat account; (iii) a MoKi account; and, (iv) apost-questionnaire and a final questionnaire to collect hersubjective perception about the specific laboratory sessionand the tool. Moreover, each DE received a description ofthe domain to be formalized, which, given the strict timeconstraints of the study, was intended to be used only asstarting point to help DEs in focusing on the domain10.

Before the experiment execution, subjects (both DEs andKEs) were trained in how to use MoKi (both the NCM andthe CM version). Besides a theoretical description of MoKiand of its functionalities, a 2-hours hands-on session wascarried out in order to allow the subjects to become familiarwith the tool. After the training session, subjects were askedto fill the pre-questionnaire and, after the second laboratory,to fill a final questionnaire. Each laboratory session wasdivided into the five phases reported in Table III: PH1–

10The questionnaires provided to subjects are available at https://dkm.fbk.eu/index.php/MoKi Evaluation Resources

Role PH1 PH2 PH3 PH4 PH5(≈ 25 min) (≈ 25 min) (≈ 15 min) (≈ 15 min) (≈ 40 min)

DEs • • •KEs • • •

TABLE III: Laboratory division in five phases

RQ Alt. Factor Variable Unit/ Descriptionhp Scale

RQ1 H1involvement AxAN integer avg. # of edited axioms

(and productivity) EdOpN integer # of editing operationsRQ2 H2 effort to interact ConvL integer conversation # of characters

RQ3 H3

OEss [0, 4] perceived effectivenessOEoU [0, 4] perceived ease of use

users’ subjective AwEss

[0, 4]

perceived effectiveness forCommEss, collaborative aspects (namely

perception CoordEss, awareness, communication,DMEss, coordination, decisionTBEss making and team building)

RQ4 H4process activity

PATSN integer # of switches among processsequence activity macro-typologies

RQ5 H5entity activity

EATSN integer # of switches amongsequence entity activity typologies

TABLE IV: Summary of the dependent variables

PH4 aiming to simulate the asynchronous collaborationbetween DEs and KEs (only DEs working on the ontologyin phases PH1 and PH3, while in phases PH2 and PH4only KEs being active); and PH5 investigating the processof synchronous collaborative modelling between them (bothDEs and KEs working simultaneously on the ontology).

D. VariablesIn order to accept (or reject) the alternative hypoth-

esis associated to each of the research questions listedin Subsection IV-A, we evaluate the effect of the wikicollaborative features, that is of the treatment, on the factorsinvestigated in the hypotheses (e.g., the effort required toteam members to interact, in case of H2).

In the study, the independent variable (i.e., the controlledvariable that is supposed to affect the factors analysedin the research questions) is the version of MoKi usedfor the collaborative ontology modelling. The independentvariable can therefore assume only two values, i.e., the twotreatments: NCM or CM.

The dependent variables (i.e., the variables associatedto the factors that potentially “depend” upon the treat-ment) considered in the study to evaluate the effect ofthe treatment on the factors, are shown in Table IV. Indetail, for each research question, the table reports (i) thecorresponding alternative hypothesis, i.e, the hypothesis wewant to confirm to positively answer the research question;(ii) the factor investigated by the research question; (iii) thespecific variable(s) used to measure the factor; (iv) the unitor the scale used for measuring each variable; and, finally(v) a short description of the variables.

In detail, for evaluating the effectiveness of the wikicollaborative features in increasing the involvement of DEsin modelling activities (RQ1), we chose to look at both themodelling product (by considering the average number ofaxioms AxAN per axiom category of the final ontologyedited by DEs) and at the modelling process (by computingthe number of editing operations EdOpN performed byDEs, where an edit operation is any saved change per-formed on the ontology). We believe in fact that thesetwo variables represent a reasonable indicator of the active






involvement of DEs in the ontology authoring. They bothaim at evaluating the quantity of work carried out by themodellers11. AxAN is computed by manually inspectingthe final ontology and computing the edited axioms foreach axiom category: (a) concept, (b) object property,(c) data property, (d) individual creation axioms; (e) “is-a” axioms; (f) restriction axioms and (g) other axiomsinvolving complement, union and intersection. EdOpN iscomputed by looking at the editing operations on ontologyelements contained in the final ontology logged in the MoKidatabase.

Evaluating the effort required to team members to in-teract (RQ2) is particularly complex; measuring the effortrequired to create the content of a message is indeed unfea-sible. However, the overall length of exchanged messages(by e-mail, chat or discussion) in terms of number ofcharacters (ConvL) provides a reasonable approximationof the quantity of information exchanged for interacting.

RQ3 (i.e., the users’ subjective perception) is evaluatedby taking into account the subjects’ perception about theoverall effectiveness of the MoKi functionalities (OEss),their ease of use (OEoU ), as well as the support theyprovide to each of the five levels of interaction presentedin Section III (AwEss, CommEss, CoordEss, DMEssand TBEss). All the subjective evaluations, collected fromthe final questionnaire filled by the subjects, are expressedon a 5-point Likert scale.

To quantitatively answer RQ4, i.e., to determine thedifference between the sequences of macro-typologies ofactivities (edit, view and interaction) carried out by teammembers with and without the collaborative features, wemeasured, as an approximation, the frequency of changesamong these macro-typologies. In detail, given the sameinterval time for both sequences (2 hours), we measuredPATSN , i.e., the number of switches among macro-typologies of activities occurring in the log files obtainedby combining MoKi, chat and email logs. This measure,although disregarding the order of the activities, providesan estimation of their variability in the modelling process.

To quantitatively measure the difference in the sequenceof activity types (entity creation, description editing, axiomediting, renaming and deletion) in the lifecycle of entitiesbuilt with and without the support of collaborative features(RQ5), we looked at the number of switches among thetypologies of activities occurring in MoKi files (EATSN ).Indeed, we believe that also in this case, computing thenumber of switches among the typologies of entity lifecycleactivities represents a reasonable approximation of thedifference between the two sequences.

V. THE EMPIRICAL EVALUATION: EXPERIMENTRESULTS

In this section we describe the statistical analyses per-formed on the collected data, including the objective data

11Although we are aware of the importance of the quality of the finalresult, in the evaluation we focused more on measuring the involvementof and the effort spent by the modellers in the ontology editing activities,rather than on the measuring quality of the resultant artifact. Neverthelesswe manually inspected the resultant ontology in order to verify thecorrectness/meaningfulness of the produced axioms.

related to the process and ontology analysis as well as thesubjective users’ evaluations. We analysed the influence ofthe treatments on each of the variables reported in Table IV.Due to the non-applicability of parametric statistical testsbecause of the violation of the preconditions (small numberof data points and non-normal distribution), we appliednon-parametric tests.

In detail, for the objective data related to subjects, i.e.,for which each of the subjects performed the task bothwith NCM and with CM (RQ1, RQ2 and RQ4), a pairedstatistical test, comparing the data with NCM and with CMrelated to the same subject, has been applied. Starting fromthese requirements, we resorted to the Wilcoxon test [10],a non-parametric paired test. For the objective data thatare not related to subjects, i.e., for unpaired data, as incase of the number of switches of activities per ontologyentity (RQ5), we applied a non-parametric unpaired test,the Mann-Whitney test [10]. According to the direction ofthe hypothesis to verify, we applied a one (if the directionis known a-priori and stated in the hypothesis) or two-tailed analysis (if the hypothesis only states a differencebetween the treatments). Instead, for the analysis of thesubjective evaluation (RQ3), we used a non-parametric testfor independence, the Chi-squared [10], which is used withcategorical variables.

All the analyses are performed with a level of confidenceof 95% (p-value < 0.05), i.e., there is only a 5% ofprobability that the results are obtained by chance.

A. Data Analysis

Research Question 1 (RQ1) Despite the final ontologiesrealized with NCM and CM were almost similar in size (onaverage, 16 concepts and 3.5 properties), differences existin the axiom authorship. Figure 2 and Figure 3 depict theboxplots12 related to the variables AxAN and EdOpN , forboth DEs and for the KEs. The boxplots on the left showthat the wiki collaborative features determine an increase inthe average number of ontology axioms per axiom category(i.e., concept creation, object property creation, data prop-erty creation, individual creation, “is-a” axioms, restrictionaxioms and other axioms) in the final ontology formalizedby DEs, as well as of the editing operations they performed.In other terms, the CM approach makes DEs more involvedin the modelling activities and hence more productive interms of number of axioms formalized. The boxplots onthe right, instead, reveal a reduction in the number of bothaverage axioms per axiom category and editing operationsfor KEs with CM. A possible explanation for this result isa shift of the KEs work towards other types of activities,like formalization checks and DEs support (e.g., providingmore fine-grained formalizations).

Table V, providing the descriptive statistics correspond-ing to the boxplots and the p-values obtained by applying

12A boxplot is a graphical representation of groups of numerical data,that allows to depict the minimum and the maximum values, the first andthe third quartile (where quartiles are the values dividing a set of valuesin four parts equally populated), and the median (i.e., the second quartile).In detail, a boxplot splits the data set into quartiles, through the “box” inthe boxplot. The box goes from the first to the third quartile, while theline within the box represents the median of the data set.




(a) DEs average axiom number (b) KEs average axiom number

Fig. 2: Boxplots representing the axioms per categoryformalized by DEs and KEs with NCM and CM

(a) DEs editing operations on theontology

(b) KEs editing operations on theontology

Fig. 3: Boxplots representing the editing operations per-formed by DEs and KEs with NCM and CM

the paired Wilcoxon test, confirms these trends as well astheir statistical relevance. In detail, the number of axiomsin the final ontology edited by DEs is on average peraxiom category 1.55 with NCM (for a total of 10.9 axiomsover all the seven categories) and 2.32 (for a total of16.2 axioms) with CM, while the number of editing op-erations they carried out is on average 21.63 versus 29.13,respectively. Conversely, for KEs, the average number ofediting decreases from 31 with NCM to 11.75 with CMand number of axioms per axiom category KEs edited inthe final ontology decreases from 2.36 with NCM to 1.32with CM, that is in total from 16.5 with NCM to 9.25with CM. Summarizing, the total number of axioms editedby DEs is about half of the number of editing operations(which is reasonable considering that the editing operationsalso include the concept descriptions), while the situation isdifferent for KEs. Indeed, with CM the number of editedaxioms and operations performed by KEs is almost thesame, while in the non-collaborative case KEs performeda number of operations that is almost double with respect tothe axioms they edited as last author. This could be due tothe difficulty of KEs, unaware of the domain, to modelit in a straightforward way. On the contrary, the activeinvolvement of DEs seems to help KEs to better focus theirmodelling effort. We can reject the null hypothesis H10and accept the alternative one H1: the wiki collaborativefeatures increase the involvement of DEs in the ontology

Role Variable Mean Median Std. Dev. p-valueNCM CM NCM CM NCM CM

DEs AxAN 1.55 2.32 0.13 1.4 2.72 2.45 0.03EdOpN 21.63 29.13 27 30 9.29 16.17 0.038

KEs AxAN 2.36 1.32 2.25 0.75 1.13 1.07 0.025EdOpN 31 11.75 18.5 14 13.98 5.12 0.049

TABLE V: Descriptive statistics and p-values for H1

Role Variable Mean Median Std. Dev. p-valueNCM CM NCM CM NCM CMDEs & KEs ConvL 3919.25 3319.92 3786.5 3372 1207.14 420.49 0.046

TABLE VI: Descriptive statistics and p-values for H2

Variable Investigated Factor Neg. Pos. Median p-valueOEss overall effectiveness 0 8 3 0.004678OEoU ease of use 0 12 2.56 0.000532AwEss effectiveness for awareness 0 10 3 0.001565

CommEss effectiveness for communication 1 8 3 0,01963CoordEss effectiveness for coordination 1 5 2 0.1025DMEss effectiveness for decision making 0 9 3 0.0027TBEss effectiveness for team building 3 6 2.5 0.3173

TABLE VII: Descriptive statistics and p-values for H3

editing.Research Question 2 (RQ2) Table VI reports the descrip-tive statistics related to the variable ConvL that we used forevaluating the effort required by members of the team forthe interaction. In case of the NCM approach, the overalllength of messages (in terms of number of characters)exchanged among teams’ members is higher than the samenumber for the CM approach: on average 3919 characters(951 for email, and 2968 for chat) with NCM and 3320(49 for email, 2678 for chat, and 593 for discussion) withCM. The result is statistically confirmed. We can hencereject H20 and confirm that the wiki collaborative featuresreduce the effort required by team members to interact (andhence collaborate).Research Question 3 (RQ3) Table VII reports the descrip-tive statistics of the subjective evaluation provided by usersabout the overall effectiveness, the average ease of use andthe effectiveness of the wiki collaborative features in ad-dressing the interaction levels described in Section III (i.e.,awareness, communication, coordination, decision makingand team building). We report in the table the overall num-ber of positive (rating strictly greater than 3) and negative(rating strictly lower than 3) subjective evaluation on the5-point (1 to 5) Likert scale. Overall, the subjects provideda positive evaluation about all the different aspects relatedto the effectiveness of the collaborative authoring features(the positive evaluations are always more than the negativeones, which, in turn, are almost always null). Except forcoordination and team building, the positive evaluation ofthe users is also corroborated by statistical significance. Thelack of statistical significance of coordination and teambuilding should not surprise given the small size of theteams. This demands for further investigations of thesefactors in case studies with larger teams. Relying on thepositive perception by users about the overall effectivenessand ease of use of the features, as well as on their positiveevaluation about the effectiveness in supporting three out offive interaction aspects and on the last consideration aboutthe team size, we can reject the null hypotheses H30 andconfirm the alternative one, H3.Research Question 4 (RQ4) Table VIII shows the de-scriptive statistics related to the number of switches acrossmacro-typology of activities (edit, view and interaction)carried out by subjects. The analysis of these data revealsthat the collaborative modelling processes performed withCM present a number of switches slightly higher than theone obtained with NCM (1207 versus 1050), i.e., there




Fig. 4: An example of the processes of collaborative modelling in NCM and CM (related to the team TA)

Variable Total Mean Median Std. Dev.NCM CM NCM CM NCM CM NCM CM

PATS 1050 1207 87.5 100.58 87 80 30.21 50.73

TABLE VIII: Descriptive statistics related to the number ofswitches among macro-typologies of activities for H4

exists a difference in the sequence of macro-typologiesof activities performed by subjects. Such a difference alsoappears by qualitatively comparing the processes of collab-orative modelling carried out by the same team with NCMand CM. For example Figure 4, reporting the processesfor Team TA, clearly shows in the last phase (i.e., the onecontemporaneously involving both DEs and the KE), thatthe process of ontology development carried out with CMis less rigid and encourages more agile interactions thanthe one carried out with NCM. Nevertheless, by lookingat the level of usage of the MoKi collaborative authoringfunctionalities, we noticed that a high usage of thesefunctionalities corresponds to an increase in the overallnumber of switches among typologies of activities carriedout by team members. In detail, exploiting as threshold themedian value among the CM usage values, we classifiedthe usage of the functionalities in low usage and highusage. By applying an unpaired Mann-Whitney test [10],we found that such a result is statistically significant (p-value = 0.002): a high usage of the MoKi collaborativefunctionalities corresponds to a high dynamism in thetypology of activities carried out. Moreover, in order tounderstand which ones among the CM functionalities hadsuch an impact on the overall number of operations carriedout, we performed an ANOVA test [10] and we found thatthis effect is mainly due to the use of the MoKi lightly-structured access mode (for editing/visualization purposes).Research Question 5 (RQ5) Figure 5 reports the processmodels related to the ontology entity lifecycles mined13

13The process models have been mined by applying the Heuristicminer plugin of the ProM 6.1 tool (http://www.promtools.org/prom6/)with the default values and then abstracted by grouping similar activities.The Heuristic miner plugin mines Heuristic Nets by retrieving frequentpatterns, thus focusing on the main behaviours in the event logs.

Fig. 5: Heuristic Nets describing the lifecycle of the entitiesbuilt with NCM and CM

Variable Total Mean Median Std. Dev.NCM CM NCM CM NCM CM NCM CM

EATS 342 306 2.8 2.47 3 2 1.26 1.23

TABLE IX: Descriptive statistics related to switches amongtypologies of entity lifecycle activities for H5

from the log files. The process models, represented asHeuristic Nets [18] (oriented graphs that can be easily con-verted into Petri Nets) in which the split and join semanticshas been hidden [18], show that the lifecycle of the entitiesbuilt with NCM is overall different from the one of theNCM entities. Besides discussions (Disc), differences existin the order in which the modelling activities are carriedout. For example, in the CM entity lifecycle, differentlyfrom the one of NCM entities, a precedence relation existsoverall between the description (DE) and the axiom (AxE)editing. Such a qualitative analysis is corroborated by thedata in Table IX, reporting the descriptive statistics relatedto the number of switches across typology of editing activ-ities (creation, description editing, axiom editing, renamingand deletion) carried out for each ontology entity, i.e., foreach MoKi page. The analysis of these data, indeed, revealsthat the lifecycle of ontology entities built with CM presentsa number of switches slightly lower than the one obtained

http://www.promtools.org/prom6/




Role Variable Mean Median Std. Dev.NCM CM NCM CM NCM CM

DEs & KEs AxTN 12.75 13.58 9 10 10.52 12.66EdOpN 24.75 23.33 24.5 15.5 11,38 15.71

TABLE X: DEs and KEs axiom creation and editingoperations

with NCM (on average 2.8 versus 2.47), i.e., there existsa difference in the process used for the entity construction,and that such a difference is also statistically significant(p-value = 0.03). We can hence reject the null hypothesisH50 and confirm that the wiki collaborative features havean impact on the way ontology entities are built.

B. Additional Findings And Discussion

Besides the outcomes reported so far, the analysis of thecollected data also provides additional interesting (thoughnon-statistically significant) findings, showing data trends,and further explaining the above results.

The increase in the DEs authoring activities (RQ1) isalso quantitatively confirmed: in detail, the percentage ofactivities related to the enrichment of the entities withaxioms carried out by DEs (with respect to the total numberof activities of the same type) increased from about 49%with NCM to 77% with CM. Such an increase in the au-thoring activity and in the amount of formalized knowledgecarried out by DEs, is also confirmed at team level: the totalnumber of axioms over all the axiom categories (AxTN )in the final NCM ontology is, on average, lower than thetotal number of axioms in the final CM ontology, thoughthe number of editing operations follows the contrarypattern. In other terms, as shown in Table X, while thenumber of axioms in the final ontology is on average12.75 with NCM and 13.58 with CM, the average numberof editing operations is higher for the NCM treatment:24.75 with NCM versus 23.33 with CM. Although we didnot perform a rigorous evaluation of the quality of theaxioms, no big differences among them emerged from themanual verification performed. This result suggests that theoperations carried out for the formalization with CM weremore effective than those carried out with NCM.

By better inspecting the log files of both NCM andCM, it comes out that the participation of DEs in thecollaborative modelling with CM has not been limited to theaxiomatization, but it also affected the renaming activities.In detail, the percentage of renaming carried out by DEswas 25% with NCM and 81% with CM.

A qualitative inspection at the collaborative modellingthrough the MoKi log files also reveals that, with CM, KEsdid not use at all the unstructured editing as well as thatthey reduced the usage of the structured editing and view.With respect to the NCM process, indeed, the number ofoperations for both structured editing and view is reducedof about one third. Moreover, an increase in the use ofthe lightly structured view by KEs can be observed. Theseresults, on one side, confirm the role of the collaborativefeatures in increasing the participation quantity and qualityof DEs in the ontology construction (there is no need ofrefinements on entity descriptions by KEs with CM and areduction of their effort in the axiomatization is observed);

Role Edit View InteractionNCM CM NCM CM NCM CM

DEs & KEs 0.081 0.081 0.438 0.451 0.48 0.468DEs 0.1 0.095 0.37 0.406 0.53 0.499KEs 0.045 0.049 0.578 0.555 0.377 0.396

TABLE XI: Time distribution of edit, view and interactionactivity typology

on the other side, the positive reaction of KEs to the useof the lightly-structured view can be mapped to the factthat, although the structured access mode provides themwith the full expressive power, the lightly-structured viewrepresents a useful means to get a global picture of thechanges performed by DEs.

Besides the increase in the involvement of DEs in theontology formalization, the wiki collaborative features arealso effective in reducing the overall effort spent by teammembers in interacting, i.e., in communicating about col-laborative aspects (RQ2). To such a decrease in the effort,however, does not correspond a decrease in the overallsatisfaction of the team members about the produced ontol-ogy. Indeed, comparing the answers provided by subjectsto the (post-questionnaire) questions about correctness,completeness and accuracy of the authored ontology, wedid not find any statistically significant difference amongthe NCM and CM evaluations. In order to understand thereason behind such a decrease, we manually inspectedchats, emails and discussion messages exchanged amongteam members and we qualitatively classified their topics.In detail, we found that such a decrease in the interactioneffort can be mainly justified with a reduction in the effortspent by (i) DEs to ask KEs to formalize some knowledge(4252 characters for NCM versus 47 characters for CM);and (ii) by team members in “useless” discussions, i.e.,discussions leading to no formalization in the final ontology(4096 characters for NCM and no “useless” discussion forCM). This analysis suggests us that the effort reductioncan be possibly attributed to the capability of the wikicollaborative features (i) to increase the DEs involvementand (ii) to make the interaction more focused on the issues.Moreover, the availability of wiki collaborative features alsoleads to a change in the preferred communication means.Subjects, in case of asynchronous communication, preferdiscussions associated to the specific page under analysis,to a well-known instrument such as the email exchange (19and 3 emails have been exchanged in the NCM and in theCM processes, respectively).

Furthermore, we noticed that, although the concreteusage of collaborative wiki authoring features impacts onthe overall process of collaborative modelling, increasingthe number of switches among the three main typologiesof activities (RQ4), the overall time distribution of thethree activity typologies in the NCM and CM processesremains almost unchanged (Table XI). By inspecting theinformation in Table XI, it is possible to observe that, ingeneral, team members spent more than 40% of their timein view and interaction activities and less than 10% inediting activities. In detail, almost half of the DEs’ timeis spent in interacting, around 40% for looking at whathas been done and around 10% for editing. Contrariwise,KEs spent most of their time looking at and trying to




understand what has been done (≈ 60%), almost 40% forinteracting and finally for editing. The wiki collaborativefunctionalities, though leaving unchanged such a generaland role-specific order in the proportion among the threetypologies of activities, cause an overall decrease in theaverage time of editing operations (≈ 7.77 sec. for NCMand ≈ 5.66 sec. for CM per editing operation).

Such a decrease in terms of time spent in editingoperations, however, does not imply a decrease in termsof ontology quality. Figure 5, indeed, besides confirmingthe existence of a difference between the NCM and CMentity lifecycles (RQ5), also shows that the CM entitiesare overall built according to a process more structuredthan the one used for building NCM entities. In detail, thelifecycle of the entities built with NCM does not revealthe existence of any strict precedence relations betweenthe description editing (DE) and the entity axiomatization(AxE), while, among the patterns characterizing the CMentity lifecycle, there exists one in which the entity creation(CR), is followed by a phase of description editing (DE)and then by one of axiomatization (AxE), thus showinga gradual enrichment of the ontology entities first withthe unstructured and then with the structured content. Thisresult is not in contrast with the less rigid process of on-tology construction characterizing the CM process (RQ4).In collaborative modelling settings, it is reasonable that astructured entity editing process (as in case of CM entitylifecycle) is carried out in an environment in which teammodellers are constantly aware of what other members ofthe team are doing and interact each other for collaborationpurposes, i.e., in which modellers frequently interleave editactivities with view and interaction ones.

C. Cofactors AnalysisBesides the level of use of the wiki collaborative func-

tionalities, we also investigated other possible factors in-fluencing the experiment results. In detail, we applied theANOVA test [10] to analyse the influence of the laboratory,of the object (i.e., the domain to be formalized), as well asthe experience of both DEs and KEs. For each subject weanalysed the role-specific knowledge and experience (i.e.,in the pedagogical domain and in formalizing ontologies,respectively), the technical experience (e.g., in using wikipages and the latex2owl language, respectively), as wellas the experience of working in teams and with KEsand DEs, respectively. We only found an influence of thelaboratory on the number of produced axioms, on the totalnumber of operations and on the number of switches amongactivity typologies. This can be explained with the increasedexperience of the teams in the second laboratory session.Nevertheless, we tried to limit the impact of such a learningeffect by applying a balanced design.

D. Threats to validity.We present the main threats to validity affecting the

conducted study, grouped by category [10].Conclusion Validity. Conclusion validity deals with therelation between the treatment and the outcome. In orderto ensure such a validity, since not all the preconditions

required by parametric statistical tests held in our study,we used non-parametric tests (the Wilcoxon and Mann-Whitney tests) for our analysis of the main factor. ANOVAwas instead used for the analysis of the cofactors. Thoughit is parametric and thus it would require the satisfactionof the requirements for the application of the parametricstatistical tests, it is a robust test and part of its results isalso checked against the outcomes of the non-parametricWilcoxon test. For the evaluation we chose to use bothobjective and subjective metrics. The first type, providesa real and robust measurement of the performance of thetwo approaches. However, since our goal is to evaluate thesupport provided by the wiki collaborative features to thecollaborative authoring, we believe that also the subjectiveperception has to be taken into account. To this aim, weresorted to personal judgements about the effectiveness ofthe proposed features, their ease of use, their usefulnessand, their support for each specific level of interaction. Weused standard settings and scales to apply statistical teststo the collected data.Internal Validity. Internal validity threats concern externalfactors that could affect the dependent variables. By per-forming an analysis of the possible cofactors (by means ofANOVA), we found that some of them have an influence onthe dependent variables (e.g., the actual usage of the wikicollaborative features), as well as the laboratory session.Nevertheless, the adoption of the balanced design limitedthe influence of the laboratory on the obtained results.Construct Validity. Construct validity concerns the rela-tionship between theory and observation: possible threats,hence, mainly relate to the lack of (i) variable represen-tativeness and (ii) measure reliability in the study opera-tionalization.

In our study, possible construct validity threats falling inthe first group, could be raised by the restrictions imposedby the need to guarantee reliable variable measures inboth synchronous and asynchronous settings, while limitinginvasiveness. Indeed, although in usual scenarios ontologymodelling requires weeks of work and team modellers canremotely communicate through both written and verbalcommunication means, in this study, the time available forcarrying out the work is limited (2 hours) and dividedin sessions of synchronous and asynchronous work, andthe communication means restricted to emails, chats anddiscussions, thus originating possible threats. Nevertheless,teams were free to focus on the core parts of the domainand hence able to perform the task within the time limit;moreover, the verbal communication is only possible insynchronous settings and less suitable than written com-munication to limited-time contexts [19].

In order to limit possible threats falling in the secondgroup, instead, we carefully measured the collected data.The number of axioms and operations carried out byteam members, as well as, log information about subjectsand lifecycle activities, were objectively and accuratelymeasured and traced in the systems, respectively. Finally,though subjective, personal judgements too were measuredby means of standard scales.External Validity. External validity is related to the gen-eralization of the findings. The number of subjects, of




members per team, as well as of teams (and hence thenumber of points considered), was not high (12 peoplein 4 teams). Nevertheless, the involved subjects were realdomain experts (professionals with at least 2 or moreyears of experience in the specific domain as well as withthe concrete need of formalizing their knowledge) andknowledge engineers with at least 1 year of experience inthe field of knowledge representation. Moreover, such smallteams, as we experienced several times in past and on-goingreal projects (see e.g., [8]), is rather typical in SME (SmallMedium Enterprise) contexts, where limited resources areallocated for this kind of tasks. This makes our results reallyclose to a real and generalizable scenario. A second externalvalidity threat is related to the usage of MoKi. Nevertheless,MoKi represents a good prototype of the wiki tools forwhat concerns the collaborative features. Indeed, most ofits collaborative features (e.g., the discussion, the historyand the notification mechanisms) are implemented exactlyas typically done in wikis.

VI. RELATED WORKS

Several approaches to foster collaboration in ontologyengineering were presented. [2] presents a methodologyand tool (OntoEdit, later evolved in OntoStudio14) for col-laborative ontology development. [1] introduces a holisticapproach to collaborative ontology development based onchange management. In [3], a web-based ontology editor(iCat) is applied to the collaborative revision and extensionof version 11 of the WHO International Classification ofDiseases (ICD-11). In the last ten years, taking advan-tage of the wiki popularity and ease of use principles, anumber of wiki systems (e.g., Semantic MediaWiki+ [20],IkeWiki [21], OntoWiki [7], MoKi [8]) were developed tosupport the collaborative authoring of structured content,including ontologies. To support a more active involvementof domain experts in ontology editing, some attemptswere also made by exploiting controlled subsets of naturallanguage (e.g., ROO [4], ACEWiki [22]).

Some works have investigated the role and effectivenessof tools and methodologies in the process of collaborativeontology modelling. For instance, in [4] and [22], theevaluation of the effectiveness of the tools in supportingthe domain experts in the ontology authoring is carried outin a non-collaborative setting or limiting the collaborationonly to domain experts. In [1], a preliminary analysis ofthe application of the proposed methodology and tool ona real life scenario is reported. Three ontology editorsplaying (two) different roles were asked to perform veryspecific tasks on a shared ontology, thus providing insightsmore on the usability of the tool than on the process ofcollaborative modelling. In [23], instead, the iCat authorspropose a work more in the flavour of our empiricalevaluation, although focused on different aspects. Theypresent a tool, iCat Analytics, for the exploration of theontology engineering process and use it for providing somepreliminary insights about the collaborative construction ofthe ICD-11 ontology.

14http://www.semafora-systems.com/en/products/ontostudio/

Further works analysed the collaborative aspects in on-tology modelling. [24] describes an observation study with13 users on the support provided by Collaborative Protegeto address requirements (e.g., concurrent editings, trackchanges, discussion mechanisms) for collaborative ontologydevelopment. In [25], authors presented an ethnographicstudy to identify how the process of ontology developmentunfolded in practice during the course of the building ofa cell-type ontology. In [26], a set of indicators is pro-posed and applied to understand the social arrangements incommunity-based ontology evolution. In [27], the authorsanalysed the experience of applying an agile method forthe community development of the Software Ontology. [28]investigates the implicit roles of authors in collaborativeontology modelling, and analyses the relationship betweenontology changes and how users communicate.

In this paper we substantially extend the work presentedin [11] by providing a wider perspective of the findingsand a finer-grained analysis of the support provided by thewiki collaborative features to the collaborative authoring.In detail, we introduce a theoretical analysis of the contri-bution of wiki collaboration features with respect to socialinteraction aspects; we empirically investigate the impact ofthe collaborative features on both the resulting process ofcollaborative modelling and the ontology entity lifecycle(two additional research questions have been introduced,RQ4 and RQ5); we provide a deeper analysis and discus-sion of the mined results, and we finally carefully analysepossible threats in our findings.

We conclude by pointing out that, although the tools andapproaches recalled at the beginning of this section maynot exactly rely on the same wiki collaborative featuresthat we considered in our work, the results and insights ofthe empirical and theoretical evaluation that we performedare relevant also for them, suggesting easy extensions forwhat concerns the collaboration support aspects.

VII. CONCLUSION

The rigorous theoretical analysis and empirical evalu-ation presented in this paper shows that wiki collabora-tive features for ontology authoring, by actively involvingdomain experts in the authoring process and supportingthe interaction of modellers with other team members,effectively support and affect the process of collaborativeontology authoring, as well as the lifecycle (and possiblythe quality) of the built ontology entities.

This result on one side highlights the support provided bywiki collaborative features in actively involving DEs in the(collaboratively) building of ontologies; on the other side,it encourages other collaborative non-wiki based tool toextend their functionalities adopting these simple but usefulcollaborative features.

Starting from results and feedbacks obtained in thisanalysis, we plan to further investigate how the supportprovided by wiki authoring features can be improved forspecific interaction levels (e.g., decision making), as well ashow users can be guided (e.g., by means of good practices)in the process of collaborative modelling so as to improveboth the effective collaboration and the resulting ontology.




ACKNOWLEDGMENT

We thank Dr. Vaschetto and Dr. Cramerotti for their sup-port in arranging the evaluation, as well as the members ofFBK and Edizioni Erickson who took part to the experimentfor their availability and the valuable feedback provided.

REFERENCES

[1] R. Palma, O. Corcho, A. Gomez-Perez, and P. Haase, “A holisticapproach to collaborative ontology development based on changemanagement,” Web Semantics: Science, Services and Agents on theWorld Wide Web, vol. 9, no. 3, 2011.

[2] Y. Sure, M. Erdmann, J. Angele, S. Staab, R. Studer, and D. Wenke,“Ontoedit: Collaborative ontology development for the semanticweb,” in 1st Int. Semantic Web Conf., Springer, 2002, pp. 221–235.

[3] T. Tudorache, S. M. Falconer, N. F. Noy, C. Nyulas, T. B. stn, M.-A. D. Storey, and M. A. Musen, “Ontology development for themasses: Creating icd-11 in webprotege,” in 17th Int. Conf. EKAW2010, ser. LNCS, vol. 6317. Springer, 2010, pp. 74–89.

[4] V. Dimitrova, R. Denaux, G. Hart, C. Dolbear, I. Holt, and A. G.Cohn, “Involving domain experts in authoring owl ontologies,” in7th Int. Semantic Web Conference (ISWC 2008). Springer, 2008.

[5] D. Bagni, M. Cappella, M. T. Pazienza, and A. Stellato, “Congas:A collaborative ontology development framework based on namedgraphs,” in XIth Int. Conf. of the Italian Association for ArtificialIntelligence. Springer, 2009, pp. 42–51.

[6] M. Krotzsch, D. Vrandecic, M. Volkel, H. Haller, and R. Studer,“Semantic wikipedia,” Journal of Web Semantics, vol. 5, pp. 251–261, 2007.

[7] S. Auer, S. Dietzold, and T. Riechert, “Ontowiki - a tool for social,semantic collaboration,” in Proc. of the 5th Int. Semantic Web Conf.,vol. 4273. Springer, 2006, pp. 736–749.

[8] C. Ghidini, M. Rospocher, and L. Serafini, “Modeling in a wikiwith moki: Reference architecture, implementation, and usages,”International Journal On Advances in Life Sciences, vol. 4, no. 4,pp. 111–124, 2012.

[9] J. Mendling, J. C. Recker, and J. Wolf, “Collaboration features incurrent bpm tools,” EMISA Forum, vol. 32, no. 1, pp. 48–65, 2012.

[10] C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson, B. Regnell, andA. Wesslen, Experimentation in software engineering: an introduc-tion. Norwell, MA, USA: Kluwer Academic Publishers, 2000.

[11] C. Di Francescomarino, C. Ghidini, and M. Rospocher, “Evaluat-ing wiki-enhanced ontology authoring,” in The 18th InternationalConference on Knowledge Engineering and Knowledge Management(EKAW2012), Galway, Ireland, 2012.

[12] B. Leuf and W. Cunningham, The Wiki way: quick collaboration onthe Web. London: Addison-Wesley, 2001.

[13] J. J. Jung, “Semantic wiki-based knowledge management system byinterleaving ontology mapping tool,” Int. J. of Software Engineeringand Knowledge Engineering, vol. 23, no. 01, pp. 51–63, 2013.

[14] Wikimedia Foundation, “Mediawiki,” http://www.mediawiki.org,Last Accessed on 6 Nov 2011.

[15] K. Kaljurand and N. E. Fuchs, “Verbalizing owl in attempto con-trolled english,” in 3rd Int. Workshop on OWL: Experiences andDirections, Innsbruck, Austria, vol. 258, 2007.

[16] T. W. Malone and K. Crowston, “The interdisciplinary study ofcoordination,” ACM Computing Surveys, vol. 26, pp. 87–119, 1994.

[17] C. Gutwin and S. Greenberg, “A descriptive framework of workspaceawareness for real-time groupware,” Computer Supported Coopera-tive Work, vol. 11, pp. 411–446, 2002.

[18] H. M. W. Verbeek, J. C. A. M. Buijs, B. F. v. Dongen, and W. M. P.v. d. Aalst, “Prom 6: The process mining toolkit,” in Proc. of BPMDemo Track 2010, vol. 615. CEUR-WS.org, 2010, pp. 34–39.

[19] M. C. Pichiliani and C. M. Hirata, “Analyzing the use of voip tech-nology in collaborative modeling,” in 13th Int. Conf. on ComputerSupported Cooperative Work in Design. IEEE Comp. Soc., 2009.

[20] D. Hansch and H.-P. Schnurr, “Practical applications of semanticmediawiki in commercial environments - case study: semantic-based project management,” in 3rd European Semantic TechnologyConference (ESTC2009), 2009.

[21] S. Schaffert, “Ikewiki: A semantic wiki for collaborative knowledgemanagement,” in 1st Int. Ws. on Semantic Technologies in Collabo-rative Applications (STICA’06), 2006.

[22] T. Kuhn, “Controlled English for Knowledge Representation,” Ph.D.dissertation, University of Zurich, 2010.

[23] J. Poeschko, M. Strohmaier, T. Tudorache, N. F. Noy, and M. Musen,“Pragmatic analysis of crowd-based knowledge production systemswith icat analytics: visualizing changes to the icd-11 ontology,” inAAAI Spring Symposium on Wisdom of the Crowd, 2012.

[24] D. Schober, J. Malone, and R. Stevens, “Observations in collabora-tive ontology editing using collaborative protege,” in Workshop onCollaborative Construction, Management and Linking of StructuredKnowledge, 2009.

[25] D. Randall, R. Steven, W. Sharrock, R. N. Procter, Y. Lin, andM. Poschen, “What about sea urchins? collaborative ontology build-ing among bio-informaticians,” in 5th International Conference one-Social Science, June 2009.

[26] P. D. Leenheer, C. Debruyne, and J. Peeters, “Towards socialperformance indicators for community-based ontology evolution,” inWorkshop on Collaborative Construction, Management and Linkingof Structured Knowledge, 2009.

[27] M. Copeland, A. Brown, H. E. Parkinson, R. Stevens, and J. Malone,“The swo project: A case study for applying agile ontology engi-neering methods for community driven ontologies,” in ICBO 2012,ser. CEUR-WS, 2012.

[28] S. Falconer, T. Tudorache, and N. F. Noy, “An analysis of collabo-rative patterns in large-scale ontology development projects,” SixthInternational Conference on Knowledge Capture, K-CAP, vol. 11,pp. 25–32, 2011.

Chiara Di Francescomarino is a Post-Docresearcher at Fondazione Bruno Kessler (FBK)in the Data and Knowledge Management (DKM)unit. She received her laurea degree in ComputerScience at University of L’Aquila, Italy, in 2003and her PhD in Information and CommunicationTechnologies at FBK and University of Trento,Italy, in 2011. She is currently working in thefield of business processes: on their enrichmentwith semantic knowledge (and the reasoningservices that can be provided on top of them),

as well as on their reverse engineering from execution traces. Moreover,her research interests include the collaborative modelling and the empiricalevaluation of tools and techniques for its support.

Chiara Ghidini is a Senior Research Scientist atFondazione Bruno Kessler within the Data andKnowledge Management (DKM) research unit.Dr. Ghidini received her PhD from the Universityof Rome “La Sapienza” in 1998 and subse-quently worked at the Manchester MetropolitanUniversity, and at the University of Liverpool.Her work in the area of distributed knowledgerepresentation is well known and internation-ally recognized and she has published over onhundred conference and journal papers on the

topics. She has served on many organising and program committeesfor conferences and workshops in the areas of multi-agent systems andthe semantic web, as general chair of the 2nd European workshop onmulti-agent systems (EUMAS 2004), and as programme chair of the 4thInternational and Interdisciplinary Conference on Modeling and UsingContext (CONTEXT 2003). She has been involved in a number of inter-national research projects, including the EU-funded projects APOSDLEand Organic.Lingua and is currently leading the interdisciplinary SHELLproject on procedural and ontological knowledge acquisition and evolutionat the Fondazione Bruno Kessler.

Marco Rospocher is a research scientist atFondazione Bruno Kessler, within the Data andKnowledge Management (DKM) research unit.He received his PhD in Information and Com-munication Technologies from the University ofTrento in 2006. His current research interests arein the area of Semantic Web and KnowledgeRepresentation, focusing in particular on ontolo-gies, formalisms for Knowledge Representationand Reasoning, and methodologies and tools forKnowledge Acquisition. He has been involved

in a number of international research projects, including the EU-fundedprojects APOSDLE, PESCaDO, and NewsReader. He served as a programcommittee member in several international conferences and workshops.

evaluating wiki collaborative features in ontology authoring

Documents