Constructing Conceptual Knowledge Artefacts: Activity Patterns in the Ontology Authoring Process

Markel Vigo, Caroline Jay, Robert Stevens

University of Manchester (UK)

CHI 2015, Seoul (Korea)

@markelvigo

markel.vigo@manchester.ac.uk

What are ontologies?

Mad cow

Lion

Giraffe Cow

Animal

Classes

eats

SubClassOf

Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal

What are ontologies?

Mad cow

SubClassOf

Lion Giraffe Cow

Animal

Classes

Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal eats

What are ontologies?

Mad cow

SubClassOf

Lion Giraffe Cow

Animal

Classes

Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal

Vegetarian à Animal and (eats only (not (Animal)))

eats

What are ontologies?

Mad cow

SubClassOf

Lion

Giraffe Cow

Animal Classes

Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal

Vegetarian à Animal and (eats only (not (Animal)))

eats

Vegetarian

What are ontologies?

Mad cow

SubClassOf

Lion

Giraffe Cow

Animal Classes

Properties Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal

Vegetarian à Animal and (eats only (not (Animal)))

eats

Vegetarian

Complex artefacts

•  Large size

•  Domain expertise required

• Critical

• Highly expressive

Problem

We don’t know...

§  if existing ontology authoring tools are effective

§  how ontologist go about authoring

§  the authoring workflows

Authoring environment: Protégé

Authoring environment: Protégé

File menu

Class hierarchy

Property hierarchy

Annotations-Usage

Description

Mad cow

SubClassOf

Lion Giraffe Cow

Animal

Classes

Properties

Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal

Vegetarian à Animal and (eats only (not (Animal)))

eats

File menu

Class hierarchy

Property hierarchy

Annotations-Usage

Description

Pop up

Explanation

Edit entity

Authoring environment: Protégé

Mad cow

SubClassOf

Lion Giraffe Cow

Animal

Classes

Properties

Axioms Cow, Giraffe, Lion à SubClassOf Animal

Mad cow à SubClassOf Cow

Mad cow, lion à Animal eats some Animal

Vegetarian à Animal and (eats only (not (Animal)))

eats

Study •  16 ontology authors, 3 typical tasks •  Protégé à Protégé4US •  Eye-tracking •  Sync and merge interaction + fixation data • N-gram analysis

1389973572771,eye,Class hierarchy!1389973577038,eye,Description!1389973584775,log,EntitySelected!1389973586349,log,DescriptionSelected!1389973598978,log,EntityModified!1389973603166,log,EntitySelected!1389973605053,log,DescriptionSelected!1389973607847,log,EntityModified!1389973616404,eye,Description!1389973616754,eye,Class hierarchy!1389973617221,eye,Description!

Findings: interaction log data

•  Interaction events account for 65% of events while authoring events are 30%

•  The top 3 events (entity selection, description selection and invocation of editing menu) account for 56% of events

61219232839476182

113139142

182199

267314332

426617

9601004

14052793

BackUndo

Get explanationEntity renamed

Set propertyEntity dragged

Property additionEntity deletedLoad ontology

Hierarchy collapsed(i)Save

Description selected(i)Run reasoner

Hierarchy collapsedConvert into defined

Hierarchy expanded(i)Class addition

Entity selected(i)Hierarchy expanded

Entity edited:finishEntity edited:start

Description selectedEntity selected

0 1000 2000

Findings: eye-tracking data

The class hierarchy is the pivotal area •  Index of the ontology •  External memory

Transitions between AOIs

from

to

Ann−Usage

Class hierarchy

Description

Explanation

File menu

Pop up

Edit Entity

Prop. hierarchy

Ann−U

sage

Class h

ierarc

hy

Descri

ption

Explan

ation

File men

u

Pop up

Edit Enti

ty

Prop. h

ierarc

hy0

1000

2000

3000

4000

5000

6000

Findings: eye-tracking data

The class hierarchy receives users’ attention 45% of the time

0

100

200

300

400

File men

u

Ann.−U

sage

Class h

ierarc

hy

Descri

ption

Popup

Edit en

tity

Prop. h

ierarc

hy

Explan

ation

AOI

time

(sec

)

Findings: workflows

Select description

Select entity

0.29 Modify entity

0.37

0.63

0.59

Editing activity

Run reasoner

Convert into defined class

Save

Select description

0.16

0.15

0.17

0.40

Expand inferred

hierarchy

0.30

Select entity

0.41

0.37

0.43

Select inferred entity

0.54

0.25 0.12

Reasoning activity

Select entity

Expand hierarchy

0.48

0.31

Select inferred entity

Expand inferred

hierarchy

0.25

0.43

0.12

0.54

Load ontology

0.52

0.31

Expand hierarchy

Select description

0.29

0.37

Exploration activity Exploration workflow Editing workflow

Reasoning workflow

Implications: from raw data to workflows Workflows can be automatically identified

raw data

cleaning

merging

filtering

workflow detection

~7K rows

~200 rows

•  Different authoring styles •  Time distribution per workflow •  Identification of confounding variables

Implications for design

•  Support for bulk editing

•  Anticipation of reasoner invocation

•  Automatic detection of authoring problems

• Make changes to the inferred hierarchy explicit

tl;dr

•  Identification of activity patterns when dealing with complex interactive artefacts

•  Interaction log data + eye gaze data

•  Data-driven

•  Application on knowledge artefacts

Markel Vigo, Caroline Jay, Robert Stevens

University of Manchester (UK)

@markelvigo

markel.vigo@manchester.ac.uk