04/11/23 Danica Damljanović 1

Natural Language Interfaces to Ontologies: usability and performance(Transfer report)

Student: Danica Damljanović

Supervisor: Prof. Dr Hamish Cunningham

University of Sheffield NLP

Introduction

• Semantic Web

• Ontologies and knowledge bases Semantic search

• Semantic Web interfaces

University of Sheffield NLP

404/11/23 Danica Damljanović

Semantic Search Interfaces

• Protégé (Noy et al., 2001)• KIM (Kiryakov et al., 2004)• TAP (Guha et al., 2003)

GetData Reflection TAPSearch

• SemSearch (Lei et al., 2006)• NLIs to KBs

University of Sheffield NLP

Danica Damljanović

User preferences

(Kaufmann and Bernstein, 2007)

• Natural Language Interfaces preferred to keywords, menu-guided, and graphical interfaces

(Linckels, 2007): • keywords preferred

to NL interfaces

University of Sheffield NLP

604/11/23 Danica Damljanović

NLIs to KBs: challenges

• Robustness

• Portability

• What to show?

• Understanding information need

• Habitability

• Usability

University of Sheffield NLP

704/11/23 Danica Damljanović

Natural Language Interfaces

• Usability Effectiveness Efficiency User satisfaction Who uses NLIs?

• Application developers: customisation• End users: search

University of Sheffield NLP

804/11/23 Danica Damljanović

Customisation and Retrieval Performance

University of Sheffield NLP

Customisation of NLIs to KBsOntology editing (e.g. using Protege)

Domain lexicon

NLI for querying

…

Domain knowledge

WordNet

Domain expert

Ontology engineerNLI for Ontology authoring

University of Sheffield NLP

1004/11/23 Danica Damljanović

End users point of view

• Can the User and the System speakthe same language?

University of Sheffield NLP

1104/11/23 Danica Damljanović

Design recommendations• System Vocabulary>>User Vocabulary:

Feedback Guided interfaces Personalised vocabulary

• User Vocabulary>> System Vocabulary: Clarification dialogs Query refinement Controlling the relevance Ranking suggestions Defining similarity

• User profiles

University of Sheffield NLP

1204/11/23 Danica Damljanović

First experiments: Question-based Interface to Ontologies

University of Sheffield NLP

1404/11/23

Danica Damljanović

NL --> SeRQL query

University of Sheffield NLP

An Example

1.15

1.19

compare

University of Sheffield NLP

Danica Damljanović

Evaluation: Initial results

We evaluated:

• coverage and correctness

• scalability and portability

University of Sheffield NLP

2004/11/23 Danica Damljanović

Comparison with Aqualog

• We removed 6 questions that we knew were not supported by Aqualog• 1 conjunction query “What are the run parameters of POS Tagger and Sentence splitter?”• 1 query with brackets “Does GATE have a coreference resolution component (PR)?”• 1 query starting with “How many. . . ”• 3 queries not in a form of a full-blown question, for example “I cannot get Wordnet plugin to work“.

University of Sheffield NLP

Danica Damljanović

Evaluation on scalability

University of Sheffield NLP

2404/11/23 Danica Damljanović

Expected contributions and research plan

• Addressing challenges without compromising retrieval performance: How existing NLIs to KBs can increase the

performance without a significant additional cost for customisation?

How methods for assisting the user in formulating the query can influence the retrieval performance?

• Richer navigation and semantic search interface which extend the concept of faceted search Improving QuestIO

University of Sheffield NLP

Expected contributions and research plan

• Evaluation with golden standard dataset which would be one of the contributions of this PhD: Knowledge base Set of queries Set of relevant SPARQL queries

• Exploring methods for usability and their impact on retrieval performance

University of Sheffield NLP

Related publicationsJournals:• Danica Damljanovic, Kalina Bontcheva: Towards Enhanced Usability of Natural Language

Interfaces. Annals of IS: Special Issue on “Semantic Web & Web 2.0”. Springer. 2009. (to appear)

Workshops:• Damljanovic, D.,Bontcheva, K.. Enhanced Semantic Access to Software Artefacts. In Workshop

on Semantic Web Enabled Software Engineering (SWESE) held in conjunction with ISWC'08, Karlsruhe, Germany, October 2008.

Conferences: • Tablan, V., Damljanovic, D., Bontcheva, K.. A natural language query interface to structured

information. In Proceedings of the 5h European Semantic Web Conference (ESWC 2008), Tenerife, Spain, June, 2008.

• Damljanovic, D., Tablan, V., Bontcheva, K.: A text-based query interface to owl ontologies. In: 6th Language Resources and Evaluation Conference (LREC), Mar- rakech, Morocco, ELRA (May 2008).

Posters: • Natural Language Queries for Enhanced Knowledge Access presented at Summer School on

Multimedia Semantics Analysis, Annotation, Retrieval and Applications (SSMS07) , Glasgow, UK, July 15-21, 2007.

University of Sheffield NLP

Thanks• …you for your attention!• Hamish Cunningham and Kalina Bontcheva for

continuous support• Valentin Tablan for the help with QuestIO• Vanesa Lopez from Open University for allowing

me to play with AquaLog• Professor Abraham Bernstein and Esther

Kaufmann from the University of Zurich for sharing with me the Mooney dataset in owl format.

• Milan Agatonovic for continuous support

University of Sheffield NLP

References• (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M.

Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001.

• (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004.

• (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM.

• (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006.

• (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007.

• (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.