Query Processing and Reasoning

How Useful are Natural Language Interfaces tothe Semantic Web for Casual End-users?

Esther Kaufmann and Abraham Bernstein

Presented By Stephen Lynn

Query Processing and Reasoning

Overview Natural Language Interfaces Goals/Objectives Introduce 4 Interfaces Experiment Evaluation Results Future Work

Query Processing and Reasoning

Natural Language Interfaces Plain text queries

PhrasesFull Sentences

ChallengesLinguistic Variability (ambiguous meaning)Domain IndependenceRetrieval Performance (linked to portability)Usefulness of NLIs

Query Processing and Reasoning

Goals/Objectives

Usability of NLIs

Usefulness of NLIs

Query Processing and Reasoning

Evaluation Interfaces

PortableDomain-IndependentGood Performance4 Interfaces

Least to Most Restrictive

Query Processing and Reasoning

NLP-Reduce Free-form text query Remove Stop Words/Puncuation Word Stemming Identify Triple Structures (no details)

Enhanced Triple Store (WordNet) Generate SPARQL Return Results

Query Processing and Reasoning

NLP-Reduce

Query Processing and Reasoning

Querix Parse Query Extract Query Skeleton from Syntax Tree Identifies Triple Patterns Match Triples to Knowledge Base Resources Generate SPARQL

Enhanced with WordNet Synonyms Return Results

Query Processing and Reasoning

Querix

Query Processing and Reasoning

Querix – Ambiguity Resolution

What is the biggest state in the US?

Query Processing and Reasoning

Ginseng UI based on a grammar

Built dynamically from target knowledgebases Incremental Parser Offer possible completions (code completion)

Only accepts entries in listNo invalid queries

Convert to SPARQL Return Results

Query Processing and Reasoning

Ginseng

Query Processing and Reasoning

Symantic Crystal Graphical Display of Ontology Select Elements in Ontology

No Invalid Queries Specify Constraints Incrementally Build Query Generate SPARQL Return Results

Query Processing and Reasoning

Semantic Crystal

Query Processing and Reasoning

Usability Study How usable and useful are NLI applications? Setup

48 subjects4 interfacesSame 4 questions for each interface (minor changes)

Area of Alaska? Number of lakes in Florida? States that have city named Springfield? Rivers run through state that has largest city in US?

Change sequence of interfaces

Query Processing and Reasoning

Experiment1. Read Introduction Notes2. Instructions on Interface #13. Answer 4 questions with interface4. Fill out Usability survey about Interface5. Repeat 2-4 for other Interfaces6. Fill out Comparison Questionnaire

Query Processing and Reasoning

Evaluation Results

Query Processing and Reasoning

Evaluation Results

Query Processing and Reasoning

Strengths Good General Points

Automation is good (not Sematic Crystal)Result format affects user trustBalance between freedom and restriction

User Evaluation Analysis

Query Processing and Reasoning

Weaknesses Completion time not a deciding factor in satisfaction Still pushing Semantic Crystal

Personal Attachment Unclear distinction between QL and Interface

Query Processing and Reasoning

Future Work Compare with more NLIs Multiple Domains Single Infrastructure w/Different Uis

Evaluate Usability/Usefulness