usability of keyword-driven schema-agnostic search - a comparative study of keyword search, faceted...

Click here to load reader

Post on 11-May-2015

333 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

  • 1.Usability of Schema-agnostic Search ESWC 2010 paper presentationThanh Tran1, Tobias Math2, Peter Haase21 AIFB(KIT)2 fluid Operations GmbHKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)1

2. Outline Motivation - Complex Information Needs Overview - Schema-agnostic Search Frontend Conceptual Study of Schema-agnostic SearchApproaches from Process-oriented Point of View What are the steps? What are the actions required by the user? What is the state of the art? Experimental Study of Schema-agnostic SearchSystems using Task-based ExperimentsKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)2 3. Motivating Scenario:Complex information needs on the Web of Data Growing number of triples and links Web of linked data Complex information needs (entity search, factsearch, and particularly relation search)Mary is a novice computer science student at KIT. She is eager to learn more aboutthis vast research field and decided to find information about research work ofresearchers at AIFB. How to express complex information needs? Formal structured query languages require knowledgeabout the underlying data schemas Web data is dynamically evolving ( z ). x, y.type(y,researcher )worksAt(y x) name(x, AIFB),researchWork(y,z)KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)3 Tran Thanh: Schema-agnostic Search 4. Contributions We identified a particular class of searchparadigms that we refer to as schema-agnosticsearch frontends From a process-oriented viewpoint, we conducteda theoretical study of four widely used approachesrepresenting the class of keyword-driven schema-agnostic search We implemented these approaches under onecommon framework and carried out anexperimental evaluation to assess efficiency,effectiveness and usabilityKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)4 Tran Thanh: Schema-agnostic Search 5. Schema-agnostic Search Frontend Schema-agnostic frontend the user does not know the schema Schema-agnostic search approaches Application-specific form-based interfaces Generic form-based interfaces: faceted search & browsing Graphical query interfaces Natural language interfaces IR-style keyword search DB-style keyword search Computing queries vs. computing complex results Auto-completion: query vs. result completionKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)5 6. Keyword-driven Schema-agnostic Search Process Search paradigm from the process point of view A search process that starts with the use ofkeywords followed by result inspection,refinement and further browsing, and ends withthe retrieval of final results.1. IR-style keyword search + resource-based browsing2. IR-style keyword search + faceted-based browsing3. DB-style keyword search + result completion4. DB-style keyword search + query completionKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)6 Tran Thanh: Schema-agnostic Search 7. IR-style Keyword Search + Resource-based Browsing Information about research work of researchers at AIFB SearchInitial InspectionFurther Inspection, Browsing and RetrievalKeywords k2,k3Result Set FacetsElena SimperlPublication Result Setk1=publication,AIFBs Publication 1k2=researcher,Andreas HarthNameRudi StuderAddress AIFBs Publication 2K3=aifbThanh Tran Projectk1,k2,k3Query Result Tuple SetResource Result Settype(y,researcher), Publication 1 AIFB HarthTran Trans Publication 3worksAt(y,x), Publication 2 AIFB StuderTrans Publication 4name(x,AIFB), Publication 3 AIFB Tranpublication(y,z)Publication 4 AIFB TranSearchInitial InspectionFurther Inspection, Browsing and Retrieval FacetedSearchFacet-basedFacet-basedResult Set Keyword SearchResult SetExpansion &Inspection Keyword InspectionRefinementSearchResultCompletion Query SetResult Tuple SetResource Resource-based Inspection Inspection InspectionNavigation QueryCompletionKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)7 Tran Thanh: Schema-agnostic Search 8. IR-style Keyword Search + Faceted Browsing Information about research work of researchers at AIFB SearchInitial InspectionFurther Inspection, Browsing and RetrievalKeywords k2,k3Result Set FacetsElena SimperlPublication Result Setk1=publication,AIFBs Publication 1k2=researcher,Andreas HarthNameRudi StuderAddress AIFBs Publication 2K3=aifbThanh Tran Projectk1,k2,k3Query Result Tuple SetResource Result Settype(y,researcher), Publication 1 AIFB HarthTran Trans Publication 3worksAt(y,x), Publication 2 AIFB StuderTrans Publication 4name(x,AIFB), Publication 3 AIFB Tranpublication(y,z)Publication 4 AIFB TranSearchInitial InspectionFurther Inspection, Browsing and Retrieval FacetedSearchFacet-basedFacet-basedResult Set Keyword SearchResult SetExpansion &Inspection Keyword InspectionRefinementSearchResultCompletion Query SetResult Tuple SetResource Resource-based Inspection Inspection InspectionNavigation QueryCompletionKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)8 Tran Thanh: Schema-agnostic Search 9. DB-style Keyword Search + Result Completion Information about research work of researchers at AIFB SearchInitial InspectionFurther Inspection, Browsing and RetrievalKeywords k2,k3Result Set FacetsElena SimperlPublication Result Setk1=publication,AIFBs Publication 1k2=researcher,Andreas HarthNameRudi StuderAddress AIFBs Publication 2K3=aifbThanh Tran Projectk1,k2,k3Query Result Tuple SetResource Result Settype(y,researcher), Publication 1 AIFB HarthTran Trans Publication 3worksAt(y,x), Publication 2 AIFB StuderTrans Publication 4name(x,AIFB), Publication 3 AIFB Tranpublication(y,z)Publication 4 AIFB TranSearchInitial InspectionFurther Inspection, Browsing and Retrieval FacetedSearchFacet-basedFacet-basedResult Set Keyword SearchResult SetExpansion &Inspection Keyword InspectionRefinementSearchResultCompletion Query SetResult Tuple SetResource Resource-based Inspection Inspection InspectionNavigation QueryCompletionKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)9 Tran Thanh: Schema-agnostic Search 10. IR-style Keyword Search + Faceted BrowsingInformation about research work of researchers at AIFBSearchInitial InspectionFurther Inspection, Browsing and Retrieval Keywordsk2,k3Result Set Facets Elena SimperlPublication Result Set k1=publication,AIFBs Publication 1 k2=researcher,Andreas HarthName Rudi StuderAddress AIFBs Publication 2 K3=aifb Thanh Tran Project k1,k2,k3 Query Result Tuple SetResource Result Set type(y,researcher), Publication 1 AIFB HarthTran Trans Publication 3 worksAt(y,x), Publication 2 AIFB StuderTrans Publication 4 name(x,AIFB), Publication 3 AIFB Tran publication(y,z)Publication 4 AIFB Tran SearchInitial InspectionFurther Inspection, Browsing and Retrieval Faceted Search Facet-basedFacet-based Result SetKeyword SearchResult SetExpansion & Inspection KeywordInspectionRefinement Search Result CompletionQuery SetResult Tuple SetResource Resource-basedInspection Inspection InspectionNavigation Query Completion KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)10 Tran Thanh: Schema-agnostic Search 11. Experimental Study: Objectives Given complex needs, we aimed to assess whether users are able to address them(effectiveness) how much effort they have to invest to addressthem (efficiency) and how is their experience as a whole (toobtain initial usability results) KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)11 Tran Thanh: Schema-agnostic Search 12. Experimental Study: Information WorkbenchKeywords Query & Result compeltions Term Completions FacetsVorlesung Knowledge Discovery - Institut AIFBKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH) 13. Experimental Study: Information Workbench KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)13 14. Experimental Study: Tasks Nine tasks of varying complexity All tasks were solvable Tasks aborted when not solvable with reasonable effort / when more than 3 minutes are needed TaskDescription Search Type Task 1Find The Beatles (the band) Entity Search Task 4Find the occupation of Barack Obama Fact Search Task 9Find all people whose birthplace is Albany, together with Relation Search their deathplace KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)14 Tran Thanh: Schema-agnostic Search 15. Experimental Study: Overall Setting 19 participants DBpedia together with YAGO as a domain- independent dataset Top-10 queries for query completion and top-25 results for result completion Standard commodity PCs, Intel Pentium Core2Duo with 2 x 2Ghz CPU and 3 GB RAM All actions taken by the participants and the system responses were recorded as screencasts Study results based on these screencasts and questionnaires KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)15 Tran Thanh: Schema-agnostic Search 16. Experimental Study: EffectivenessPercentage of Unsolved Tasks50.00keyword search40.00faceted search30.00result completion20.00query translation10.00 -Entity-Queries Fact-Queries Tuple-Queries No difference for entity search Slight difference for fact search Largest difference for relation search Keyword search requires large browsing effort to go through all entities DB-style query completion best (5 % failure) DB-style result completion requires inspecting large number of results& asking queries with very specific keywords KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)16 Tran Thanh: Schema-agnostic Search 17. Experimental Study: Efficiency Average Time Needed (in seconds)120.00Entity-Queries100.00Fact-Queries 80.00Tuple-Queries 60.00Overall 40.00 20.00- KeywordsearchFaceted Search Result-Completion Query-Completion Time needed for keyword search and faceted search increases rapidly with the complexity of the information needs. No significant increase for completion-based approaches Slight differences: result completions faster than query completions in some cases KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)17 Tran Thanh: Schema-agnostic Search 18. Experimental Study: Usablity Most popular 14 12 10 8 6 4 2 0Query-Translation Result-Completionno preferencesEase of Query Completion Ease of Result Completion 10 109988776655443322101 0 very prettyfairpretty very very pretty fair pretty very easyeasyhard hardeasyeasyhardhardKIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)18Tran Thanh: Schema-agnostic Search 19. Conclusions Schema-agnostic search is essential for searching the large and evolving Web of data! Keyword-driven schema-agnostic search is natural evolution of Web search, from simple document search to entity search to complex relation search that address complex information needs Simple IR-style keyword search is sufficient for simpleinformation needs (entity and fact search) Faceted search not superior when need is specific Query- and result completion have advantages forcomplex needs (relation search) Query completion more effective and usable whileresult completion slightly more efficient KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)19 Tran Thanh: Schema-agnostic Search 20. Thanks! Tran Duc [email protected]://sites.google.com/site/kimducthanh/KIT die Kooperation von Forschungszentrum Karlsruhe GmbH und Universitt Karlsruhe (TH)