Benchmarking Faceted Browsing Capabilities of Triple Stores

Horizon 2020GA No 688227

01/12/2015 – 30/11/2018

Henning Petzka, Claus Stadler, Georgios Katsimpras, Bastian Haarmann, Jens Lehmann

13.09.2017

SEMANTiCS Amsterdam 2017

HOllistic Benchmarking of Big lInked daTa

Rationale:A community-driven unified benchmarking platform for the community

• Focus on Big Linked Data• Provide benchmarks and baselines• Provide reference implementation of KPIs• Extensible and referenceable• Result analysis• Open Source

http://project-hobbit.eu

Platform Overview

• Benchmarks I: Generation & Acquisitionmeasures performance of SPARQL query processing systems when faced with streams ofdata in terms of efficiency and completeness

• Benchmarks II: Analysis & Processingtest performance on instance matching tools for Linked Data and performance on machinelearning methods for data analytics

• Benchmarks III: Storage & Curationhas its focus on storage components and versioning systems to efficiently manage evolvinglinked datasets

• Benchmarks IV: Visualization & Serviceshas its focus on benchmarks regarding question answering and faceted browsing.

Faceted Browsingstands for a session-based and state-dependent

interactive method for query formulation over a multi-dimensional information space.

A browsing scenario consists of applying (or removing) filter restrictions defined by object-valued properties or of changing the range of a property value of various data types.

[Google Shopping]

Faceted Browsing - Example

Faceted Browsing - Example

[Google Shopping]

Choke Points

! In a browsing scenario it is the efficient transitionfrom one state to next one that determines the user

experience !

Three basic types of transition

1. Class-based transition2. Property- or property path-based transition3. Entity type switch

Choke Points

We collected a list of 14 choke points:

The underlying dataset

Scenarios• make sense in a real-world browing scenario and• cover all types of transitions as specified by the choke points

Key Performance Indicators

• Instance retrieval:• Query-per-second score• Precision• Recall• F1-Score

• Facet counts:• Query-per-second score• Several metrics for accuracy

Over all queries and for each choke pointindividually

MOCHA Challenge at ESWC 2017

Benchmark on Faceted Browsing was part of theMighty Storage Challenge at the ESWC 2017

Two participants vs. baseline system• QUAD by Ontos• Virtuoso 8.0 Commercial Edition (beta release)

vs. Virtuoso 7.2 Open-Source Edition

No results for QUAD due to time out.

Preliminary results

Georgala, Spasic, Jovanovik, Petzka, Röder, Ngonga Ngomo. MOCHA2017: The Mighty Storage Challenge at ESWC 2017, ESWC challenge proceedings (Springer)

Problems for generic solutionsDependency issue

Problems for generic solutionsNon-changing transitions

Problems for generic solutionsTree-based transitions

Thank you!

http://project-hobbit.eu https://twitter.com/hobbit_project

This work was supported by grants from the EU H2020 Framework Programme provided for theproject HOBBIT(GA no. 688227).