hobbit project overview @ eswc hobbit workshop

HOBBITProject Overview

Axel Ngonga

Horizon 2020GA No 688227

01/12/2016–30/11/2018

ESWC 2016Crete, GreeceJune 1, 2016

Axel Ngonga (InfAI) Project Overview June 1, 2016 1 / 13

A Lot of Data

1

1http://www.ibmbigdatahub.com/infographic/four-vs-big-dataAxel Ngonga (InfAI) Project Overview June 1, 2016 2 / 13

A Lot of Tools

2

2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.pngAxel Ngonga (InfAI) Project Overview June 1, 2016 3 / 13

Core Questions

Developers: How good is my tool?Vendors: Who is my tool good for?Users: Which tool(s) should I use formy application?

Axel Ngonga (InfAI) Project Overview June 1, 2016 4 / 13

Many Questions

Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available?Which key performance indicatorsare relevant?How well do or should toolsperform?How do existing solutions performw.r.t. relevant indicators?

Axel Ngonga (InfAI) Project Overview June 1, 2016 5 / 13

GERBIL

Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

GERBIL

Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× faster

Archiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

GERBIL

Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysis

Open-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

GERBIL

Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instanceFeedback for developers and users

Axel Ngonga (InfAI) Project Overview June 1, 2016 6 / 13

GERBIL

Annotator TasksNIF-based Annotators 2519Babelfy 958DBpedia Spotlight 922TagMe 2 811WAT 787Kea 763Wikipedia Miner 714NERD-ML 639Dexter 587AGDISTIS 443Entityclassifier.eu NER 410FOX 352Cetus 1

Axel Ngonga (InfAI) Project Overview June 1, 2016 7 / 13

HOBBIT

Rationale

A community-driven benchmarking framework for the community

Focus on Big Linked DataCover all steps of the Linked Data lifecycle

Used by a growing number of companiesMature and maturing technologies

Open benchmarks based on industrial dataand use cases

Axel Ngonga (InfAI) Project Overview June 1, 2016 8 / 13

HOBBIT

Rationale

A community-driven benchmarking framework for the community

Focus on Big Linked DataCover all steps of the Linked Data lifecycle

Used by a growing number of companiesMature and maturing technologies

Open benchmarks based on industrial dataand use cases

Axel Ngonga (InfAI) Project Overview June 1, 2016 8 / 13

Aims

1 Gather real requirementsPerformance indicatorsPerformance thresholds

2 Develop benchmarks based on real data3 Provide universal benchmarking platform

Standardized hardwareComparable results

4 Periodic benchmarking challenges5 Periodic reporting6 Found independent Hobbit association

Axel Ngonga (InfAI) Project Overview June 1, 2016 9 / 13

Overview

Data Collection

Industrydata

Measure Collection

Benchmark Creation

Benchmark 1

KPIsTasks

KPIsTasksKPIsTasks

KPIsTasks

KPIsTasks

KPIsTasks

Benchmark 2

Benchmark n

HOBBITPlatform

Solution 1

Solution k

Solution 2

Challenges

Reports

Participants/Community

Axel Ngonga (InfAI) Project Overview June 1, 2016 10 / 13

Architecture

Controller

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

FrontendSystem Adapter

System

data flowcreates component

Store

SPARQL Endpoint

Analysis

BenchmarkEvaluator Module

Eval. Store

Message BusNode Observer

Logging

Axel Ngonga (InfAI) Project Overview June 1, 2016 11 / 13

We Offer Benchmarks

Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity

Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction

Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised

Storage and CurationTriple storesVersioningIncl. updates

Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks

Axel Ngonga (InfAI) Project Overview June 1, 2016 12 / 13

Features of the HOBBIT platform

Addresses all steps of the LinkedData LifecycleBenchmarks derived from industryuse casesReal data under the bechmarksScalable size of benchmarksOpen-source implementationOnline instance on server clusterUses established deploymenttechnologies

Axel Ngonga (InfAI) Project Overview June 1, 2016 13 / 13

Join HOBBIT

Participate in the surveyJoin the HOBBIT communityJoin the split sessionsProvide KPIsProvide datasetsJoin the platform development

Axel Ngonga (InfAI) Project Overview June 1, 2016 14 / 13

Thank You

http://project-hobbit.eu/get-involved/

http://goo.gl/forms/1iRIoG4Xpb

https://twitter.com/hobbit_project

Axel Ngonga (InfAI) Project Overview June 1, 2016 15 / 13