hobbit in a nutshell - edf2016

HOBBITin a Nutshell

Axel Ngonga

Horizon 2020GA No 688227

01/12/2016–30/11/2018

Joint Event Post-EDF 2016Eindhoven, Netherlands

July 1st, 2016

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 1 / 17

A Lot of Data

1

1http://www.ibmbigdatahub.com/infographic/four-vs-big-dataNgonga (InfAI) HOBBIT in a nutshell July 1st, 2016 2 / 17

A Lot of Tools

2

2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.pngNgonga (InfAI) HOBBIT in a nutshell July 1st, 2016 3 / 17

Core Questions

Developers: How good is my tool?Vendors: Who is my tool good for?Users: Which tool(s) should I use formy application?

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 4 / 17

Many Questions

Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available?Which key performance indicatorsare relevant?How well do or should toolsperform?How do existing solutions performw.r.t. relevant indicators?

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 5 / 17

GERBIL

Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instance athttp://gerbil.aksw.org/

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 6 / 17

GERBIL

Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instance athttp://gerbil.aksw.org/

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 6 / 17

GERBIL

Annotator TasksNIF-based Annotators 2519Babelfy 958DBpedia Spotlight 922TagMe 2 811WAT 787Kea 763Wikipedia Miner 714NERD-ML 639Dexter 587AGDISTIS 443Entityclassifier.eu NER 410FOX 352Cetus 1

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 7 / 17

HOBBIT

Rationale

A community-driven benchmarking framework for the community

Focus on Big Linked DataCover all steps of the Linked Data lifecycle

Used by a growing number of companiesMature and maturing technologies

Open benchmarks based on industrial dataand use cases

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 8 / 17

HOBBIT

Rationale

A community-driven benchmarking framework for the community

Focus on Big Linked DataCover all steps of the Linked Data lifecycle

Used by a growing number of companiesMature and maturing technologies

Open benchmarks based on industrial dataand use cases

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 8 / 17

HOBBIT

36-month projectProject begin: Dec. 1st, 2015Project volume: ca. 4 million Euros10 partners

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 9 / 17

Aims

1 Gather real requirementsPerformance indicatorsPerformance thresholds

2 Provide universal benchmarking platformStandardized hardwareComparable results

3 Develop benchmarks based on real data4 Periodic benchmarking challenges5 Periodic reporting6 Found independent Hobbit association

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 10 / 17

Overview

Data Collection

Industrydata

Measure Collection

Benchmark Creation

Benchmark 1

KPIsTasks

KPIsTasksKPIsTasks

KPIsTasks

KPIsTasks

KPIsTasks

Benchmark 2

Benchmark n

HOBBITPlatform

Solution 1

Solution k

Solution 2

Challenges

Reports

Participants/Community

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 11 / 17

We offer a benchmarking platform

Controller

Data Generator

Task Generator

Data Generator

Data Generator

Task Generator

Task Generator

FrontendSystem Adapter

System

data flowcreates component

Store

SPARQL Endpoint

Analysis

BenchmarkEvaluator Module

Eval. Store

Message BusNode Observer

Logging

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 12 / 17

We offer a benchmarking platform

Addresses all steps of the LinkedData LifecycleBenchmarks derived from industryuse casesReal data under the benchmarksScalable size of benchmarksOpen-source implementationLocal instance on server clusterUses established deploymenttechnologies

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 13 / 17

We offer benchmarks

Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity

Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction

Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised

Storage and CurationTriple storesVersioningIncl. updates

Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 14 / 17

We offer datasets

Twitter7 datasetca. 476 million tweetsca. 17 million users

ClueWeb12ca. 733 million websites1+ billion annotations

Printing Machineryca. 6.5 trillion events1500 printing machines

LIVEDca. 2.5 billion measurements6 households, two years

Injection molding industryca. 120 million measurements

Traffic data archiveca. 15 trillion speed measurements100+ million road segments

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 15 / 17

We need ...

Your use casesParticipate in the surveyJoin the HOBBIT communityProvide KPIsProvide datasetsJoin the platform development

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 16 / 17

Thank You

http://project-hobbit.eu/get-involved/

http://goo.gl/forms/1iRIoG4Xpb

https://twitter.com/hobbit_project

Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 17 / 17