hobbit in a nutshell - edf2016
TRANSCRIPT
HOBBITin a Nutshell
Axel Ngonga
Horizon 2020GA No 688227
01/12/2016–30/11/2018
Joint Event Post-EDF 2016Eindhoven, Netherlands
July 1st, 2016
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 1 / 17
A Lot of Data
1
1http://www.ibmbigdatahub.com/infographic/four-vs-big-dataNgonga (InfAI) HOBBIT in a nutshell July 1st, 2016 2 / 17
A Lot of Tools
2
2https://cdn.datafloq.com/cms/os_big_data_open_source_tools-v2.pngNgonga (InfAI) HOBBIT in a nutshell July 1st, 2016 3 / 17
Core Questions
Developers: How good is my tool?Vendors: Who is my tool good for?Users: Which tool(s) should I use formy application?
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 4 / 17
Many Questions
Where are the current bottlenecks?Which steps of the data lifecycle arecritical?Which solutions are available?Which key performance indicatorsare relevant?How well do or should toolsperform?How do existing solutions performw.r.t. relevant indicators?
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 5 / 17
GERBIL
Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instance athttp://gerbil.aksw.org/
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 6 / 17
GERBIL
Evaluation platform for NER/NEL9 reference annotation systems11 reference datasetsBenchmarking 10× fasterArchiving of resultsCiteable URIsAdditional analysisOpen-source projectLocal deploymentNormalized implementation of KPIsOnline instance athttp://gerbil.aksw.org/
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 6 / 17
GERBIL
Annotator TasksNIF-based Annotators 2519Babelfy 958DBpedia Spotlight 922TagMe 2 811WAT 787Kea 763Wikipedia Miner 714NERD-ML 639Dexter 587AGDISTIS 443Entityclassifier.eu NER 410FOX 352Cetus 1
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 7 / 17
HOBBIT
Rationale
A community-driven benchmarking framework for the community
Focus on Big Linked DataCover all steps of the Linked Data lifecycle
Used by a growing number of companiesMature and maturing technologies
Open benchmarks based on industrial dataand use cases
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 8 / 17
HOBBIT
Rationale
A community-driven benchmarking framework for the community
Focus on Big Linked DataCover all steps of the Linked Data lifecycle
Used by a growing number of companiesMature and maturing technologies
Open benchmarks based on industrial dataand use cases
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 8 / 17
HOBBIT
36-month projectProject begin: Dec. 1st, 2015Project volume: ca. 4 million Euros10 partners
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 9 / 17
Aims
1 Gather real requirementsPerformance indicatorsPerformance thresholds
2 Provide universal benchmarking platformStandardized hardwareComparable results
3 Develop benchmarks based on real data4 Periodic benchmarking challenges5 Periodic reporting6 Found independent Hobbit association
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 10 / 17
Overview
Data Collection
Industrydata
Measure Collection
Benchmark Creation
Benchmark 1
KPIsTasks
KPIsTasksKPIsTasks
KPIsTasks
KPIsTasks
KPIsTasks
Benchmark 2
Benchmark n
HOBBITPlatform
Solution 1
Solution k
Solution 2
Challenges
Reports
Participants/Community
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 11 / 17
We offer a benchmarking platform
Controller
Data Generator
Task Generator
Data Generator
Data Generator
Task Generator
Task Generator
FrontendSystem Adapter
System
data flowcreates component
Store
SPARQL Endpoint
Analysis
BenchmarkEvaluator Module
Eval. Store
Message BusNode Observer
Logging
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 12 / 17
We offer a benchmarking platform
Addresses all steps of the LinkedData LifecycleBenchmarks derived from industryuse casesReal data under the benchmarksScalable size of benchmarksOpen-source implementationLocal instance on server clusterUses established deploymenttechnologies
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 13 / 17
We offer benchmarks
Streaming and static deterministic benchmarksRealistic benchmarksControlled volume and velocity
Generation and AcquisitionConversion of XML into RDFEntity recognition and linkingRelation extraction
Analysis and ProcessingLink DiscoveryMachine LearningSupervised and unsupervised
Storage and CurationTriple storesVersioningIncl. updates
Visualization and ServicesQuestion AnsweringFaceted BrowsingUsage-based benchmarks
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 14 / 17
We offer datasets
Twitter7 datasetca. 476 million tweetsca. 17 million users
ClueWeb12ca. 733 million websites1+ billion annotations
Printing Machineryca. 6.5 trillion events1500 printing machines
LIVEDca. 2.5 billion measurements6 households, two years
Injection molding industryca. 120 million measurements
Traffic data archiveca. 15 trillion speed measurements100+ million road segments
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 15 / 17
We need ...
Your use casesParticipate in the surveyJoin the HOBBIT communityProvide KPIsProvide datasetsJoin the platform development
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 16 / 17
Thank You
http://project-hobbit.eu/get-involved/
http://goo.gl/forms/1iRIoG4Xpb
https://twitter.com/hobbit_project
Ngonga (InfAI) HOBBIT in a nutshell July 1st, 2016 17 / 17