opengovintelligence workshop at ntts2017

NTTS2017,16March2017,Brussels

Hands-onWorkshoponLinkedOpenStatisticalData

213-17March2017,BrusselsNTTS2017

Time Duration Session

18:30 – 18:40 00:10 WelcomeaddressChristine Kormann (Eurostat)

18:40- 18:50 00:10 The OpenGovIntelligence ProjectEvangelosKalampokis(UniversityofMacedonia)

18:50– 19:05 00:15 LODin theESS:initiatives,enablersandchallenges- aPwCstudyforEurostatNikolaosLoutas (PwC)

19:05– 19:20 00:15 Theuseof LinkedOpenStatisticalDataintheFlemishGovernmentPaulHermans (ProXML)

19:20– 19:30 00:10 OpenGovIntelligence ToolsBillRoberts(Swirrl)

19:30– 20:30 00:60 Hands-onevaluationofthetools

AgendaOverview









AgendaOverview

NTTS2017,16March2017,Brussels

TheOpenGovIntelligenceProject


OpenGovIntelligence§ Consortium

§ 7Europeancountries§ 6R&Dpartners§ 6pilotpartners

§ Duration§ February2016– January2019

http://OpenGovIntelligence.eu

@OpenGovInt


§ Public administration publishes Open Data in an ad-hoc manner based onexisting processes, according to their mandate, and often under unclearlicenses. They also design and deliver services in a top-down manner.

§ On the other hand, society has needs and data-driven public services, notraw data, can address these needs.

§ As a result, society should be involved in service co-production to ensurethat public services address their needs.

Motivation– OpenData


§ Atthesametime,themajorpartofOGDisofstatisticalnature,meaningthatconsistofnumericvaluesthatarehighlystructured.

§ Thistypeofdataisahugebutyetunexploitedresource,whichcouldpotentiallyplayasignificantroleinthecreationofvalueforthesociety.

Motivation– OpenStatisticalData


§OpenStatisticalDataarefragmented

§ Searchingdata.gov.uk for“unemployment”datasets:§ 122results(linksandfiles)§ Theseresultsprovideaccessto56filesand610links

§ Theselinksleadto18otherportals

§ Throughthemtomorethan2000otherfiles

Motivation– Fragmentation

E.Kalampokis,E.Tambouris,A.Karamanou,K.Tarabanis (2016)OpenStatistics:TheRiseofanewEraforOpenData?,EGOV2016,LNCS9820,pp.31-43,Springer.


§ Allthesewebportalsprovidecomplementary viewsoftheunemploymentdata.

§ Forexample,focusingongeodimension:§ DataaboutunemploymentindifferentadministrativelevelsintheUK.

§ ONS,NOMIS,NeSS andOpenDataCommunitiesprovidedataaboutthewholecountry.

§ Localgovernmentportalsprovidedataforspecificareas(e.g.Warwickshire,Cambridgeshire)

Motivation– ComplementarityLevel0 UK ONS

Level1 Countries ONS

Level2 Regions ONS,NOMIS,NeSS,

Level3 Counties NOMIS

Level4 Districts/Boroughs/Divisions ODC

Level5 Local Enterprise Parttnership ONS,NOMIS

Level6 LocalAuthorities/Communities

FirstAreas

ONS,NOMIS,NeSS

Level7 ParliamentaryConstituencies ONS,NOMIS

Level8 Wards Warkwickshire,

Cambridgeshire

Level9 MarketTowns Cambridgeshire

Level10 SuperOutputArea Warkwickshire

Level11 SuperOutputAreaMiddleLayer NeSS

Level12 SuperOutputAreaLowerLayer NeSS

Level13 OutputArea NeSS

Level14 Parishes Cambridgeshire


§ LinkedOpenStatisticalData(akalinkeddatacubes)havethepotentialtoaddressdatainteroperabilityandfacilitatedataintegrationontheWeb.

§ Performinganalyticsontopofmultipledatasets(previouslyisolated)inordertocreateaddedvalueservices.

LinkedOpenStatisticalData

http://lod-cloud.net


LOSDInnovationEcosystem

DataProvider PublicServiceProvider ServiceConsumer

PAs ProvisionofOpenGovernmentData

Designanddeliverofpublicservice

Providepublicservices

Inpolicymakingand/orinternaldecisionmaking

Businesses Businessdata(private)tobeusedinservices

Co-designand/orco-deliverofpublicservice

Inbusinessintelligence,decisionmakingetc.

Citizens/NGOs

Citizenprovideddata Co-designand/orco-deliverofpublicservice

Informationprovision,transparencyetc.


OpenGovIntelligence tools

JSONQBAPI

Table2qb

DataCube

Builder

DataCube

Explorer

DataCube

Aggregator

Cube

Visualizer

OLAP

Browser

Grafter


OpenGovIntelligence approach

JSONQBAPI

CustomappsDataCubeExplorerCubeVisualizerOLAPBrowser

UI

Businesslogic

UI

Businesslogic

UI

Businesslogic

PublishingRecommendations


OpenGovIntelligence approach

JSONQBAPI

CustomappsDataCubeExplorerCubeVisualizerOLAPBrowser

UI

Businesslogic

UI

Businesslogic

UI

Businesslogic

PublishingRecommendations

Userfriendliness

Dataintegration


Pilotsin6EuropeanCountries

EnvironmentalplanninginBelgium

ManaginggovernmentvehiclesinGreece BusinessplanninginLithuania

RealestateinEstoniaSearchandrescueinIreland

Worklessness inTrafford


OpenPublicReviewPilot

https://ec.europa.eu/futurium/en/openpublicreview


Thankyou!!!

http://OpenGovIntelligence.eu

@OpenGovInt

https://medium.com/opengovintelligence









AgendaOverview

LOD in the ESS: initiatives, enablers and challenges - a PwC study for Eurostat

www.pwc.be

NTTS 2017 - Hands-on workshop on Linked Open Statistical Data

Brussels, 17 March 2017

Nikolaos LoutasPwC Data & Analytics

PwC

Outline

• Scope and approach of the study

• LOD initiatives in the ESS

• Value propositions and benefits

• Statistical LOD customer segments

• Key resources for implementing statistical LOD

• Means of dissemination

• Costs

• Enablers and good practices

• Roadblocks and challenges

2

PwC

Key Partnerships

Key Activities

Value Propositions

Customer Segments

Cost Structure

Revenue Systems

Key Resources

Channels

Scope and approach

Desk research

On site visits

Interviews

3

The identification and study of LOD initiatives in the ESS to identify use cases, benefits, good practices, enablers and challenges, which provided input to a proposal for a joint LOD strategy for the ESS.

All credits to businessmodelgeneration.com

PwC

LOD initiatives in the ESS

4

Central Statistics Office (CSO)

Institut national de la statistique et des études économiques (INSEE)

Office for National Statistics (ONS)

Statistics Scotland

Federal Statistical Office (FSO)

Istituto nazionaledi statistica(ISTAT)

PwC

• Interconnect several official statistics datasets (covering both data and metadata) housed in different databases, data stores and data warehouses within a NSI.

• Interconnect several official statistics datasets (covering both data and metadata) housed in different databases, data stores and data warehouses of different NSIs and/or Eurostat.

• Publish official statistics in a linkable, machine-readable format, which can easily be reused and integrated with other types of data, e.g. geospatial, weather, etc.

6

Why are NSIs using LOD

1

2

3

PwC

A selection of statistical LOD use cases

• LOD for territorial bases (ISTAT, Italy)

• Selecting the best place to live or to invest (MaynoothUniversity)

• LOD for fact-checking• Finding data for a postcode

(ONS Geography)• Accessing and querying census

data (CSO, Ireland)• Evolution of Swiss communes

(FSO, Switzerland)

7

• Scottish Index of Multiple Deprivation (Scottish Government)

• Relate/correlate different sources which provide information about a specific domain (EvangelosKalampokis)

• Providing catalogues of linked metadata of open datasets (EU Open Data Portal and European Data Portal)

• ModernStats - Linked Open Metadata (UNECE)

• Integrated access to EU and BEA data (Eurostat and BEA)

• Digital Agenda Scoreboard (DG CONNECT)1 2 3

Interconnect datasets within a NSI

Interconnect official statistics datasets from different NSIs and/or

Eurostat

Publish official statistics in machine-readable, linkable

formats

PwC

LOD value propositions & benefits

National Statistical Institutes

� Having a unified view over data, thanks to easier integration;

� More flexible means of data dissemination and wider outreach;

� Increased standardisation, interoperability andcollaboration opportunities;

� Easier to innovate and evolve;

� Cost reductions, collect and publish once, reuse many times.

Data reusers

� Using the right data at the right time in the right format;

� Better understanding of the data as the data and the model are closely interwoven;

� Increased trust, thanks to traceability and provenance;

� Easier integration with other data from various domains;

� Enhanced data exploration by navigating the links;

� Innovation.

8

PwC

Statistical LOD customer segments

9

Businesses

Publicadministrations

NGOs

NSIs and Eurostat

Data journalists

Academia and researchers

Citizens

PwC

Key resources for implementing statistical LOD

Technology &Infrastructure

Building blocks frequently used:

• Data preparation • SPARQL endpoint• LOD portal

• JSON-stat API• REST API• Data browsers

10

Web standards, such as HTTP URIs and RDF Data standards, such as SDMX, RDF Data Cube and StatDCAT-AP.

Data & Metadata

Skill may be available in house, outsourced or a combination of both. • Technical skills (e.g. PHP, JAVA, data management, data quality)• LOD knowledge (e.g. LOD principles, standards and technologies)• Communication and promotion (e.g. people able to communicate

the why and get buy-in)• Statistical knowledge

People & Capabilities

Linked Data Governance

Define overall priorities with respect to the main value propositionPerforming common analysis and on-going evaluation Data licensing: most common licence is Creative Commons Attribution 4.0.URI policy: to guarantee persistence, resolvability, and uniformity of Web identifiers.

PwC

Key resources for implementing statistical LOD

Key partners

11

Industry Outsourcing technical development, consultancy

Academia Knowledge and expertise sharing, common projects, tools development

Key activities

Requirements Pre-implementation analysis, on-going evaluation

DevelopmentSelection of data, creation of tools to transform, link, publish and visualise data

Maintenance Governance, management, user support

Promotion Communication and publicity

PwC

Means of statistical LOD dissemination

Channels

12

NSIs portals Browser-based access to LOD, e.g. intuitive link navigation

Endpoint/API SPARQL, REST, URI dereferencing

Mobile Apps

Customer relationships

Contests Hackathons, app contests, prizes

Feedback Customer support, CRM

PwC

LOD cost structure

13

Development Subcontracting creation of toolsIn-house development

Maintenance Technical maintenance, only limited as in most cases LOD is in pilot phase

Promotion Communication and publicity, contests, prizes

Licensing Licensing for LOD tools in case open source solutions are not used

PwC

Enablers and good practices for implementing statistical LOD

• Flexible way of integrating data, with minimum impact on current infrastructure

• Allows to access data at different levels of granularity – from data points to datasets

• Promotes the need for data standardisation

• Opportunities for new data-enabled services

• Ease of data navigation via browsing the links (URIs)

• Ease of model updates because of the flexibility of RDF

• Emerging best practice guidance

• Creates partnerships between public administration, academia, standards organisations and industry

• Identify clear use cases, target users and benefits

• Start small, think big

• Use a trial and error approach with well-defined iterations

• Look for support and knowledge from the community and collaborate

• Rely on standards for data and metadata –contribute to standardisation discussions

• Provide different ways of accessing the data, from APIs to visual interfaces, to cater for different types of users

• Provide persistent URIs and open licensing

• Measure the use and the expected benefits

14

Enablers Good practices

PwC

Roadblocks and challenges for implementing statistical LOD

• Low awareness within the ESS of the technology and its benefits

• Insufficient promotion of success stories and implemented use cases

• Perceived lack of users’ demand for LOD

• Lack of management buy-in and support

• Organisational resistance because of changes in dissemination of official statistics (new technology, new data formats, new data standards…)

• Perceived scarcity of necessary skills and competencies in the ESS, combined with limited training opportunities and resources

• Proliferation of so-called standards . The ESS is not partaking in the development of standards for LOD in statistics

• Very limited collaboration and knowledge sharing between NSIs in statistical LOD

15

PwC

Get in touch with us to know more

16

Nikolaos [email protected]

Daniel Brulé[email protected]

This publication has been prepared by PwC EU Services and is reporting on a study delivered for Eurostat under DI07171 specific contract 353. “PwC” refers to PwC Enterprise Advisory bvba which is a member firm of PricewaterhouseCoopers International Limited, each member firm of which is a separate legal entity.

Eurostat Project Officer: [email protected]









AgendaOverview

[email protected]

L(O)SDatdep.LNE,FlemishGov.


Context


§ Supportingdecisionmakingonenvironmentalpermitsandinspections§ Asacitizen Iwouldliketoknowwhichemissionsarehappeninginour/aneighborhood.

§ AsacivilservantIwanttoknowthealreadyreportedemissionsinaneighborhoodtoevaluatenewemissionpermitrequestsforthatsameareaandplanemissioninspectionbasedonpreviousreportingtoenhanceefficiency.

§ Asacompany Iwanttocomparemyemissionvalueswithsimilarcompanies.

Objective


§ IMJV(integralenvironmentalyearreport)§ AirEmissions§ WaterEmissions§ Waste§ …

§ CBB(companyregistry)§GPBV,IntegratedPollutionPreventionandControl(IPPC)§ KBO§NIS/NACE§ Supportingcontrolledvocabularies

Data


Observation

measure

dimension

dimension

dimension

attribute

attribute


Application


Analytics


Q&A









AgendaOverview

NTTS2017,Brussels,16March2017

OpenGovΙntelligence ToolsBillRoberts


Standards!


§Fordatamodel§Forstatisticaldimensionvalues(‘codelists’)§ForAPIandquerymechanisms


§Easierdiscovery§Re-usabletools§Combinedatafromdifferentsources


OpenGovIntelligence tools

JSONqb API

Table2qb

DataCubeBuilder

DataCubeExplorer

DataCubeAggregator

CubeVisualizer

OLAPBrowser

Grafter


Publishingtools

DataCubeBuilder

Table2qb

Defineandusecommonpublishing

practices

Data Cube


LinkedStatisticalDataApplications

Data Cube

Customapps

DataCubeExplorerCubeVisualizerOLAPBrowser

UI

Businesslogic

Dataaccess

SPARQLendpoint

UI

Businesslogic

Dataaccess

UI

Businesslogic

Dataaccess


JSONqb API

JSONqb API

Customapps

DashboardsCharts,mapsDatabrowserandsearchtools

Data Cube

SPARQLendpoint

Re-useexistingJavaScriptlibrariese.g.D3.js,PivotTable.js

§ SupportwebdevelopersusestatisticaldatastoredintheformofanRDFDataCube

§ AssumeminimalknowledgeofLinkedData


§ EnablestheexplorationofanRDFdatacubebypresentingatwo-dimensionalsliceofthecubeasatable

§ UsestheJSONqbAPI

OLAPBrowserAPIcall:getmetadata

APIcall:getdata

APIcall:getcubes


CubeVisualizer§ Presentsagraphicalrepresentationofaone-dimensionalsliceofacube

§ UsestheJSONqbAPI

APIcall:getmetadata

APIcall:getdata


DataCubeExplorer§ Summarizes,analysesandvisualizesdatacubes

§ Availablecharts:barchart,ScatteredChart,AreaChart,HeatMap

§ UsestheJSONqbAPI

APIcall:getmetadata

APIcall:getdata


§ Formoreinformation§ https://github.com/OpenGovIntelligence§ http://www.opengovintelligence.eu/

OpenGovintelligence Tools


GETdimensionsParameters:dataset(required)Sampleresult:

GETdimension-valuesParameters:dataset(required),dimension(required)Sampleresult:

JSONqb API- metadata

[{"@id":"http://purl.org/linked-data/sdmx/2009/dimension#sex","label":"sex"},{"@id":"http://example.com#timePeriod","label":"Time Period"},

{"@id":"http://example.com#refArea","label":"Reference Area”}]

{"dimension":{"URI":"http://example.com#timePeriod,"label":"Time Period”}"values":[

{"@id":"http://example.com/concept/year2004#id", "label":"2004"},

{"@id":"http://example.com/concept/year2005#id", "label":"2005"},{"@id":"http://example.com/concept/year2006#id", "label":"2006"},

...]}

Allthedimensionsofacube

Allthevaluesofadimensionthatappearataspecificcube


GETtable

Parameters:col (required),row(required),measure(required),lockeddimensions(optional)Sampleresult:

JSONqb API- data

{"structure":{"free_dimensions":{

"timePeriod":{"@id":"http://example.com#timePeriod","label": "Time Period"},

"refArea":{"@id":"http://example.com#refArea","label":"Reference Area"}},

"locked_dimensions":{

"sex":{"@id":"http://purl.org/linked-data/sdmx/2009/dimension#sex","label":"sex",

"lockedValue":{"@id":"http://purl.org/linked-data/sdmx/2009/code#sex-F","label":"sex-F"}}},

"dimension_values":{

"refArea":{

"S12000033":{"@id":"http://statistics.gov.scot/S12000033","label":"Aberdeen City“},

"S12000034":{"@id":"http://statistics.gov.scot/S12000034","label":"Aberdeenshire“}…},

"timePeriod":{

"year2004":{"@id":"http://example.com/concept/year2004#id","label":"2004"},

"year2005":{"@id":"http://example.com/concept/year2005#id","label":"2005"}…}}},

"headers":{"columns":{"refArea":["S12000033","S12000034“,..."]},

"rows":{"timePeriod":["year2004", "year2005",..."]}},

"data":[[73.4,79.6, ...], [76.6,78.8]]}

Τablerepresentationofthecube’sobservationsthatmatchtoparticularcriteria









AgendaOverview


PublishMyData Demo


http://statistics.gov.scot


Employmentvslifeexpectancyatbirth