ogi d3.4 - report on opengovintelligence ict tools ... - rep… · d3.4 report on...
Post on 10-Jun-2020
16 Views
Preview:
TRANSCRIPT
ThisprojecthasbeenfundedwiththesupportoftheH2020ProgrammeoftheEuropeanUnionãCopyrightbytheOpenGovIntelligenceConsortium
.
OpenGovIntelligence
FosteringInnovationandCreativityinEuropethroughPublicAdministrationModernizationtowardsSupplyingandExploiting
LinkedOpenStatisticalData
Deliverable3.4
ReportonOpenGovIntelligenceICTtools–secondrelease
Leadingpartner: NUIG
Participatingpartners: CERTH,TUT,ProXML,SWIRRL,NUIG
Version-Status: 1.0
Disseminationlevel: PU
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page2of25
DeliverablefactsheetProjectNumber: 693849
ProjectAcronym: OpenGovIntelligence
ProjectTitle:Fostering Innovation and Creativity in Europe through PublicAdministrationModernizationtowardsSupplyingandExploitingLinkedOpenStatisticalData
Deliverabletitle: ReportonOpenGovIntelligenceICTtools–secondrelease
Deliverablenumber: D3.4
Officialsubmissiondate: 31October2017
Actualsubmissiondate: 31October2017
Editor(s):ArkadiuszStasiewicz(NUIG)
MohamedAdel(NUIG)
Author(s):
DimitriosZeginis(CERTH)
AretiKaramanou(CERTH)
EvangelosKalampokis(CERTH)
KonstantinosTarabanis(CERTH)
ArkadiuszStasiewicz(NUIG)
MohamedAdel(NUIG)
PaulHermans(ProXML)
BillRoberts(SWIRRL)
RickMoynihan(SWIRRL)
Reviewer(s) BillRoberts(SWIRRL)
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page3of25
Abstract: Thisdeliverableprovidesdetailsontheprototypesofsoftwarecomponents delivered as a result of the second developmentstageofOpenGovIntelligenceproject.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page4of25
EffortofParticipatingPartners Name ShortName Role PersonMonths
1. CentreforResearch&Technology-Hellas CERTH Participant 1
2. DelftUniversityofTechnology TUDelft None 0
3. NationalUniversityofIreland,Galway NUIG Leader 1.5
4. TallinnUniversityofTechnology TUT Participant 0.2
5. ProXMLbvba ProXML Participant 0.1
6. SwirrlITLimited SWIRRL Participant 0.5
7. Traffordcouncil TRAF None 0
8. FlemishGovernment VLO None 0
9. MinistryofInteriorandAdministrativeReconstruction MAREG None 0
10. MinistryofEconomicAffairsandCommunication MKM None 0
11. MarineInstitute MI None 0
12. PublicInstitutionEnterpriseLithuania EL None 0
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page5of25
RevisionHistoryVersion Date Revisedby Reason
0.1 18-Sep-2017 A.Stasiewicz(NUIG) FirstDraft
0.2 11-Oct-2017 A.Stasiewicz(NUIG) Toolslistupdate
0.3 19-Oct-2017 A.Stasiewicz(NUIG) Generalupdates
0.4 26-Oct-2017 A.Stasiewicz(NUIG) Generalupdates
0.5 31-Oct-2017 A.Stasiewicz(NUIG) Addressinginternalreviewcomments
1.0 31-Oct-2017 CERTH SubmissiontoEC
Statementoforiginality:This deliverable contains original unpublishedwork exceptwhere clearly indicated otherwise.Acknowledgement of previously publishedmaterial andof thework of others has beenmadethroughappropriatecitation,quotationorboth.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page6of25
TableofContentsDELIVERABLEFACTSHEET................................................................................................................2
EFFORTOFPARTICIPATINGPARTNERS...........................................................................................4
REVISIONHISTORY.........................................................................................................................5
TABLEOFCONTENTS......................................................................................................................6
LISTOFFIGURES.............................................................................................................................7
LISTOFTABLES...............................................................................................................................8
LISTOFABBREVIATIONS.................................................................................................................9
EXECUTIVESUMMARY..................................................................................................................10
1 INTRODUCTION....................................................................................................................11
1.1 SCOPE...................................................................................................................................................111.2 AUDIENCE..............................................................................................................................................111.3 STRUCTURE............................................................................................................................................111.4 ADDRESSINGREVIEWER’SCOMMENTS..........................................................................................................11
2 OPENGOVINTELLIGENCEICTTOOLS–OVERVIEW..................................................................13
2.1 TOOLSDEVELOPEDDURINGTHESECONDDEVELOPMENTSTAGE.........................................................................13
3 OPENGOVINTELLIGENCEICTTOOLS-SECONDRELEASE.........................................................15
3.1 ASSISTEDCUBESCHEMACREATOR..............................................................................................................153.2 LOSDMACHINELEARNINGCOMPONENT.....................................................................................................173.3 QBMULTI-DIMENSIONALCHARTING..........................................................................................................183.4 RDFDATACUBEGEODATADASHBOARD....................................................................................................203.5 SPARQLCONNECTORFOREXPLORATORY....................................................................................................213.6 JSONAPIFORDATACUBESPECIFICATION–UPDATE.....................................................................................223.7 JSONAPIFORDATACUBEIMPLEMENTATION-UPDATE.................................................................................223.8 TABLE2QBANDGRAFTER-UPDATE............................................................................................................223.9 DATACUBEAGGREGATOR-UPDATE............................................................................................................23
4 CONCLUSION........................................................................................................................24
REFERENCES.................................................................................................................................25
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page7of25
ListofFiguresFIGURE 1. LOSD TOOL ECOSYSTEM ...................................................................................................................... 14FIGURE 2 OGI LOSD CREATION AND USAGE FLOWCHART – RDF DATA CUBE SCHEMA MAPPING STAGE .......... 17FIGURE 3 QB MULTI-DIMENSIONAL CHARTING DASHBOARD ................................................................................ 19FIGURE 4 LITHUANIAN DATA CUBES GEO BASED VISUALIZATION SCREEN SAMPLES – A...................................... 20FIGURE 5 LITHUANIAN DATA CUBES GEO BASED VISUALIZATION SCREEN SAMPLES – B ...................................... 21
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page8of25
ListofTablesTABLE 1 RECOMMENDATIONS CONCERNING FUTURE WORK .................................................................................. 12TABLE 2. OGI ICT TOOLS TECHNICAL STARTING POINT ........................................................................................ 13
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page9of25
ListofAbbreviationsThefollowingtablepresentstheacronymsusedinthedeliverableinalphabeticalorder.
Abbreviation Description
API ApplicationProgrammingInterface
CMS ContentManagementSystem
ICT InformationandCommunicationTechnologies
LOSD LinkedOpenStatisticalData
OLAP OnLineAnalyticalProcessing
RDF ResourceDescriptionFramework
UI UserInterface
URI UniformResourceIdentifier
WP WorkPackage
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page10of25
ExecutiveSummaryThisdocumentisthedeliverable“D3.4–ReportonOpenGovIntelligenceICTtools–secondrelease”(referred to as D3.4). It provides detailed information about the result of the second stageoftheOpenGovIntelligencedevelopmentinWP3.
ThegoalofWP3(“ICTtoolsdevelopment”)istodeveloptheOpenGovIntelligenceICTtoolsasasuiteof open source and commercial tools. The final version of the developed ICT tools will supporttheOpenGovIntelligenceframeworkcreatedinWP2(“Frameworkcreation”)andwillenable:
a) thecreationofLinkedOpenStatisticalData(LOSD)fromvarioussources,b) theexpansionofLOSDwithdatasetsfromexistingsources,c) theexploitationofLOSDfortheco-productionofpublicservices.
With regards to thecreationofLOSD,OpenGovIntelligencedesignedanddeveloped ICT tools thatenable the transformation of public sector data to standardmachine readable forms (specificallytoRDFdatacubes)andthevalidationofthegeneratedRDF.WithregardstotheexploitationofLOSD,OpenGovIntelligence designed and developed ICT tools that enable visualisation and analysisofstatisticaldata.
While the first release of theOpenGovIntelligence ICT toolswas focused on the tools supportingthecreationandexploitationofLOSD,thenextreleasefocusesonexploitationofLOSDbyprovidingnew tools and improvements (functional and performance) to tools arleardy developed andevaluated.
Thetoolspresentedinthisdeliverablewillbeevaluatedduringthepilotimplementationstage.Basedontheoutcomes,theexistingtoolswillbeimproved.MoreovernewtoolswillbedevelopedinordertocoverremainingfeaturesoftheOpenGovIntelligenceFramework.
The second release of the OpenGovIntelligence ICT tools was guided by challenges and needsidentified in WP1 (D1.1 OpenGovIntelligence challenges and needs), the OpenGovIntelligenceframework (D2.2 OpenGovIntelligence framework) as well as the pilots operation and initialOpenGovIntelligenceevaluation(D4.2Evaluationresults-Firstround).
Results presented in this document are the outputs of Task 3.2 OpenGovIntelligence ICT tools –second release. This deliverable will be updated together with the ongoing dvelopment of theOpenGovIntelligence tools as D3.6 Report on OpenGovIntelligence ICT tools. The final releaseisscheduledforOctober2018.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page11of25
1 IntroductionThissectionintroducesthebackgroundoftheworkcarriedoutinWP3“ICTtoolsdevelopment”.Sub-section1.1presentsthescopeandtheobjectivesofthecurrentdocument,sub-section1.2describesthe intendedaudience for thisdocument, sub-section1.3outlines the structureof thedocument,whilesub-section1.4addressesthecommentsreceivedfromthe1stProjectReview.
1.1 ScopeThisreportdocumentsthe ICTtoolsdevelopedduringthesecondphaseofWP3.Toguidereadersin understanding the context of these tools, Section 2 presents the overviewof the OpenGovIntelligence tools. The Second release provided tools that enables creation andexploitationofLOSD.FinalreleaseofthetoolswillaimtocoverexpansionofexistingdatasetsandexploitationofLOSDforco-productionofpublicservices.
1.2 AudienceTheintendedaudienceforthisdocumentistheOpenGovIntelligenceconsortium,inparticularpartnerorganisationsresponsibleforthedevelopmentofpilottrials,theEuropeanCommission(EC)andthosewhoareinterestedinchallengesandneedsforopening-upandexploitingLOSDfortheco-productionofinnovativedata-drivenservicesongovernments.
1.3 StructureThestructureofthedocumentisasfollows:
• Section2presentstheOpenGovIntelligenceICTtoolsoverview• Section3providesadetaileddescriptionoftheindividualOpenGovIntelligenceICTtools
developedduringtheseconddevelopmentstageofOpenGovIntelligenceproject;• Finally,Section4concludesthereportandoutlinesthefuturedevelopmentplan.
1.4 Addressingreviewer’scommentsThis sub-section address the general comments from the 1st Consolidated Expert Review Report.The Review Report did not include specific issues concerning D3.2. Therefore, in Table 1Recommendationsconcerningfutureworkwepresentrecommendationsoverthefuturework.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page12of25
Table1Recommendationsconcerningfuturework
No. Comment Addressedin
1
It is recommended that future iterations ofD3.2 should make a clear statement aboutthetechnicalstartingpointoftheprojectandwhat has transferred from the DaPaaS andOpenCubeprojects.
DetailsareprovidedinSection2.
2“FigurethatgivesOGIarchitecture isshownin this document, where the same figure isusedinthedeliverableD.2.1(Figure4).”
While D2.1 provides overview of thearchitecture,D3.2explainindetaileach
oftheelements.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page13of25
2 OpenGovIntelligenceICTtools–overviewThissectionaimstoprovidebackgroundinformationrelatedtoLOSDandOpenGovIntelligencetools.
Ingeneral,thedevelopedtoolscoverpartsoftheOpenGovIntelligenceArchitectureandarepartofalooselyconnectedecosystemoftools.TheOpenGovIntelligenceArchitecture(presentedinD3.2)forLOSDanddata-drivenpublicservicesenablesstakeholderstocollaboratetowardstheproductionofinnovative data-driven public services by exploiting LinkedOpen Data technologies and statisticaldatasets.TheinitialversionofthearchitecturewasdocumentedindetailinD3.2andwastestedandrefinedduringthefirstroundofpilotevaluations.Thearchitectureisorganisedasfollows:
• Thearchitectureisdividedintofivelayers:(i)DataProvision,(ii)DataPlatform,(iii)ProcessLayer,(iv)ServiceDesign,and(v)ServiceProvision.
• Keymanagementresponsibilitiesaresharedacrossalllayers.• Eachlayerhasasetofcomponentsthatperformstasksspecifictothatlayer.
Thearchitectureguidesthepilotimplementations,aswellastheotherfutureimplementationsoftheOGIsoftwareinotherprojectstogetherwithOpenGovIntelligenceframework(documentedinD2.2).
2.1 ToolsdevelopedduringtheseconddevelopmentstageIn the first implementation stage, the developed tools were mainly supporting data conversionandexploitationofthedata.ThesecondreleaseofOpenGovIntelligenceincludesadditionaltoolsandsupportsalargerpartoftheOpenGovIntelligenceframework.
While the development of majority of the tools were initialted during the OpenGovIntelligenceproject,someofthemwereusingoutputsoftheDaPaaS1andOpenCube2projectsasthetechnicalstarting point, and subsequently improved and extended within OpenGovIntelligence. Details arepresented in Table 2. OGI ICT Tools technical starting point and visualised in Figure 1. LOSD ToolEcosystem.Also,Table2showswhichtoolsarenewcomponents,startedinyear2,andwhichtoolswerefurtherdevelopedinyear2buildingonworkstartedearlier.
Table2.OGIICTToolstechnicalstartingpoint
No. Name StartedinYear2 Foundation
1 JSONAPIForDataCubeSpecification No OpenGovIntelligence
2 JSONAPIForDataCubeImplementation No OpenGovIntelligence
3 Table2QBAndGrafter No DaPaaS
1https://project.dapaas.eu 2http://opencube-project.eu
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page14of25
4 AssistedCubeSchemaCreator Yes OpenGovIntelligence
5 DataCubeBuilder No OpenGovIntelligence
6 DataCubeExplorer No OpenCube
7 DataCubeAggregator No OpenCube
8 LOSDMachineLearningComponent Yes OpenGovIntelligence
9 OLAPBrowser No OpenCube
10 QBMulti-DimensionalCharting Yes OpenGovIntelligence
11 RDFDataCubeGeoDataDashboard Yes OpenGovIntelligence
12 SPARQLConnectorforExploratory Yes OpenGovIntelligence
Section3documentsthetools,whichweredeliveredduringtheseconddevelopmentstageaswellaslinks by which they can be accessed. In general, tools developed during the project lifecycle canbeaccessedattheGitHubrepositoryavailableat:http://github.com/OpenGovIntelligence
Figure1.LOSDToolEcosystem
Note: The OpenGovIntelligence project will not develop software tools for every componentof the architecture or a single integrated system that matches this architecture as a whole.ImplementationsoftheOpenGovIntelligenceapproachwillcombineaselectionoftoolsandprocessesrelevanttotheusecasestheyaddress,followingtheprinciplessetoutinthearchitecture.Moreover,somepartsofthearchitecturemightendupbeinghumanprocessesoruseappropriatepre-existingtoolsfromoutsideoftheOpenGovIntelligenceproject.Existingtoolsavailableforthepurposewillre-usedwhereeverpossibleandnewtoolswillbedevelopedwhenrequired.Wehavealreadyidentifiedanumberoftoolsthatcanbere-usedforexampleincollaborationspace.ThislistoftoolsisincludedindeliverableD2.1–OpenGovIntelligenceframework–firstrelease.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page15of25
3 OpenGovIntelligenceICTtools-secondreleaseThissectionprovidesadetaileddescriptionoftheindividualOpenGovIntelligenceICTtoolsdevelopedduringtheseconddevelopmentstageof theOpenGovIntelligenceprojectanddetailsof themajorupdates to the tools developed it the first stage of the OpenGovIntelligence project. Previouslydevelopedtoolsdescriptionsareavailableindeliverable“D3.2–ReportonOpenGovIntelligenceICTtools–firstrelease”.ThetoolsdevelopedduringtheprojectlifecyclecanbeaccessedattheGitHubrepositoryavailableat:http://github.com/OpenGovIntelligence.
3.1 AssistedCubeSchemaCreatorLOSD publishing process always starts with the process of mapping of the desired data set. Thisprocess is required to conformwith RDF Data Cube Vocabulary defined concepts and propertiesinordertoenableinteroperability.
In the first stages of LOSD tools experimentation and development schemamapping processwasaccomplishedbyanalysisof theavailabledatasetsandmanualprocessingof themapping.Resultswereusedase.g.forinputsforRDFDataCubeBuildertool.Anychangesofthesourcedatarequiredrepetition of the mapping process. For sustainability reasons, we could not depend on thismethodologyforourLOSDpublishingprocess.
TheanalysisoftheotherLOSDcreationmethodologies345678leadtodevelopmentofAssistedCubeSchemaCreator.Toensuresustainability,thetoolwasdesignedtosupportvarietyofdatasetsanddifferentusecases.Moreover,toolhadtobesimpletousebydifferentusertypes.
3.1.1 FunctionalityDescriptionThispublishingpipelineiscreatedtoeasethemappingprocessofgovernmentstatisticaldatasetsintoRDFaccordingtotheRDFDataCubevocabulary.Thistoolintegratesspreadsheets,OpenRefine9,RDFRefine10andRDFDataCube11toproduceagenericLOSDmappingtoolthatfitsanypilotorusecase.
3.1.2 ImplementationDescription
Thefollowingstepsareincludedinthepublishingpipeline:
3https://github.com/GMDSP-Linked-Data/GenericStatsCube4http://www-etis.ensea.fr/WOD2013/wp-content/uploads/2013/06/Publishing-Census-as-Linked-Open-Data..pdf5http://wifo5-03.informatik.uni-mannheim.de/bizer/r2r/6https://www.w3.org/2001/sw/rdb2rdf/wiki/Implementations7http://wifo5-03.informatik.uni-mannheim.de/latc/toollibrary/screencast.html8https://www.w3.org/TR/r2rml/#definitions 9http://openrefine.org10http://refine.deri.ie/publications11https://www.w3.org/TR/vocab-data-cube
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page16of25
A. First step in themappingprocess is implementedby collectingdata set toRDFDataCubemappingdetailsfromtheuser.Tocollectinformationaboutthemappingdetailsfromusersaspreadsheettemplatewithbuiltinguidelineswasprepared.Thisspreadsheettemplateeasetheinformationcollectionprocessandmakeitsimplerforalltypesofusers.Usersareableto:
a. Startdefiningexistingdatasetheadersandtheirnewnamesifwanted.b. Defining the RDFData Cube types of those headers i.e. dimensions,measures, or
attributes.c. LinkingthoseheaderswithexitinglinkeddataURIthatmatchestheheaderconcept
andproperties.d. Defininglabels,comments,relatedconceptsanddatatypesforthoseelements.e. DefiningtheRDFdataCube’sDataStructureDefinition(DSD)elementbylinkingthe
dimensions,measures,andattributeswithDSD.f. Finally,userdefinetheRDFDataCube’sDataSet (DS)elementthat includeLinked
Dataannotationabilitiesi.e.datasetpublisherURIdefinitions,relatedsubject(s)URIdefinitions,relatedlocation(s)andorganization(s)definitions.Moreover,userdefinesthelabel,descriptionandcommentsfortheDS.
B. ThesecondstepinthemappingprocessiscarriedoutusingOpenRefinewiththeRDFRefineextension.TheRDFDataCubeVocabularystructurewaspredefinedinJSONformatthatcanbeconsumedbyOpenRefinewithassistancefromtheRDFRefineextension.ThisJSONRDFData CubeVocabulary structure is coupledwith the spreadsheet template. The process isdesignedtobeeasytouseforalltypesofusers.ItsimplifiesandminimisesthedatasettoRDF Data Cube mapping and conversion steps. As demonstrated in Figure 2 OGI LOSDCreationandUsageFlowchart–RDFDataCubeSchemaMappingStage, theuserneedstoload,applyandextracttheRDFschemabyfollowingthefollowingsteps:
a. LoadingmappingspreadsheettoOpenRefine+RDFrefineextension:theuserstartsstep2oftheschemacreationprocessbyloadingthespreadsheetcontainingthedatasettoRDFDataCubeVocabularymappingdetailsintotherunningOpenRefineserverasanOpenRefineproject.
b. ApplyingthepredefinedRDFDataCubeSchematotheinputspreadsheet:theusersimply loads the provided JSON file containing the RDF Data Cube VocabularyStructureandcouplesthiswiththespreadsheettemplate.Thisstepisdoneusingthestandard'applyoperations'proceduresavailableonOpenRefine.
c. ExtractingtheschemamappingofthedatasetinRDF/XMLorTTLformats:finally,theuser uses the RDF refine extension to extract the RDF Data Cube schema file inRDF/XMLorTTLformats.ThisfileisproducedbyapplyingtheJSONfileinstructiontothespreadsheet.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page17of25
Figure2OGILOSDCreationandUsageFlowchart–RDFDataCubeSchemaMappingStage
3.1.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-assisted-schema-creator
3.1.4 LicenseThesoftwareisavailableasopensourceundertheMITLicense.
3.1.5 PilotsinvolvedTheIrishMarineInstitute,andEnterpriseLithuaniaaretheinitialusersofthetool.ThistoolisgenericandcanbeusedacrosstheOpenGovIntelligenceConsortium.
3.1.6 Futuredevelopmentplan
Theplansfornextyeardevelopmentinclude:
1. Updatestothespreadsheetfillingguidelines2. Evaluationandfeedbackcollection3. Updatingthetoolwithuserfeedback,andtooltestingresults.
3.2 LOSDMachineLearningComponentMachinelearningenablesthecreationofpredictivemodelsbasedontheanalysisofhighvolumesofdata. Thewealth of the statistical data that is freely available online can contribute towards thisdirection.Themulti-dimensionalnatureofstatisticaldataenablestheextractionofnumerousfeaturesfroma singledatasetbasedon the valueof themeasured variable for thedifferent valuesof thedimensionsorthedifferentaggregatedfunctions(e.g.min,max,averageetc.)thatcanbeappliedtothemeasure.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page18of25
3.2.1 FunctionalityDescriptionTheMachineLearningComponentenablestheautomaticextractionofnumerousfeaturesfromLOSDbasedontheneedsoftheusersandthepredictivescenariothatisimplemented.ItalsoenablestheperformanceofdimensionreductionbasedonrelevantalgorithmssuchasForwardSubset,BackwardSubset,andLassoinauser-friendlyapproach.Inthisway,userswillbeabletoincludebigvolumesofLOSDinmachinelearningscenariosandfilteronlythepredictorsthatarerelevanttothescenarioathand.Thesepredictorscanbelaterusedinthecreationofpredictivemodels.
3.2.2 ImplementationDescription
TheimplementationoftheMachineLearningComponentisbasedontheJSON-QBAPIandRserver.
3.2.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-machine-learning-component
3.2.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense(v2.0).
3.2.5 PilotsinvolvedAtthisstage,thetoolshavenotbeenactivelyusedinthepilotusecases,thusthetoolisgenericandcanbeusedacrosstheOpenGovIntelligenceConsortium.
3.2.6 Futuredevelopmentplan
Theplansfornextyeardevelopmentinclude:
1. Collectingusersfeedbackonthetool.2. Updatingthetoolbasedontheuserfeedback,andtooltestingresults.
3.3 QBMulti-DimensionalChartingQB Multi-Dimensional Charting tool is a multi-dimensional charting dashboard which is usinga JavaScript charting library as a base. It supports cross filtering and allowing highly efficientexplorationon largemulti-dimensional. It leveragesd3enginetorendercharts inCSSfriendlySVGformat.Chartsrenderedusingdc.jsarenaturallydatadrivenandreactivethereforeprovidinginstantfeedbackonuser'sinteraction.
ThemainobjectiveofthistoolistoprovideaneasyyetpowerfulJavaScriptdashboardwhichcanbeutilizedtoperformdatavisualizationforcubedataandanalysisinbrowseraswellasonmobiledevice.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page19of25
3.3.1 FunctionalityDescriptionThistool isprovidinganeasytousemulti-dimensionalchartingdashboardwhichcanbeutilizedtoperformdatavisualization,datafiltering,explorationandanalysisforcubedatainbrowseraswellasonmobiledevice.
Figure3QBMulti-DimensionalChartingdashboard
3.3.2 ImplementationDescription
QBMulti-DimensionalChartingdashboardiswritteninJavaScriptontopofanotherJavaScriptlibrarieslikedc.js,Leaflet.jsandcrossfilter.js;dc.jsisaJavaScriptlibrarythatisusedtocreateinteractivechartsandacorrespondingtable.Moreover,itsupportsaninteractivemap.
3.3.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-multi-dimensional-charting
3.3.4 LicenseThesoftwareisavailableasOpenSourceundertheApache2.0License.
3.3.5 PilotsinvolvedTheIrishMarineInstitute,andEnterpriseLithuaniaaretheinitialusersofthetool.
3.3.6 Futuredevelopmentplan
Theplansfornextyeardevelopmentincludes
1. Furtherintegrationofthetools
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page20of25
2. ReleasingthetoolsasaWordPressplugin
3.4 RDFDataCubeGeoDataDashboardTheRDFDataCubeGeoDataDashboardallowsvisualisationanddiscoveryofgeoannotateddata.RDFDataCubeGeoDataDashboarduses the JSONQBAPI toaccess thegeoannotatedRDFDatastoredinaRDFdatabase.Theretrieveddataarevisualizedasmapbaseddashboardtoenableeaseofinformationunderstandingtoalltypesofusers.
3.4.1 FunctionalityDescriptionThisdashboardisdesignedandimplementedtovisualisegeoannotateddataandsupportpilotspecificusecases.
3.4.2 ImplementationDescription
This toolwaswritten in JavaScript using several libraries and tools as follow: Leaflet.js, GeoJSON,OpenStreetMap;Leafletjsisalightweightopen-sourceJavaScriptlibraryforbuildingmobile-friendlyinteractivemapsthankstoitsstrongmappingfeatures;GeoJSONisaformatforencodingavarietyofgeographicdatastructures,itsupportsthedifferentgeometrytypeslikePoint,Polygon,MultiPoint,andMultiPolygon;OpenStreetMap(OSM)isacollaborativeprojecttocreateafreeeditablemapoftheworld.Figure4andFigure5presentsUserInterface:informationfromtheselectedDataCubearevisualisedasataggedmap.
3.4.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-geo-ui
Figure4LithuanianDataCubesGeobasedVisualizationscreensamples–A
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page21of25
Figure5LithuanianDataCubesGeobasedVisualizationscreensamples–B
3.4.4 LicenseThesoftwareisavailableasopensourceundertheMITLicense.
3.4.5 PilotsinvolvedEnterpriseLithuaniaistheinitialuserofthetool.
3.4.6 Futuredevelopmentplan
Theplansfornextyeardevelopmentinclude:
1. Collectinguserfeedbackonthetool.2. Updatingthetoolbasedontheuserfeedbackandtooltestingresults.
3.5 SPARQLConnectorforExploratorySPARQLconnectorforExploratoryallowstoconnecttheExploratoryDataSciencetooltoaSPARQLendpoint.ThiscomponentisprovidedasRcodetogetherwithaJSONconfigurationfile.Ithasbecomepartofawidelydeployedproduct.
3.5.1 FunctionalityDescriptionThiscomponentenablesimportofdatastoredasRDFintotheExploratorysystemintoExploratory.
3.5.2 ImplementationDescription
ThiscomponentisprovidedasRcodetogetherwithJSONconfigurationfile.
3.5.3 AvailabilityExtensionisavailableat:https://github.com/OpenGovIntelligence/exploratory_sparql_plugin
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page22of25
3.5.4 LicenseForthelicensedetailspleasecheckhttps://exploratory.io/terms
3.5.5 PilotsinvolvedTheFlemishGovernmentistheinitialusersofthetool.
3.5.6 Futuredevelopmentplan
hiscomponenthasbecomepartofawidelydeployedproductandalreadymeetsitsdesignobjectivessonofurtherdevelopmentofthisisrequiredorforeseen.
3.6 JSONAPIforDataCubeSpecification–updateFollowingevaluationofinitialimplementations,itwasdecidedtoimplementasecondversionoftheAPIusingtheGraphQLquerylanguageforAPIs,dueitsfastgrowingadoptioninthewebdevelopercommunityandbecauseitsfeaturesareagoodmatchforourrequirementstosearchforandfilterRDFDataCubes.
ThesetofAPImethodshasbeenre-specifiedusingtheGraphQLapproach,andexpandedasadditionalusefulfunctionshavebeenidentified
TheworkontheAPIisavailableathttps://github.com/Swirrl/graphql-qb
3.7 JSONAPIforDataCubeImplementation-updateDuringyear2oftheproject,theAPIhasbeenimplementedusingGraphQL.Workiscontinuingtoaddnewdataaccessmethods,tooptimiseperformanceandtoimproveuserdocumentation-availableathttps://github.com/Swirrl/graphql-qb
3.8 Table2QBAndGrafter-updateDuringyear2of theproject, further researchhasbeencarriedout into theoptimaldesignof thetable2qbinputformat,tomakeitflexibleenoughfortherangeofusecasestobesupported,butstilleasyforstatisticiansoranalyststoproduce,withoutneedingtechnicalknowledgeofLinkedData.
• Furtherprototypeshavebeenimplementedandtested• ImprovementstotheGrafterlibrarieshavebeenimplementedtoimproveperformanceand
scalabilitywhenprocessinglargedatacubes.
Thetable2qbspecificationcanbefoundathttps://github.com/Swirrl/table2qb
Graftercanbefoundathttps://github.com/Swirrl/grafter
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page23of25
3.9 DataCubeAggregator-updateDuringyear2oftheproject,theDataCubeAggregatorwasextendedtosupportthecomputationofaggregationsfromrawRDFdata.TheextensiontakesasinputrawRDFdataandcreatesacubethatcontainsthecorrespondingaggregatedobservations.ThecurrentversionoftheDataCubeAggregatoris integrated with version 1 of the JSON-QB API available at GitHub repository:https://github.com/OpenGovIntelligence/json-qb-api-implementation
FutureworkisaimingtomigratetheDataCubeAggregatortothenewAPIimplementationbasedonGraphQL.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page24of25
4 Conclusion
This deliverable provides description of the components delivered in the second phaseof OpenGovIntelligence project (Month 21 of the project). Detailed information about usageofpreviouslyavailabletoolsaredocumentedinDeliverableD3.2–“ReportonOpenGovIntelligenceICTtools–firstrelease”.
AllthetoolsavailableastheresultofOpenGovInteligenceprojectareconstantlyupdatedbasedontheevaluation result. Theplans fornext yeardevelopment include theusability andperformanceimprovements. It is expected that all project pilots will use suitable tool. Moreover, numerousgovernment organisations external to the project are expected to start usingOpenGovInteligencetoolsinyear3oftheproject.
ContinuousJSON-QBAPIdevelopmentisexpectedtoextendtherangeofmethodssupportedbytheAPI,deployinoperationalenvironments.Oneofthemajorfeaturesunderdevelopmentisaggregationacrosshierarchies. In thiscase theobservationsareaggregatedacrossahierarchyofadimension.Forexample, if a cube contains theelection results atmunicipality level, then theAggregator cancompute the results at region and at country level with the prerequisite that the correspondinghierarchy(municipality→region→country)exists.
ThisdeliverablewillbeupdatedtogetherwiththeongoingdvelopmentoftheOpenGovIntelligencetools.ThefinalreleaseofthedevelopedcomponentswillbedocumentedindeliverableD3.6“ReportonOpenGovIntelligenceICTtools”plannedatOctober2018.
D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease
Page25of25
References[1]D1.1OpenGovIntelligencechallengesandneeds
[2]D2.1OpenGovIntelligenceframework-firstrelease
[3]D2.2OpenGovIntelligenceframework
[4]D3.1OpenGovIntelligenceICTtools-firstrelease
[5]D3.2ReportonOpenGovIntelligenceICTtools-firstrelease
[6]D4.1PilotsandEvaluationplan
top related