d8.4 planetdata program assessment report d8.4 1st... · the deliverable presents the planetdata...

PlanetData Network of Excellence

FP7 – 257641

D8.4 PlanetData Program Assessment Report

Coordinator: Andreea Elena Gagiu (UIBK) With contributions from: Anna Fensel (UIBK)

Quality reviewer: Snorri H. Gudmundsson (IceStat)

Deliverable nature: Report (R) Dissemination level: (Confidentiality)

Public (PU)

Contractual delivery date: M24 Actual delivery date: M24 Version: 0.3 Total number of pages: 26 Keywords: Linked Data, Web of Data, Linked Open Data, evaluation, design, reliability of

results, potential, impact, contributions, evolution of requirements, consuming LOD, quality assurance, urban linked data, Norwegian LOD, Human Computation

PlanetData Deliverable D8.4

of (26)

Abstract

The deliverable is focused on presenting the PlanetData Program assessment for the three proposals (UrbanGames, NorthPole and ParkMe) accepted as part of the PlanetData Program 1 (Call 1): Consuming Linked Data. The deliverable offers an assessment framework split in three main categories: Performance of the Project (overall accordance to requirements and overall performance), Impact of the Project and Evolution of Requirements (assessment of the ways the applications’ requirements and features changed during the implementation stage, such as changes that were made due to technical motivations, glitches that have hinged the progress of a project, as well as unexpected or surprising results). The framework has been applied on the three proposals.

Deliverable D8.4 PlanetData

of (26)

Executive summary The deliverable presents the PlanetData Program assessment for the proposals accepted as part of PlanetData Program 1 First Call: Consuming Linked Data. It presents an overview of PlanetData (a short description of the project, its objectives and reasons for creating calls), the defining features of Call 1 (description of the call’s objectives, main topics of interests, formats of the projects, application procedures and facts, as well as the evaluation criteria utilized to select the proposals), descriptions of the proposals accepted, as well as an evaluation framework (an assessment approach, assessment criteria, comparisons and an evaluation matrix) to assess the progress of each project implicated in the Call.

The analysis of the three proposals shows that they have fulfilled the research goals they have set out to fulfil. Moreover, the end-products of the proposals are three applications which have received both positive end-user assessments, as well as interest from commercial enterprises.


of (26)

Document Information IST Project Number

FP7 - 257641 Acronym PlanetData

Full Title PlanetData Project URL http://www.planet-data.eu/ Document URL EU Project Officer Leonhard Maqua Deliverable Number D8.4 Title PlanetData

Program Assessment Report

Work Package Number WP08 Title Administrative and financial management

Date of Delivery Contractual M24 Actual M24 Status version 0.3 final □ Nature prototype □ report X dissemination □ Dissemination level public X consortium □ Authors (Partner) Andreea Elena Gagiu (UIBK)

Responsible Author Name Andreea Elena Gagiu E-mail [email protected] Partner UIBK Phone +43 512 507 6489

Abstract (for dissemination)

The deliverable is focused on presenting the PlanetData Program assessment for the three proposals (UrbanGames, NorthPole and ParkMe) accepted as part of the PlanetData Program 1 (Call 1): Consuming Linked Data. The deliverable offers an assessment framework split in three main categories: Performance of the Project, Impact of the Project and Evolution of Requirements. The framework has been applied on the three proposals.

Keywords Linked Data, Web of Data, Linked Open Data, evaluation, design, reliability of results, potential, impact, contributions, evolution of requirements, consuming LOD, quality assurance, urban linked data, Norwegian LOD, Human Computation

Version Log Issue Date Rev. No. Author Change 2012 – 09 – 04 0.1 Andreea Elena Gagiu Initial version collected from wiki

pages. 2012 – 09 - 19 0.2 Andreea Elena Gagiu, Anna

Fensel Revisions of the deliverable towards its quality improvement and addition of the missing details.

2012 – 09 - 28 0.3 Andreea Elena Gagiu Deliverable modified using reviewer indications.

https://host8.ssl-secured.eu/squirrelmail/src/compose.php?send_to=Andreea-Elena.Gagiu%40sti2.at�


of (26)

Table of Contents

Executive summary ........................................................................................................................................... 3 Document Information ...................................................................................................................................... 4 Table of Contents .............................................................................................................................................. 5 List of figures .................................................................................................................................................... 6 List of tables ...................................................................................................................................................... 7 Abbreviations .................................................................................................................................................... 8 Definitions ......................................................................................................................................................... 9 1 Introduction .............................................................................................................................................. 10 2 Call 1: Consuming Linked Data ............................................................................................................... 11

2.1 Description ........................................................................................................................................ 12 2.2 Evaluation/Impact Criteria ................................................................................................................ 13

3 Accepted Proposals .................................................................................................................................. 14 3.1 Proposal 1: UrbanGames .................................................................................................................. 14 3.2 Proposal 2: PlanetData-NorthPole .................................................................................................... 15 3.3 Proposal 3: ParkMe ........................................................................................................................... 16

4 Evaluation Framework ............................................................................................................................. 17 4.1 Assessment Framework .................................................................................................................... 17 4.2 Evaluation Methodology ................................................................................................................... 19 4.3 Results ............................................................................................................................................... 20

5 Conclusion ................................................................................................................................................ 24 References ....................................................................................................................................................... 25


of (26)

List of figures Figure 1: Linked Open Data cloud diagram illustrating the published data sets and the relationships that link

them together (note: each node in the diagram represents a distinct data set published as Linked Data, as of September 2011) .................................................................................................................................. 11


of (26)

List of tables Table 1: Summary of evaluation framework and grading system ................................................................... 17 Table 2: Results for the Performance of the Project ........................................................................................ 20 Table 3: Results for the Impact of the Project ................................................................................................. 22


of (26)

Abbreviations LOD – Linked Open Data

URI – Unique Resource Identifier

GWAP – Game with a Purpose

RDF – Resource Description Framework


of (26)

Definitions Human computation is the paradigm to leverage human capabilities to solve tasks that computers are not yet able to properly undertake.

OpenStreetMap project creates and provides free geographic data and mapping. The data is collected through a collaborative process, similar to wiki collaborations, from the community and relies on a crowdsourcing mechanism1

User experience – “Hassenzahl & Tractinsky define UX as “a consequence of a user’s internal state, the characteristics of the designed system, and the context within which the interaction occurs”. They specify user’s internal state as predispositions, expectations, needs, motivation, mood, etc., and characteristics of the designed system as complexity, purpose, usability, functionality, etc. The context (or the environment) can be for example an organizational or social setting, meaningfulness of the activity, voluntariness of use, etc.”

.

2

Player engagement (video game) – elements that determine increased attention and devotion from players in achieving the game’s tasks.

1 http://code.google.com/p/androjena 2 http://www.userintelligence.com/ideas/blog/2011/04/meaning-user-experience

http://code.google.com/p/androjena�

http://www.userintelligence.com/ideas/blog/2011/04/meaning-user-experience�


of (26)

1 Introduction In this deliverable we present the PlanetData Program assessment for the proposals accepted as part of PlanetData Program 1 (Call 1): Consuming Linked Data.

The aim of PlanetData is to establish a sustainable European community of researchers that supports organizations in exposing their data in new and useful ways. Due to the abundance of data continuously published online, including data streams, social media posts (e.g. blogs, micro-blogs, forums, social networks, etc.), digital archives, public sector data sets, eScience resources, and the Linked Open Data Cloud, the ability to efficiently and effectively utilize it has become a main component for Europe’s transition to a knowledge society. Tapping into such a wealth of information can enable businesses, governments, communities and individuals make informed decisions, while ensuring competitive advantages and general welfare [1]. PlanetData’s purpose is to drive forward the state-of-the-art in large scale data management and its application to create useful open data sets. Such purpose is motivated by the business’s increasing reliance on large public data; the uptake of open data principles in many vertical sectors (such as eGovernment); as well as the need of research communities to understand the petabytes of scientific data, and describe and expose them to encourage and enable collaboration.

PlanetData mission revolves around three main objectives: (1) research – create approaches to large-scale data management from different disciplines and develop holistic solutions to the challenges of dealing with planetary-scale data; (2) data provisioning and management – offer software to support large-scale data provisioning (via the PlanetData Lab), create definitive vocabularies for the description of data sets and their context; build a catalogue of data sets, publicize guidelines and best practices for provisioning (to enable the consumption of available data sets by end-users); and (3) impact – to provide a medium through which the research results and empirical findings of the PlanetData network can be used to improve the education level related to large scale management in both academia and industry; to create a community of researchers from different disciplines; to encourage (industrial) uptake through standardization, strategic dissemination and network events.

Throughout the duration of the PlanetData project, open calls are regularly issued. The purpose of such calls is to allow interested organizations to apply for funding for research activities related to the PlanetData project. In this respect, PlanetData First Call (discussed in this deliverable) has been issued on the 1st of January 2011, while the PlanetData Second Call will be open beginning with the 15th of August. The current deliverable will present the elements which compose Call 1 (its description, topics of interest, and acceptance requirements), present the projects which have been accepted by the consortium, as well as offer an assessment of the success of such proposals.

The remainder of the deliverable is structured as follows: Section 2 focuses on Call 1 on the subject and requirements of the call (presented in Section 2.1), as well as on the evaluation criteria for the submitted proposals (presented in Section 2.2). Section 3 presents in detail the proposals accepted for the call: (1) “Consuming and Quality Assessment of Linked Data in Urban Environments through Games with a Purpose” - UrbanGames, (2) “Consuming and Improving Norwegian Linked Open Data for Regional Development and Environmental Friendly Behaviour” - NorthPole, and (3) “ParkMe: Linked Open Parking Data” - ParkMe. Section 4 describes the evaluation framework and evaluation matrix used in the assessment of the accepted proposals. The section is split in three subsections: Section 4.1, which deals with the assessment framework which will be applied on all three proposals, Section 4.2, which mentions how the assessment has been conducted (i.e. which deliverables have been used, where the data for the assessment originated, etc.) and Section 4.3, which offers the results of the application of the evaluation framework on the three proposals. Section 5 presents the conclusions of the deliverable.


of (26)

2 Call 1: Consuming Linked Data Linked Data refers to the use of Resource Description Framework (RDF) and Hypertext Transfer Protocol (HTTP) to publish and interlink structured data on the Web and to connect such data between different sources, enabling data in one source to be linked to data in another source [2]. The principles of Linked Data have been coined by Berners-Lee in 2006 [3] and mention guidelines to publish data on the Web of Data: (1) use URIs as names for things; (2) use HTTP URIs to enable people to look up those names; (3) provide useful information (using standards, such as RDF* or SPARQL) whenever someone looks up a URI; and (4) include links to the other URIs to enable the discovery of more things [3].

The Web of Data can be accessed using Linked Data browsers in a fashion similar to that of HTML browsers for the traditional Web of documents; however, in this case the users do not navigate between HTML pages, but between different data sources using the RDF links [2]. These RDF connections allow the users to start with one data source and move through potentially endless sources.

The adoption of the Linked Data best practices for publishing has enabled the extension of the Web with a global data space connecting data from domains as diverse as people, business, books, medicine, scientific publications, entertainment (e.g. film, music, television, etc.), genes, proteins, online communities, etc. [4]. The most prominent example of Linked Data principles adoption is the Linking Open Data project [5] whose aim is to bootstrap the Web of Data by identifying existing data sets available under open licenses, converting them to RDF (according the before mentioned principles) and publishing them on the Web [4]. Initially the participants in the project were researchers and developers in universities, and small companies. With time, the project has grown considerably, as it now includes involvement from large organisations, such as the BBC, Thomson Reuters, the Library of Congress, the New York Times, and the UK and US governments. Such growth is enabled and sustained by the project’s open nature, which allows anyone to participate by publishing a data set according to the Linked Data principles and interlinking it with existing data sets [4]. An indication of the scale and development of the project can be viewed in Figure 1.

Figure 1: Linked Open Data cloud diagram illustrating the published data sets and the relationships

that link them together (note: each node in the diagram represents a distinct data set published as Linked Data, as of September 2011)


of (26)

With the abundance of Linked Data published, efforts have been made to research and build applications that will exploit the Web of Data. Bizer, Heath and Berners-Lee [4] argue that in these efforts can be classified into three categories: Linked Data browsers, Linked Data search engines, and domain-specific Linked Data applications. Similar to Web browsers, which allow users to navigate between HTML pages following hypertext links, Linked Data browsers allow users to navigate between data resources following the links contained by the RDF triples. On the other hand, search engines crawl Linked Data using the RDF links and provide query capabilities over the aggregated data [4]. Apart for the applications created to provide generic functionality on the Linked Data, there are applications developed to offer more domain-specific functionality by “mashing up” data from various Linked Data sources [4]. Such applications include [4] Revyu [6] - a generic reviewing and rating site based on Linked Data principles and Semantic Web technology stack; DBpedia Mobile [7] – a location-aware Linked Data browser for iPhones and other mobile devices; Talis Aspire [8] – a Web-based Resource List Management application; BBC Programmes and Music [9] use Linked Data internally as a lightweight data integration technology; DERI Pipes [10] – modelled on Yahoo! Pipes to enable data sources to be plugged together and form new feeds of data.

However, despite these advances, applications that consume that are not yet widespread. Reasons may include the lack of methods for seamless integration of Linked Data from multiple sources, dynamic discovery of available data and data sources, provenance and information quality assessment, application development environments, and appropriate end user interfaces [11]. Through Call 1, PlanetData aimed to provide a conduit for discussion and work on these open research problems and to offer an opportunity for organisations for scientific discourse and for showcasing research and technologies (including systematic analysis and rigorous evaluation) of concepts, algorithms and approaches for consuming Linked Data.

The call has been open for submissions from the 1st of January 2011 until the 16th of February 2011. The expected duration of the projects participating in the call ranged between 4 and 12 months, starting earliest in June 2011. One requirement for participation in the “Consuming Linked Data” Call states that the proposals must be submitted in English and can include one or more organisations eligible for EU funding. The indicative funding for the proposals has been selected as between €15,000 and €100,000.

2.1 Description

The call solicited research projects with a focus on demonstration of developed technologies in a concrete application context making use of publically available Linked Data sets [11]. Examples of topics of interest listed included:

• Web-scale data management (e.g. indexing, crawling, etc.);

• Searching and query processing over multiple linked datasets;

• Reasoning and Data from multiple sources

• Dataset dynamics;

• Linked Sensor Data;

• Data quality assessment and Data curation;

• Analytics and Web of Data mining;

• User interface and visualisation research;

• Linked Sensor Data;

• Business and incentives models for data-driven services and application.

Submissions were chosen based on their demonstrated applicability of their research results with an application that consumes data from the open Web, ideally of societal value, including, but not limited to the following domains:

• Public sector information;

• Statistical data;


of (26)

• Bibliographic and media sector information;

• Life sciences;

• Earth science and environmental data;

• Geospatial data;

• Transport and urban management.

As mentioned before, the duration of the projects was selected between four and twelve months. All applicants become part of the PlanetData network and can leverage the expertise of the Network of Excellence.

2.2 Evaluation/Impact Criteria

The evaluation framework chosen to select the submitted proposals included three main criteria:

• Scientific and/or technological excellence (relevant to the topics addressed by the call);

• Quality and efficiency of the implementation plan

• Potential impact.


of (26)

3 Accepted Proposals As a response to the call, 37 proposals have been received and 36 were valid and evaluated according to scientific and/or technological excellence, quality, and impact. Out of the 36 proposals, 3 have been strong enough to be accepted: (1) “Consuming and Quality Assessment of Linked Data in Urban Environments through Games with a Purpose” (UrbanGames), (2) consuming and Improving Norwegian Linked Open Data for Regional Development and Environmental Friendly Behaviour” (PlanetData-NorthPole), and (3) “ParkMe: Linked Open Parking Data” (ParkMe). More details of the selected proposals are presented in the remainder of this section.

3.1 Proposal 1: UrbanGames

The first proposal selected was “Consuming and Quality Assessment of Linked Data in Urban Environments through Games with a Purpose”, led by CEFRIEL. UrbanGames is a proposal to build a linked data-empowered application for tourists moving in the context of an urban environment. The application is created as a location-based mobile app, in order to be operated while “on the go” to provide urban exploration paths to users. UrbanGames has been shaped as a Game with a Purpose (GWAP), i.e. as an interactive gaming application that serves the double purpose of entertaining tourists and gathering high-quality data about the urban environment through social computation.

UrbanGames can:

• Consume existing “clean” urban linked data;

• Assess and improve urban linked data of “doubtful” quality;

• Produce new “unknown” urban linked data.

UrbanGames is created to reuse available open datasets in a Human Computation fashion and collect information about urban environments. The Android application, Urbanopoly, is a social, mobile and location-based Game with a Purpose similar to the Monopoly board game [12]. The research investigation purpose is to determine whether the physical presence in the urban environment, together with location-based technologies, can provide a valuable contribution to Human Computation tasks related to geo-spatial information [12]. The App is presented to the user as a consistent game, with “hidden” Human Computation tasks, where he has to pursue the game’s internal objective, i.e. becoming a landlord that owns a large number of venues. The Human Computation main objectives of the Urbanopoly game are [12]:

• Verify the data already provided and collected by the OpenStreetMap community;

• Correct imprecise or inaccurate pre-existing data;

• Collect new data, gathering both pre-defined information about geo-spatial elements and additional data.

By participating, the player also contributes to the achievement of the game’s “true” purpose: to verify, correct and collect urban data about the game venues (real places in the player’s surroundings, such as shops, restaurants, monuments, etc.) [12]. The data collected by the game is intended to complement the pre-existing OpenStreetMap data, enhance and correct it. To collect the information regarding the venues, Urbanopoly uses features and tags from OpenStreetMap, together with those defined by the developing team to complement them. To represent urban venues and their characteristics an OWL2 ontological model had been built.

The tasks proposed in Urbanopoly are [12]:

• Advertise action: tasks are encapsulated in creative challenges, the player is asked to provide a number of elementary inputs to create a complex artefact;

• Quiz challenges: the player is asked to select the “right” answer (the option that requires validation is presented together with possible alternative values);


of (26)

• Rating questions: a set of information about the same entity is presented as single artefact and the player is asked to judge that artefact of a 5-star rating scale.

The tasks, designed to be carried out mainly in the urban environment, can be presented on various levels of complexity and various degrees of collaboration amongst players (the result can be obtained only if two or more users work together) [12].

The intended result of the game is a clean and exploitable urban-related dataset, which can be of interest for stakeholders such as local authorities for governmental policy purposes, local businesses for marketing/attractiveness analysis and visitors services for tourism information.

3.2 Proposal 2: PlanetData-NorthPole

The second proposal selected was “Consuming and Improving Norwegian Linked Open Data for Regional Development and Environmental Friendly Behaviour”, led by Computas AS. Since Norway is one of the few countries outside the English speaking world with a clear governmental strategy and commitment to open data, as well as one of the highest Internet penetration and mobile access in Europe, it offers interesting opportunities for becoming a great testbed for consuming Linked Open Data (LOD).

PlanetData-NorthPole serves the following purposes:

• Publishing and managing new species of interlinked data sets;

• Improving the usefulness of existing data sources;

• Data sets, vocabularies, best practices for publishing self-descriptive data;

• Portal with data provisioning and management tools;

• Training infrastructure, learning resources, summer schools, standards.

The aim of the project is to establish a LOD consuming testbed – PlanetData-NorthPole – by creating applications that consume Norwegian LOD and show practical benefits of aggregating open data. In order to achieve this goal, two case studies will be developed in highly sensitive domains for governments and the general public, such as regional development and environmental friendly behaviour and then offer the gained experience to other countries interested in applying LOD in such domains. The test cases studied are:

• CS1 Monitoring Regional Development Application: Data journalism for regional development (especially for communes in Norway) [13]

o Background: Journalists spend significant time collecting and aggregating data for analysis trends and monitoring regional developments.

o Problem: How can we speed up and improve the process of collecting and aggregating data for monitoring regional developments?

o Added value proposition: enable smarter/faster monitoring of regional developments.

• CS2 Environmentally-friendly Decision Support Application

o Get from location A to location B in a town/region (focus on Oslo or Bergen)

o Possible options: public transportation (bus, tram, metro, or train), car (private, shared, electric, or taxi), cycling, walking.

o Potential constraints: time, avoid bad weather, polluted zones.

o Environmental parameters: CO2 emissions, energy efficiency

o Problem: Which are the most environmental – friendly options given the constraints?

o Added value proposition: enable smarter/faster environmental-friendly decision making for local trips when options are available.

The two case studies serve the purpose of demonstrating the benefits of open data, as well as make contributions to the improvement and extension of the Norwegian LOD.


of (26)

3.3 Proposal 3: ParkMe

The third proposal accepted bear the name “ParkMe: Linked Open Parking Data” and it is led by the Open University. ParkMe is a mobile and Web application that combines geographic data and parking space information with user location, social networks and other data sources in order to enable its users to conveniently identify parking spaces and related value-add services when coming to work or driving into town. In order to work with the multitude of heterogeneous data sources and services, and to generate and publish the crows-sourced parking data, the application uses state-of-the-art Semantic Web Service technologies – particularly those developed in the EU project SOA4All – which are aimed mainly at the integration of online services and Web APIs in distributed applications.

The objective of the project is to showcase the underlying technologies in a Linked Data application that both consumes data from diverse sources and produces linked data of its own.


of (26)

4 Evaluation Framework The current section presents the evaluation framework selected to create the assessment of the three accepted proposals. The first step in creating an evaluation framework is to determine what needs to be verified.

The remainder of this section is structured as follows: Section 4.1 presents the criteria selected for the evaluation, as well as short descriptions for each (why they were selected, what the criteria intends to measure, etc.), Section 4.2 focuses on the methods employed to assess the proposals (based on the criteria selected), while Section 4.3 presents the results of the assessment process.

4.1 Assessment Framework

The use of the same criteria across the evaluation serves the purpose of reducing the variations in approaches and reporting format between the three proposals assessed. Moreover, the consistent application of the evaluation framework aims to ensure that the three projects’ impacts are addressed in a systematic way, the results and performance are comparable (across the three proposals) and that the performance and results are more feasibly provided.

In order to achieve these goals, the evaluation framework is split in three main categories: (a) Performance of the Project, (b) Impact of the Project, and (c) Evolution of Requirements. Each category is divided into a number of elements (or sub-criteria) (see Table 1 for an overview of the framework and grading system). The assessment of each criterion will be approached separately (for each of the proposals).

Table 1: Summary of evaluation framework and grading system

Category Criterion Sub-criterion Grading System

Performance of the Project

Relevance - 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Effectiveness - 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Design and Delivery Functionality 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

User friendliness 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

User interface 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Best practices proposal

Yes (which?)/No

Performance 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Reliability of results - 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Usage of PlanetData Facilities

- Yes (which?)/No

Usage of pre-existing facilities

- Yes (which?)/No

Familiarity with related work

- Yes (which?)/No

(re)use of technologies from other EU projects

- Yes (which?)/No


of (26)

Impact of the Project Usefulness of application

- 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Potential fields of application

- Open ended question: either answer or “None”

Advance in technologies for using Linked Data

- Open ended question: either answer or “None”

Closeness to the market - Open ended question: either answer or “None”

Impact of proposal results

- 1 – 5 scale, where 1 is “Not at all fulfilled” and 5 is “Completely fulfilled”

Contributions - Open ended question: either answer or “None”

Evolution of requirements

Open ended category - Provide detailed descriptions of the elements relevant to this category.

The Performance of the Project is evaluated through the following criteria:

• Relevance: degree to which the project is consistent with individual project requirements;

• Design and Delivery:

o Is it functional? Does the application do what it set out to achieve (overall assessment of requirements – verification of whether the applications have fulfilled the goals they set up to achieve).

o Is it user friendly? This criterion focuses mostly on the concept of usability, particularly its positive aspect. The mark for this criterion will be given based on whether the application is in accordance with the following five elements: (1) learnability (“Does the application have a learning curve? How easy it is for users to accomplish basic tasks the first time they encounter the application?”), (2) efficiency (determine how quickly users can perform tasks after they have familiarized themselves with the design), (3) memorability (“Can the user perform tasks with the same efficiency after not using it for a period of time?”), (4) errors (“How many errors the user make when using it?”) and (5) satisfaction (how pleasant the design is).

o User Interface – the marks for this criterion are given based on whether the application follows the following rules: clarity (the content is conveyed quickly and accurately), discriminability (the displayed information can be easily distinguished), conciseness (users are not overloaded with unnecessary details), consistency (a design that is in conformity with what the user would expect for a particular application), detectability (the user can identify the information that is important), legibility (easy to read), and comprehensibility (the meaning is clear, concise, unambiguous).

o Best practices proposal – “Does the project propose solutions and ideas for best practices?”. From the best practices proposal point of view, the three projects have been evaluated based on the principles proposed in Deliverable 2.1 (“Conceptual model and best practices for high-quality metadata publishing”) [15].

o Performance – refers to the degree of accuracy, completeness, etc.

• Reliability of results - The current criterion refers to the degree to which the results presented by the developing teams are trusted and reliable. In order to measure this feature, two aspects have been taken into consideration: correctness and trustworthiness of software validations and those of the end-user validation.


of (26)

• Familiarity with related work – the creators of an application must be familiar with similar implementations and research, with their strengths and weaknesses, in order to make a plausible contribution.

• Usage of PlanetData facilities – mention which PlanetData tools have been used by an application;

• (re)use of technologies from other EU projects – mention which EU project facilities and technologies have been used.

• Usage of pre-existing facilities – mention which data-related facilities have been used.

Taken together, the criteria presented above capture how well the implemented proposals met the requirements of the Call, and intends to answer the following question: “Where the things done right?”. As such, it focuses on the quality of the project objectives (they have done the “right things”) and the extent that the objectives were achieved at the required time (they have been “done right”).

The Impact of the Project category is split in the following sub-criteria:

• Usefulness of the application - a quantitative mark that gives the reader an idea on the commercial potential of the application. The current criterion is couples with the following one (“Potential fields of application”) to create a complete view of what impact the application might have on the market.

• Potential fields of application. The current criterion is concerned with the domains in which the application can be utilised. In order to measure just how versatile the proposals are, the criterion will be applied for the applications offered by each proposal.

• Advance in technologies for using Linked Data – one of the requirements of the Call.

• Closeness to the market – verify whether the final product is a proof of concept or ground research. Additionally, if possible, mention the companies that would be interested in exploiting the results.

• Impact of proposal results – a quantitative mark to determine the potential success of the application on the market.

• Contributions – enumerate the contributions brought forth by the project.

The Evolution of Requirements category refers to the assessment of the ways the applications’ requirements and features changed during the implementation stage (changes that were made due to technical motivations, glitches that have hinged the progress of a project, as well as unexpected or surprising results). The purpose of this category is to address how well the individuals responsible for the implemented proposals have identified, prepared and supervised their respective projects. Additionally, this category is also intended to clarify certain aspects in the development of the project: “why the project has proceeded in a certain way (if the approach was different than what was expected)?”, “what shortcomings (if they were) can be identified and how can they be avoided in the future?” etc.

The evaluation criteria have been summarized in Table 1. For the criteria that do not have a binary form of assessment (i.e. “Yes/No”), the deliverable introduces a system of rating intended to quantify or qualify the assessment. The proposed scale is symmetrical (there are two positive and two negative ratings), and the distance between ratings is conceptually equal. The scales chosen are: High (5), Substantial (4), Modest (3), Negligible (2) and “None” (1). Additionally the deliverable will employ the use of open ended questions, designed to offer a full, meaningful answer, as opposed to closed-ended questions, which encourage short or single word answers.

4.2 Evaluation Methodology

The evaluation of the three proposals selected is performed in two stages. First, the requirements and objectives of the proposals are identified. Second, the proposals are evaluated on the criteria according to the information provided by the deliverables D9.1 [16], D10.1 [12], and D9.2 [18] for UrbanGames, D12.1.1 [13], D12.1.2 [18] and D12.2.2 [19] for NorthPole, and D13.1 [15] and D13.3 [20] for ParkMe. Additionally, for ParkMe, the assessment included notes from the developing team.


of (26)

4.3 Results

The current section presents the results of the evaluations for the tools implemented for each proposal. The tools have been analysed using the three main evaluation categories presented in Section 4.1. For the first two categories, a table (Table 2, respectively Table 3) with the criteria is provided along with a mark representing the extent to which the requirement has been fulfilled or an answer for open ended questions. The third category just enumerates the elements regarding the Evolution of Requirements. For open ended questions (questions that demand a qualitative answer as opposed to a quantitative one), the respective answers have been enumerated. The use of open ended questions is to enhance the analysis by providing examples of elements which are of great interest to the PlanetData Project. For instance, it is insufficient to mention which existing technologies have been used in the implementation of the proposal without mentioning the actual technology.

Table 2: Results for the Performance of the Project

UrbanGames PlanetData-NorthPole ParkMe

Relevance 5 4 4 Design and Delivery 5 5 4

Functionality 5 5 5 User

Friendliness 5 4 5

User Interface 5 5 5 Best Practices

proposal No Yes: for publishing self-

descriptive data No

Performance 5 5 5 Reliability of Results 5 5 5 Usage of PlanetData Facilities

Yes (D2R Server) No No

Usage of pre-existing facilities

Yes (OpenStreetMap, linkedGeoData, overpass API)

Yes (AndroJena, DBpedia)

Yes (GeoVocab, LinkedGeoData)

Familiarity with related work

5 5 5

(re-)use of technologies from other EU projects

- - Yes (Hyperdata)

The reader should note that the assessment of each criterion has been done in a subjective manner as the deliverable intends to quantify mostly qualitative data. In the following, more details are offered as to what each criterion means in the context of the PlanetData proposals and why the proposals have been rated in a certain way. The most interesting criteria presented in the table is analysed below.

• Relevance. Relevance in the context of the current deliverable refers to the degree to which the individual project requirements (as mentioned in the deliverables) have been fulfilled. Regarding this criterion, UrbanGames argue that they have fulfilled the milestones they have set up. Similarly, NorthPole states that the main requirements (those relevant to the functioning of the application and those relevant in the achievement of the goals set up) have been fulfilled fully. For NorthPole, the only requirements that they did not manage to fulfil are related to the user ability to select and automatically plug-in new data through a graphical interface (the current version requires the aid of an admin). However, these aspects do not affect the overall functionality and performance of the application. Similarly, ParkMe has fulfilled the requirements they have set up to only partially, as the project proposal overestimated what features could be implemented.

• The “Design & delivery” section refers to the applications of those decisions that make the application suitable for use and is split in five subcategories: (1) functionality – whether the application works as it is supposed to; (2) user friendliness (usability) – the degree of ease-of-use,


of (26)

i.e. whether the application creates positive user experience or frustration; (3) user interface – a quantitative mark on how the interface is designed; (4) best practices proposal – whether the application offers best practices on how to implement certain elements; and (5) performance – marks regarding the overall functionality of the application in terms of execution speed.

o Functionality. All three applications have been completed and tested on end-users. In this perspective, particularly interesting is the case of UrbanGames, which relies heavily on user input, and which was created as a game and not as an application that serves a very specific purpose that might interest users for reasons other than entertainment. Therefore, UrbanGames focuses on “enticing” users to willingly provide information in order to achieve the game’s goals, and thus suffers from the issues all games face – maintaining prolonged user interest. However, the developers have already taken this variable into considerations and are offering possible solutions to alleviate this issue. A similar issue is seen in the case of ParkMe, which is completely reliant on user input, and requires mechanisms to convince users to participate in the crowd sourcing effort.

o Regarding user friendliness, UrbanGame’ Urbanopoly ranks high as it is focused on convincing the user to participate in the joint effort. Similarly, ParkMe have employed state of the art user interface and usability guidelines. On the other hand, PlanetData NorthPole is ranked as 4 as although the application provides visualisations for users, there are segments were elements do not have a visual interface, such as the inclusion of new datasets in the application (which is done programmatically and not through a graphical interface). Moreover, new data variable selections are performed by the developer and not by the end user. However, by testing the applications, the results show that all three have high marks concerning learnability (all three are intuitive, there is no learning curve), efficiency (performing tasks poses no difficulty for users), memorability (the intuitive interface ensures that the user will always know how the applications work), errors (the application leave very little room for user error) and satisfaction (as proved by the evaluation deliverables).

o User interface. The mark for the user interface (the graphical representation intended for the user) has been given on how well the applications follow UI design and usability principles. In this perspective, all proposals have offered graphical user interfaces that are clean (not to many items cluttered on the screen), intuitive, pleasant (the colours chosen do not create a negative effect on the user), with great potential at creating positive user experience.

o Performance. All applications are completed to such a degree that they can be launched on the market (the logos and layouts are consistent, as well as all three applications have prototypes already launched and tested on the market). However, these prototypes have showed the developers that some additional decisions have to be taken in order to ensure user participation (particularly in the case of applications such as ParkMe and UrbanGames which rely on crowd sourcing).

Overall, UrbanGames presents a very detailed architecture, highly focused on the users as well as on ensuring player engagement. Similarly, PlanetData NorthPole offers a highly detailed view of its architecture, presenting use cases and very motivation for very important design decision every step of the way. Moreover, by employing a software validation, the reliability of their results and effectiveness of the tool is further sustained. In the case of ParkMe, although only a subset of the requirements mentioned in deliverable D13.1 [15] have been implemented at this point, this aspect does not affect the overall tool as the project is set to continue outside of PlanetData funding.

• Reliability of Results. Urban Games reliability is increase as they provide an in-depth analysis of both software validation and end-user validation. The authors provide in D 9.2 a critical assessment of what they have achieved with the application, as well as possible solutions to the issues they have encountered. Similar in-depth analysis is provided by NorthPole. Moreover, NorthPole results are also strengthened through the use of very specific use cases. Similar analysis has also been provided by ParkMe.

• Usage of PlanetData Facilities. As one of the main purposes of the PlanetData project is to provide data and technology, the proposals involved should take into consideration the (re)use of PlanetData


of (26)

facilities. For the current section a “Yes/No” Scale has been used. Whenever the answer has been given as “Yes”, the facility utilized is mentioned.

• Usage of Pre-existing Facilities. This section is concerned to whether the proposals use any other existing facilities: for the proposals that use existing facilities, they will be enumerated, whilst for the other, a “No” will be mentioned.

• Familiarity with related work. The current criterion is important as to ensure that the teams involved in the implementations of the proposals are aware of the similar technologies and approaches as to contribute and improve existing work.

In the following, the results of the analysis concerning the impact of the project are presented. Table 3 presents an overview of the application of the evaluation criteria on the three proposals in terms of impact.

Table 3: Results for the Impact of the Project

UrbanGames PlanetData-NorthPole ParkMe

Usefulness of the application (potential)

5 5 5

Potential fields of application

Urban planning, tourism, advertising; data collection and validation for different types of data

Journalists and citizens interested in learning more about regional developments in municipalities in Norway, environmentalists

Smartphone owners who drive and are looking for parking spaces, car park operators and local authorities.

Advance in technologies for using Linked Data

User oriented method of improving quality of available linked data

Methods of consuming available linked data and producing results interesting for a specific target population

APIs for RDF-based mobile services, mobile consumption or RDF data.

Closeness to the market

Proof of concept of GWAP

Will be used by Computas and SINTEF and further enhanced in the context of the Semicolon2 project3

App published: deployed on OU campus, interest from Milton Keynes City Council

Impact of proposal results

5 5 5

Contributions Mentioned below Mentioned below Mentioned below

The most interesting criteria presented in the above table are discussed below.

• Regarding the impact and exploitation of the product, UrbanGames propose the following venues for their application Urbanopoly: (1) Computing location-based assignments: providing incentive for players to collect specific sets of data, in defined areas or in a relative short time; (2) Social city guide: create synergies and involve the stakeholders of the urban environment; and (3) Local governance of a city: mapping strategic structures (e.g. public services or spaces in the city).

Similarly, PlanetData NorthPole intends to increase visibility regarding regional development and environmentally friendly behaviour, and enable normal users to check the data behind the illustrations and charts presented by the Norwegian Government.

Lastly, ParkMe is intended to be used in the following fields: employer campuses and business districts with high use of cars; cities and shopping districts with seasonal large demand (e.g. Christmas shopping). Moreover, the interest from OU Estates and MK City Council suggests that ParkMe is a valuable tool for the use of Linked Open Data. Moreover, parking management studies consistently note that providing information to drivers is valuable.

• Contributions. The criterion refers to what the implementations offer as main improvements.

3http://www.semicolon.no/Hjemmeside-E.html

http://www.semicolon.no/Hjemmeside-E.html�


of (26)

UrbanGames contributes by offering less expensive, crowdsourcing and user friendly method of improving and correcting the quality of open datasets. Similarly, PlanetData NorthPole main contributions are: (1) evaluation of state-of-the-art libraries/tools for JavaScript-based visualisation; and (2) development of LOD Wheel – a prototype for visualizing RDF data in graphs and charts. The main contributions of ParkMe are the investigation of real-world requirements on geographical data (the subset relevant to parking), and a discussion of the value of crowdsourcing approaches for data gathering, which depends greatly on the take-up of the application.

The final perspective in the analysis is the Evolution of Requirements that deals with the performance of the individual teams involved in the projects; this section includes unexpected glitches in the progress of the application, as well as unexpected positive surprises – moments when the expectations of the partners have been exceeded.

Although UrbanGames fulfilled the requirements it set out to achieve, they are still having issues in ensuring long term playability, as well as a more uniform participation from users (their preliminary results show high standard deviations between the user inputs), issues common in most games regardless of their purpose (e.g. commercial games, smart games, etc.). However, their results showed that with proper design choices it is not difficult to get high throughput values as the preliminary evaluation results show a user engagement beyond the expectations of the developers.

Objectives that could not be fulfilled fully (were only partially fulfilled) in the PlanetData North Pole are requirements related to the capability that the end user selects and automatically plugs-in new data sources through a graphical interface - proved difficult to implement due to the short duration of the project. However, as mentioned, such features do not affect the overall functionality and goal achievement of the application.

Regarding ParkMe, the application name had to be changed on the public Web site in the published app and on its website to ParkJam due to the fact that several other apps had appeared with the same name; however, within the deliverable the name has been kept unchanged. Moreover, not all objectives mentioned in deliverable D13.1 [15] could be implemented as the project proposal overestimated what features could be implemented.


of (26)

5 Conclusion The current deliverable we presented the PlanetData assessment of the proposals accepted as part of Call 1: Consuming Linked Data. The three proposals selected are UrbanGames, NorthPole and ParkMe. UrbanGames purpose is quality assurance on urban linked data through the use of Human Computation methods (by creating a game that collects the data from the players). The aim of NorthPole is to establish a LOD consuming testbed through applications that consume Norwegian LOD and show practical benefits of aggregating open data. ParkMe’s focus is to gather space availability about car parks based on user input.

The analysis of the three proposals shows that they have fulfilled the research goals they have set out to fulfil. Moreover, the end-products of the proposals are three applications which have received both positive end-user assessments, as well as interest from commercial enterprises. Additionally, the applications have fulfilled the goals set out by the Call requirements for acceptance (consuming linked data).


of (26)

References [1] Project Description. (n.d.). Retrieved from PlanetData homepage: http://planet-data.eu/project-

description. Last accessed 04.09.2012.

[2] Bizer , C., Heath , T., Idehen , K., & Berners-Lee , T. (2008). Linked data on the web (LDOW2008). WWW '08 Proceedings of the 17th international conference on World Wide Web (pp. 1265-1266 ). New York, NY, USA: ACM.

[3] Berners-Lee, T. (18.06.2009). Linked Data. Retrieved from W3.org: http://www.w3.org/DesignIssues/LinkedData.html. Last accessed 04.09.2012.

[4] Bizer, C., Heath, T., & Berners-Lee, T. (2009). Linked Data - The Story So Far. International Journal on Semantic Web and Information Systems (IJSWIS), Vol. 5, No. 3. , 1-22.

[5] Linking Open Data. (2012, 04). Retrieved from W3.org: http://www.w3.org/wiki/SweoIG/TaskForces/CommunityProjects/LinkingOpenData. Last accessed 04.09.2012.

[6] Heath, T., Motta, E. (2008). Revyu: Linking reviews and ratings into the Web of Data. Journal of Web Semantics, 6(4):266-273.

[7] Becker, C., Bizer, C. (2008). DBpedia Mobile - A Location-Aware Semantic Web Client. Proceedings of the Semantic Web Challenge at ISWC 2008

[8] Clarke, C. (2009). A Resource List Management Tool for Undergraduate Students based on Linked Open Data Principles. Proceedings of the 6th European Semantic Web Conference (ESWC2009).

[9] Kobilarov, G., et al. (2009). Media Meets Semantic Web - How the BBC Uses DBpedia and Linked Data to Make Conections. Proceedings of the 6th European Semantic Web Conference (ESWC2009).

[10] Le Phuoc, D., Polleres, A., Morbidoni, C., Hauswirth, M., Tummarello, G. (2009). Rapid semantic web mashup development through semantic web pipes. Proceedings of the 18th World Wide Web Conference (WWW2009).

[11] Announcement of the 1st call of the PlanetData Program. (n.d.). Retrieved from PlanetData: http://www.planet-data.eu/news/call1. Last accessed 04.09.2012.

[12] López-Cobo, J.-M. (30.06.2012). D10.1: UrbanGames Dissemination, exploitation and management report. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D10.1. Last accessed 04.09.2012.

[13] Norheim, D. (30.11.2011). D12.1.1 NorthPole Case studies: definition, requirements, design. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D12.1.1. Last accessed 04.09.2012.

[14] Kopecký, J. (n.d.). D13.1: ParkMe Requirements and Data Source Analysis. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D13.1. Last accessed 04.09.2012.

[15] Mendes, P. (n.d.). D2.1 Conceptual model and best practices for high-quality metadata publishing. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D2.1. Last accessed 04.09.2012.

[16] Celino, I. (n.d.). D9.1: UrbanGames Concept and design. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D9.1. Last accessed 04.09.2012.

[17] Celino, I. (n.d.). D9.2: UrbanGames Prototype and evaluation. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D9.2. Last accessed 04.09.2012.

[18] Celino, I. (31.03.2012). D12.1.2: NorthPole Report on prototypes, development, validation and evaluation. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D12.1.2. Last accessed 04.09.2012.

[19] Roman, D. (31.03.2012). D12.2.2: NorthPole Report on Norwegian LOD extensions. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D12.2.2. Last accessed 04.09.2012.

http://planet-data.eu/project-description�

http://planet-data.eu/project-description�

http://www.w3.org/DesignIssues/LinkedData.html.%20Last%20accessed%2004.09.2012�

http://www.w3.org/wiki/SweoIG/TaskForces/CommunityProjects/LinkingOpenData.%20Last%20accessed%2004.09.2012�

http://www.w3.org/wiki/SweoIG/TaskForces/CommunityProjects/LinkingOpenData.%20Last%20accessed%2004.09.2012�

http://www.planet-data.eu/news/call1�

http://planet-data-wiki.sti2.at/web/D10.1�

http://planet-data-wiki.sti2.at/web/D12.1.1�








of (26)

[20] Kopecký, J. (n.d.). D13.3: ParkMe 1st prototypes. Retrieved from PlanetData Wiki: http://planet-data-wiki.sti2.at/web/D13.3

d8.4 planetdata program assessment report d8.4 1st... · the deliverable presents the planetdata...

Documents