field trials evaluation -...

D4.3

Field trials evaluation

Editor: Carmen López (UC), Hyokeun Choi (SDS)

Submission date: 03/07/18

Version 1.0

Contact: [email protected], [email protected]

The document will provide the evaluation results of the field trials carried out on top of the different pilots involving end-users.

mailto:[email protected]

Editor: Carmen López (University of Cantabria)

Hyokeun Choi (Samsung SDS)

Deliverable nature: Other (O)

Dissemination level: Public

Contractual/actual delivery date: M24 M25

Disclaimer

This document contains material, which is the copyright of certain WISE-IoT consortium parties, and may

not be reproduced or copied without permission.

All WISE-IOT consortium parties have agreed to full publication of this document.

The commercial use of any information contained in this document may require a license from the

proprietor of that information.

Neither the WISE-IOT consortium as a whole, nor a certain part of the WISE-IoT consortium, warrant

that the information contained in this document is capable of use, nor that use of the information is free

from risk, accepting no liability for loss or damage suffered by any person using this information.

This project has received funding from the European Union’s H2020 Programme for research,

technological development and demonstration under grant agreement No 723156, the Swiss State

Secretariat for Education, Research and Innovation (SERI) and the South-Korean Institute for Information

& Communications Technology Promotion (IITP).

Copyright notice

2018 Participants in project WISE-IOT

Revision History

Revision Date Description Author (Organisation)

V0.1 16th April

2018

D4.3 Table of Contents Carmen López (UC),

Pablo Sotres (UC)

V0.2 28th May

2018

First version of rich parking contributions Carmen López (UC),

Pablo Sotres (UC)

V0.3 7th May

2018

Second version of rich parking contributions Carmen López (UC),

Pablo Sotres (UC), Luis

Muñoz (UC), Sonia

Sotero (SAN)

V0.4 11th June

2018

Bus Information System, Smart Resort

Management, Life Style contributions

Hoang Minh (KAIST), Hyokeun Choi (SDS), KNU

V0.5 13th June

2018

SAR evaluation in Rich Parking and Smart Skiing,

Smart parking contributions

Dustin Wüest (FHNW),

SeungMyeong (KETI)

V0.6 15th June

2018

Added executive summary and conclusions and

editorial work.

Hyokeun Choi (SDS)

and Carmen López (UC)

V0.7 22th June

2018

Some comments from review addressed Review from LJMU,

Carmen López (UC)

V0.8 26th June

2018

Comments from review addressed Carmen López (UC)

V0.9 28th June

2018

Editorial work Carmen López (UC)

V1.0 3rd July 2018 Smart Skiing contribution and editorial work.

Final verification

Levent Gurgen (CEA),

Carmen López (UC),

Pablo Sotres (UC)

Franck Le Gall (EGM)

Table of contents

Executive summary _________________________________________ 6

Authors ______________________________________________________ 7

Glossary _____________________________________________________ 8

1 Introduction ______________________________________________ 9

1.1 WISE-IoT purpose and work package objectives _______________ 9

1.2 Deliverable contribution _________________________________________ 9

2 WISE-IoT Pilots Evaluation ______________________________ 11

2.1 Methodology for pilots evaluation _____________________________ 11

2.2 Parking Use Cases _____________________________________________ 11

2.2.1 Pilot report _______________________________________________________ 11

2.2.1.1 Rich Parking (EU pilot) _________________________________________________ 11

2.2.1.2 Smart Parking (KR pilot) _______________________________________________ 16

2.2.2 Evaluation report _________________________________________________ 18

2.2.2.1 Rich Parking evaluation (EU pilot) ________________________________________ 18

2.2.2.2 Smart Parking evaluation (KR pilot) _______________________________________ 25

2.2.2.3 Parking Use Case Evaluation (EU & KR pilot results) __________________________ 28

2.3 Bus Information System _______________________________________ 34

2.3.1 Pilot report _______________________________________________________ 34


2.4 Smart Skiing ___________________________________________________ 42

2.4.1 Pilot report _______________________________________________________ 42


2.5 Smart Resort Management ____________________________________ 48

2.5.1 Pilot report _______________________________________________________ 48


2.6 Lifestyle Use Case _____________________________________________ 56

2.6.1 Pilot report _______________________________________________________ 56


3 Evaluation of the Self-Adaptive Recommender Key

Performance Indicators ____________________________________ 63

3.1 SAR in the EU Rich Parking Use Case _________________________ 64

3.1.1 Amount of data gathered _________________________________________ 64

3.1.2 KPI results ________________________________________________________ 64

3.2 SAR in the Smart Skiing Use Case ____________________________ 71

4 Evaluation of General Key Performance Indicators for Use

Cases _______________________________________________________ 72

5 Conclusions _____________________________________________ 75

6 References ______________________________________________ 76

7 Annexes _________________________________________________ 77

7.1 Rich Parking PIA _______________________________________________ 77

7.2 Smart Parking PIA ______________________________________________ 81

7.3 Bus Information system PIA ___________________________________ 84

7.4 Smart Skiing PIA _______________________________________________ 87

7.5 Smart Resort Management PIA ________________________________ 89

7.6 Lifestyle PIA ____________________________________________________ 92

7.7 Engagement documents _______________________________________ 93

7.7.1 Engagement documents (Santander Use Cases) ________________ 93

7.7.2 Leaflet Rich Parking Use Case ___________________________________ 96

7.8 Self-Adaptive Recommender Feedback Forms ________________ 97

Executive summary

Wise-IoT project aims to create a framework in which heterogeneous IoT platforms and services can be

combined, and data can be aggregated together to be used for meaningful purposes. This is achieved

through the creation of semantic modelling and automatic mediation components to enable

interoperability at the data level; thus reducing the effort needed to develop new services and

applications. In order to create such a framework, the Wise-IoT project addresses four main topics:

interoperability, mobility, trust, and standardization.

The purpose of this document is to validate the main achievements and results derived from Wise-IoT

pilots. More specifically, six pilots have been deployed on five different areas of interest: Parking, Bus

Information System, Smart Fitness, Smart Skiing, and Smart Resort Management, deployed in two

European and four South-Korean sites. During each pilot, the evaluation phase outcomes have been

processed to validate the pilot results based on the KPIs and metrics defined in deliverable D4.1.

In this deliverable a general overview of the methodology used to evaluate the different pilots as well

as information related to each specific use case are provided. In this sense, it includes how the pilot,

users engagement and the evaluation phase have been accomplished, explanations about the reasoning

of the different decisions taken, the lessons learnt during the deployment and pilot execution stages,

the pilot results, and an analysis of the feedback provided by the end users.

Authors

Chapter Author (Organisation)

Executive summary Hyokeun Choi (SDS)

1.1 Introduction Carmen López (UC)

2.1 Methodology for pilots

evaluation

Carmen López (UC)

2.2.1.1 Rich Parking pilot Carmen López (UC), Pablo Sotres (UC), Sonia Sotero (SAN),

2.2.1.2 Smart Parking pilot SeungMyeong (KETI)

2.2.2.1 Rich Parking Evaluation Carmen López (UC), Pablo Sotres (UC)

2.2.2.2 Smart Parking Evaluation SeungMyeong (KETI)

2.2.2.3 Parking Use Cases

Evaluation

Carmen López (UC), SeungMyeong (KETI)

2.3 Bus Information System Hoang Minh (KAIST)

2.4 Smart Skiing Levent Gurgen (CEA)

2.5 Smart Resort Management Hyokeun Choi (SDS)

2.6 Lifestyle Use Case KNU

3 Evaluation of SAR KPI Dustin Wüest (FHNW)

4 Evaluation of general KPI Carmen López (UC)

5 Conclusions Hyokeun Choi (SDS)

7.1 Rich Parking PIA Carmen López (UC)

7.2 Smart Parking PIA SeungMyeong (KETI)

7.3 Bus Information System PIA Hoang Minh (KAIST)

7.4 Smart Skiing PIA Levent Gurgen (CEA)

7.5 Smart Resort Management

PIA

Hyokeun Choi (SDS)

7.6 Lifestyle PIA KNU

7.7 Engagement documents Carmen López (UC), Sonia Sotero (SAN)

7.8 Supersede feedback forms Dustin Wüest (FHNW)

Glossary

Term or Abbreviation Definition (Source)

BLE Bluetooth low energy

CEMA Crowd Estimation and Mobility Analytics

EQ Engineering questions

GDPR General Data Protection Regulation

IoT Internet of Things

KPI Key performance indicator

MAPHIS Most Advanced Personalized Healthcare Intelligent Service

MMG Morphing mediation gateway

MOS Mean opinion score

OS Operating system

POI Point of interest

QoE Quality of experience

SAR Self-Adaptive Recommender

SME Small and medium-sized enterprises

UI User interface

UX User experience

Introduction

9

1 Introduction

1.1 WISE-IoT purpose and work package objectives

The rise of Internet of Things (IoT) and its concept of networking of various types of devices, data, and

services have led to a substantial increase in the amount of data exchanged and the creation of large-

scale platforms for a great amount of assets. However, this comes with the cost of increased complexity

in systems, and the need for interoperability between them. The information of each system must be

discovered and retrieved, and a common model needs to be made for the exchange of information. This

poses a serious challenge for developers, as they have to create services with careful consideration, with

the need of studying each specific system and standard they are dealing with. This ‘manual’ approach is

not scalable, and may even not be appropriate to solve problems with high degree of heterogeneity.

In dealing with the aforementioned issues, the WISE-IoT project aims to create a framework in which

heterogeneous IoT platforms and services can be interconnected, and data can be aggregated together

to be used for meaningful purposes. This is achieved through the creation of semantic modelling and

automatic mediation components to enable interoperability at the data level; thus reducing the effort

needed to develop new applications and services. Also, to create such a framework, the WISE-IoT project

closely follows four main themes of interoperability, mobility, trust, and standardization.

This work package’s (WP) purpose was to build representative use cases for the architecture developed

in WP1 as part of the WISE-IoT framework, and the technologies and components for federation and

interoperability of IoT platforms developed in WP2 and integrated in WP3. More specifically, six use

cases has been developed: Rich Parking, Smart Parking, Bus Information System, Smart Fitness, Smart

Skiing, and Smart Resort Management, deployed in two European and four South-Korean sites involving

three IoT scenarios: Smart City, Lifestyle, and Smart Skiing.

In summary, WP4 focuses on the successful achievement of the following objectives:

• O4.1: To identify key functional requirements, KPIs and metrics, which will become the bases of

the assessment of the use cases running on top of the platforms.

• O4.2: To proceed with the deployment of the selected use cases on both continents.

• O4.3: To carry out the field trials on top of the infrastructures and platforms providing support

to the services driving the use cases and engage significant number of users in the pilots.

• O4.4: To evaluate the performance of the solutions, applications and services running in the

corresponding sites according to the KPIs and metrics identified in task T4.1 “Recommendation

service deployment”.

1.2 Deliverable contribution

The purpose of this deliverable is to provide a last report of the evaluation results of the field trials

carried out in the WISE-IoT project. These field trials serve as the determination of the degree of success

of the proposed functionalities in each use case, and contribute to the evaluation of WISE-IoT objectives

such as the global mobility and remote access to data. At the same time, these field trials allow the

demonstration of real-world applications by using WISE-IoT platforms.

Initial descriptions of pilots have been included in previous deliverables [1] [2] [3] [4] [5]. In addition, on

deliverable D4.1 [3], as a result of task T4.1 focused on the definition of the evaluation metrics and KPIs,

Introduction

10

it was described a summary of the general KPIs to be fulfil by the pilots, and the metrics expected per

each use case. Finally, as a result of task T4.2, in which the pilots were deployed, it was described the

tools and services made available for the experiments.

In this deliverable, the results extracted during the evaluation phase and the lessons learnt during the

deployment of the trials will be described. Thus, this deliverable does not serve only as a metric of the

degree of success, but also serve as a reference for future research and implementations.

The reporting on the use cases is presented in Section 2. This report includes for each use case: a

description of the pilot phase, the evaluation phase, and the results. Additionally, Section 3 is devoted

to the reporting of the KPIs related to the Self-Adaptive Recommender.

The deliverable is structured as follows:

• Section 2 presents for each use case (including Rich Parking, Smart Parking, Bus Information

System, Smart Fitness, Smart Skiing, and Smart Resort Management) the pilot and evaluation

reports. On one hand, the pilots for each use case are described: how they have been organized,

how participants have been engaged, explanations about the different decisions taken by pilots’

owners during the deployment, and the lessons learnt and regulations issues which have been

found during this phase. On the other hand, the evaluation reports present the results of the

use cases specific metrics that were defined in deliverable D4.1 [3]. These reports cover the

design of the evaluation phase, how the users were approached to gather feedback, how the

questionnaire forms were designed so the different KPIs and metrics were involved, and the

results from this evaluation process.

• Section 3 presents and analyses the results of evaluating the key performance indicators

regarding the Self-Adaptive Recommender, as defined in deliverable D4.1 [3].

• Section 4 describes and analyses the results of the evaluation of general key performance

indicators for use cases defined in deliverable D4.1 [3]

• Section 5 highlights the main outputs of this document.

• Finally, section 6 provides necessary references for this deliverable.

WISE-IoT Pilots Evaluation

11

2 WISE-IoT Pilots Evaluation

2.1 Methodology for pilots evaluation

The evaluation process began in T4.1, with the definition and description of the KPIs and metrics to

measure the success of the pilots and the developments of the WISE-IoT project.

Deliverable D4.1 was the framework to present all these KPIs and metrics. For the general evaluation of

all the use cases, a general set of KPIs was described. In addition, for each use case specific metrics were

defined to measure concrete parameters related to the use case.

Once the pilots were deployed, the evaluation phase continued. In most of the cases, the applications

and services were internally tested so friendly users could provide insights about them, warn about

possible bugs, or give suggestions to improve the usability. After the launching of the pilots, the

applications and services were evaluated by end users. Depending on the use case, the evaluation

followed a different approach which is described in the following sections for each of them. For example,

some of them preferred to provide a channel of communication between the participants and the

developers during the pilot plus a final questionnaire, others preferred to include the questionnaire

within the application, etc.

At the end of the pilot, use case owners gathered all the evaluation information so they could extract all

the conclusions from users and find values in order to appraise the KPIs and metrics defined during T4.1.

In Section 4 the results for the general KPI described in D4.1 are presented, as a compilation of all the

results. In this section, a description and the evaluation results specific to each of the pilots developed

in WISE-IoT project is provided. The description of the pilot includes information about the service

provided, the engagement of the users and the lessons learnt and regulation issues find during the pilot.

The evaluation report presents how the feedback has been gathered and the results obtained.

2.2 Parking Use Cases

In this sub-section the two pilots related to the Smart Parking use case and their results are described.

As defined in previous documentation [3] [4] two applications have been developed with the objective

of enhancing the parking experience in the city. One of the applications has been developed in EU and

it is based on FIWARE, while the other has been developed in KR based on oneM2M. However, both

applications are portable, i.e. they can be used in both cities thanks to the interoperability components

developed in WISE-IoT. With these two applications we have demonstrated that a FIWARE based

application can interwork with oneM2M data platforms due to the MMG components, and vice versa.

In the following section, we describe the two pilots deployed in Santander and Busan/KETI premises,

and also the report of their evaluation.

2.2.1 Pilot report

2.2.1.1 Rich Parking (EU pilot)

As previously mentioned, Rich Parking pilot aims to enhance the parking experience of users in the city.

The application provides functionalities of parking availability, routing, statistics, etc. and can be used in


12

both cities Santander and Busan thanks to the integration of the interoperability developments provided

by WISE-IoT [4].

Within task T4.3 the schedule and the procedure for carrying out the evaluation process of the use case,

both internally and from participants of the pilot, was established. Once the application was ready for

launching, and previously to be shared with official participants of the pilot, it was shared with friendly-

users. These users were colleagues within the UC who installed the application and used it for one week.

At the end of this initial testing phase they provided feedback about any bugs encountered or comments

about things to be changed/improved. With this feedback, developers of the application could perform

some improvements, so the application was ready to be launched to real participants of the pilot.

In order to be shared with end users, the application was uploaded to the Google Market but it would

not be available for all people but only for those users who participate within the pilot. For this purpose,

it was created a beta group. To take part in this beta group and be able to download the application,

users had to provide their email. From this point, they receive an email with a link to the application and

they can download the application in the same way as other applications.

The only information users had to share with the developers was their name and the email. This

information was used to add them to the beta group and to send them emails for sharing the link to

download the application and to share with them the link to the questionnaire for the evaluation. In

order for users to share this information (name and email) and keep it during the project life, it was

requested a consent document (section 7.7.1) to be signed by users who wanted to participate in the

pilot. We have provided an explanation about lessons learnt, where it is described the obstacle that the

consent has been for the engaging numbers.

The participants for these pilots were engaged through different meetings organized by the UC and SAN.

The schedule designed for these meetings was to present the project to the attendees, later present

them the pilot, and finally invite them to participate. For the presentation of the pilot they were given a

leaflet with the important information about the pilot, as a tutorial on how to use the application

(section 7.7.2). The application also contains a help view to check all the functionalities of the app and

how to use them. In addition to the information of the pilot, they were provided with the official

documentation in case they want to be participants: the already mentioned consent, a letter of

complaint and a letter of withdrawal.

Different channels have been used to engage users:

• UC channels, such as IoTSantander meetup,

• Municipality channels such as city councillors, municipal staff and municipal website.

The first meeting with users were set on the 23rd February 2018 through a meetup at the University.

The meetup was announced through the Meetup webpage IoTSantander Meetup [6] so around 200

people received an email with the advertisement of the meeting. Unfortunately, no people came to the

meeting, maybe due to the date, little time before the meeting, the weather, etc.

In order to make a more attractive meetup, we decided to organize a second edition in a new location

and offering a more elaborated agenda. This second meetup was set up on 15th March 2018 at the Smart

City Demonstration Centre, where attendees were informed about the strategy and evolution of

Santander as Smart city, the most relevant innovation projects carried out in the city, such as Smart

Water or Streetlight projects, and also, EU projects, such as WISE-IoT project, including the pilot in

Santander and AparcaSDR app. Although only 4 people attended the meeting and they were very active

participants, unfortunately, none of them was interested in testing the application since they did not

have a car or an Android mobile.


13

Figure 1. Santander Smart City Demonstration Centre

In parallel with IoTSantander meetup activities, there was a first meeting with municipal staff from the

IT department. After a presentation of the project and the pilot, the app was shown and attendees were

invited to join.

Additionally, a couple of meetings were held with the Neighbourhood councilwoman to present her the

WISE-IoT project and Santander pilot, first, and then AparcaSDR app. The main objective of these

meetings was to request her collaboration to engage citizens as users. Although she liked the app and

wanted to become a user, it was not feasible because only android version is available; however, she

helped us to contact several neighbourhood associations in the city. In order to offer flexibility and make

things convenient for potential users, several meetings were held with citizens at different municipal

premises. To provide them a context, meetings started with a brief description of WISE-IoT project, and

then, AparcaSDR was presented, using the leaflet and showing how the app works in real time. Some

attendees provided their feedback, i.e., in terms of app usability which was reported to UC developers

to improve the app.

Figure 2. Meeting with citizens

During these meetings, most of potential users considered AparcaSDR a useful application and liked its

user interface, however not all of them wanted to join due to different reasons:

• Some of them avoid driving in the city centre or they prefer using public transport,

• IPhone users were not able to test it, because iOS version is not available,

• Maintenance works are being performed in several city centre streets, being some of them

totally and others partially closed to vehicular traffic.


14

Figure 3. Maintenance works in Santander city centre streets

Although during the evaluation phase users could evaluate the app through a questionnaire, the

municipality has preferred to organize follow-up meetings to get users feedback and know about their

feelings. Most of them agreed on being a great idea to help people to park in the city centre and also,

include extra functionalities such as walking route to your car and visualization of parking lots. However,

they commented that it was difficult to park in the parking lots suggested by the app: the parking lot

was occupied when they reached it, or there was not enough room to park.

In addition to the official meetings with users, other participants were engaged in other informal

meetings. However, although they are informal ones, users were informed about the pilot and were

provided with the documentation to be signed.

With all of this, a total of 41 people took part in the pilot study.

• Lessons learnt, regulation issues:

During the development of the different scenarios related to the use cases, some unforeseen

impediments for the execution of the initial objectives within WISE-IoT have arisen. These problems can

be encompassed in four categories: privacy regulation issues for crowd detection services, difficulties to

engage people to be beta users of the pilot, unpredicted long-term road works in the city centre, and

malfunctioning of LoRa parking devices under non-optimal coverage conditions. Even though these

issues seem to be of very different nature, they have been obstacles at different levels in the deployment

and pilot phases.

Regarding the crowd detection service in Santander city, it has been a joint effort between NEC,

Santander city council and University of Cantabria. First of all, the different steps carried out in order to

set up the deployment are depicted. During the technical phase, the different places in the city of

Santander that the municipality wanted to cover were spotted, and three different deployment areas

were agreed. In addition, different privacy related technical requirements were imposed by the

regulations to anonymise the information gathered by the CEMA (Crowd Estimation and Mobility

Analytics) devices according to the existing regulation. During the second year of the project, NEC

provided a first version of their CEMA device, so it could be internally tested by the UC on their premises.

First testing process detected a problem due to power restrictions. Devices were consuming more power


15

from the batteries than the one they were able to extract from the network; hence they were not able

to be continuously active (as once deployed in the field they would be attached to lampposts which are

only powered during night time). Once this limitation was overcome, the device was returned to the UC

to be re-tested. Results were satisfactory, and the information provided by them was available on a

FIWARE context broker hosted by NEC. However, after the testing phase, GDPR regulation came into

force and some legal (not technical) issues need to be addressed between NEC and Santander

Municipality. However, the discussions between the involved partners legal departments, together with

the uncertainty time window introduced during this initial stage of GDPR, delayed the deployment phase

beyond the deadline imposed by the project ending. Of course, from this experience some lessons have

been learnt, both technically, since the device was designed and tested so it could be part of other

projects in the future, and also legally. With this new GDPR regulation, it is necessary to carefully take

into account privacy aspects that were not a problem in the past, but now could be. Actually, the

problem is that sometimes it is difficult to know where the red line which could be considered a privacy

issue is, and what the roles of the interested parts are. From a technical point of view, there are some

considerations that should not present any privacy problems, but from a legal point of view, and under

very specific circumstances, it seems that it can be a problem.

On the other hand, during the rich parking pilot we found some obstacles derived from different aspects.

The first difficulty is related to the citizen’s engagement in the rich parking pilot in Santander, and

particularly to the agreed procedure on how to enrol them, which has been a consequence of privacy

regulations. The pilot experience aims at a time-delimited android application-based pilot, and the idea

was to use a close beta group to control its deployment, so the participant’s email was the only needed

private information. However, to gather the emails every person willing to participate in the pilot had

to hand-sign a formal consent document to cope with GDPR regulations. After that, UC stores all this

private information within a secure environment. As a result, it was decided to meet the people at

different events, present the mobile application and, once everything was clear for them, give them the

consent to sign if they wanted to participate. We consider that this way of proceeding has obstructed

the gathering of a bigger number of users, but the legal procedures have complicated the presentation

of the technical work. Besides, during the pilot it just so happened that some main streets on the parking

area where most of the parking sensors are deployed were under heavy works. This caused that some

streets were closed, so parking spots could not be used, and the roads could not be used on the routes

recommendations. Although the app took into consideration these limitations, since the algorithm was

directed not to use those roads and the parking spots were removed temporally from the database,

some people argued, and it is completely understandable, that the app was not as useful as it could have

been due to this issue.

Last but not least, after several iterations, the deployment of the LoRaWAN based parking sensors

provided by SKT in the city has been cancelled. In this case the problem has been a technical one: the

devices do not correctly behave under non-optimal coverage conditions. After some indoor lab testing,

a LoRaWAN gateway was installed at the roof of a building in the campus and a set of 10 devices were

deployed on a nearby campus parking. The results were promising in terms of functionality, although

battery levels were not correctly provided by the devices. After this limited trial a decision was made to

deploy a new batch of 25 parking sensors in a recently created caravan parking lot 400 metres away.

The idea was to provide a parking service for tourism since these parking spots are highly demanded on

weekends and during vacations time. Once all the deployment was set, and the system was finally

configured, a first testing phase under real conditions started. During this time several problems were

detected: on the one hand, a firmware bug caused the sensors to switch-off, which represents a big

problem on a real deployment as they need to be removed and opened to update them. When we did

this we realize some sensors were physically damaged by water and/or stress. Moreover, repaired


16

sensors were redeployed but most of them did run out of batteries before reaching 8 months lifetime

due to the communications protocol design. LoRaWAN networks are not designed to be reliable

networks on the downlink channel, and the protocol relays on acknowledgement messages to assure

delivery. Most of the times messages are delivered, but the sensor is not able to receive the confirmation

and it keeps retrying in a loop, prematurely exhausting the batteries. Some other sensor functionalities

are heavily impacted by the restrictions imposed by LoRaWAN networks when distance from the

gateway increases. The only solution provided by the manufacturer was to limit the distance to a range

were coverage is better. However, this is not in line with how a LoRaWAN network should work, in which

a small set of gateways covers a great area, hence UC decided to cancel the deployment and it has not

been possible to include this parking area on the WISE-IoT rich parking pilot. However, although this

system will be replaced in the future by a different technology which has already been tested in the past,

the knowledge that the university has gathered with this experiment has been of high value. In addition,

although the sensors are not going to be used for this purpose, there is a proposal to use them for

academic purposes with students, so the sensors can be refurbished to make them compatible with the

LoRaWAN system deployed. Nevertheless, although these problems have had a big impact on the pilot,

it is worth mentioning that from a technical perspective this pilot has demonstrated most of the

components created within WISE-IoT project.

2.2.1.2 Smart Parking (KR pilot)

The Smart Parking pilot developed in WISE-IoT aims to provide enhanced parking service experience to

the users and show application mobility between South Korea and Santander over the interoperability

framework that does not require new mobile application installation.

The parking application has been improved during the initial deployment by feedback and bug report

from KETI colleagues and application outsourcing SME members. Mainly, this pilot deployment is using

oneM2M platform in KETI premises which can be interworked with FIWARE Orion Context Broker

through semantic interoperability, and also parking occupancy data populated from Busan oneM2M

platform. The new deployment in KETI premises uses LoRa parking sensors which brings more modules

for the system integration along with problems to resolve during the initial phase.

Official application launch was done by releasing the application to the Google Play Store. Since the

deployment was not a large scale pilot like city-wide, a specific group of users needed to be targeted. To

promote the application to the citizens, it was designed a poster which includes the project introduction

and simple guide for the user application. It was posted at the parking lots that Smart Parking pilot is

associated with.


17

Figure 4. Poster of Smart Parking pilot on Busan Public Parking Lot

Posters are placed in the parking office at the parking lots or in the bulletin board in the facilities where

users can easily notice availability of the Smart Parking application provided by WISE-IoT. Not just in

Busan city, but this also has been promoted on KETI headquarter premises in Seongnam city near Seoul,

the capital city of South Korea. The KETI premises has a oneM2M platform instance used for Smart

Parking service for the visitor parking lot with installed LoRa parking sensors. There are many visitors to

the headquarter, and since there are a limited number of parking spots, like the Busan city centre, it has

provided a useful service to the visitors.

Figure 5. Poster of Smart Parking pilot on KETI Premises

For the evaluation of the use case, it was decided to include them on the application through daily pop-

up menu which links to the google forms service.

• Lessons learnt, regulation issues:

During the Smart Parking pilot deployment and pilot phase, there were several difficulties that were

both technical (e.g. system integration) and non-technical aspects (e.g. user engagement).

Pertaining to the technical problems, the main issue was due to different component integration. The

implementation of the interworking components between platforms (oneM2M to oneM2M, oneM2M

to Orion Context Broker), the adoption of common parking ontology to oneM2M platform, the testing


18

semantic interoperability with Adaptive Semantic Module, the LoRa interworking proxy to oneM2M

platform, and the service server integration took longer time than expected due to bugs mitigation.

Regarding non-technical issues, the biggest challenge was to gather people to use the service. Physically

the parking lots in Busan are far (2.5 hours by high speed train) from Seongnam, KETI premise location.

It was not easy to meet citizen in person like in Santander, so putting posters at parking lots was the

only choice. Also, there were five public parking lots providing occupancy data per spot, not per parking

lot, in Busan. Spot level parking data is needed for the use case implementation. This means that in that

scenario only those five lots are supported by the parking application. From a user perspective, it may

not give such a big value to install and run the new application for a small number of available parking

lots.

Therefore, as complementary deployment, KETI premises was chosen. Visitors to KETI headquarter have

only one visitor parking lot, so they would have bigger needs to use the application than Busan citizens.

The other constraint to the candidate user was the supported mobile OS, Android only. One specific

mobile OS was chosen to provide push services like check-in alert which relies on OS specific features.

IOS users cannot use the application for this reason. Probably, in the future it should be considered to

use mobile web-based application to support any of the mobile operating systems, but there should be

some limitation in terms of UX (e.g. pop-up from background).

2.2.2 Evaluation report

2.2.2.1 Rich Parking evaluation (EU pilot)

• Evaluation phase

The evaluation phase had different steps. First, before the launching of the app, it was shared with

colleagues in order to get some feedback to find bugs or improvements to be fixed. Once the app was

launched, users had the opportunity to provide feedback about the recommendations through the app.

In addition, users were informed about the possibility of sending feedback, suggestions, doubts or

complaints by email, so the developers could be informed and provide solutions rapidly. Actually, some

emails were early received to provide feedback about some appearance improvements that could

enhance the usability of the applications. Of course these emails were answered and some adjustments

were made in the application so a new update of the app was launched.

Finally, the evaluation phase with questionnaires was launched. For this phase, all participants on the

pilot were sent an email with a link to the questionnaire where they could provide their opinion about

the application and the different services offered by it.

• Form distribution

During the engagement sessions of the pilot, if the users wanted to participate on the experience, they

had to complete a consent document were they provided their email and name. They were informed

about the experiment phase, and the evaluation phase during which they would be sent by email a

questionnaire so they could provide feedback about concrete functionalities, experiences, etc.

Once the evaluation phase started, they were sent by email a link to a Google form with 14 questions

covering different aspects and available at [7]. For answering this questionnaire, they had 2 weeks, so

in the second week users were sent a kind reminder for those who had forgotten to answer it.

Instead of sending the questionnaire by email, other idea considered was to include the questionnaire

in the application. However, we did not want the questionnaire to be filled during the pilot phase (due


19

to possible updates on to the app, as it happened), nor ask the user for feedback once in a while that

could overwhelm them.

It was selected Google Form due to the ease to generate the questionnaires, the different possibilities

it offers, the appearance, it is free, and also because it provides a good analysis of the results, which

from the point of view of the developer is very useful.

• Design of the questionnaire

The questionnaire was designed so the KPIs and metrics [3], in which the users of the pilot could

contribute to their evaluation, were included. In addition, some questions related to specific

functionalities provided by the application were added, so we could have more data about their

experience and their opinion for future implementations of the application. A summary of the specific

metrics for the Smart Parking use case are listed in Table 1.

Table 1. Metrics for Smart Parking use cases

Metric ID Metric title Evaluation tool Evaluation owner

SP_001 Diversity of data to users through app

Self-assessment Smart Parking (SM) use case owners

SP_002 Diversity of data to 3rd parties Self-assessment SM use case owner

SP_003 Number of monitored parking spots

Self-assessment SM use case owner

SP_004 App crashes Self-assessment & user reports SM use case owner

SP_005 User’s experience User reports SM use case owner

SP_006 Response time Self-assessment & user reports SM use case owner

SP_007 User mobility support Self-assessment & user reports SM use case owner

SP_008 Number of users Self-assessment SM use case owner

SP_009 Service availability Self-assessment SM use case owner

It was designed a first big set of questions. From this big set, partners involved on the use case had the

opportunity to provide comments about the importance of the different questions. This was a very

important step, taking into account that it was not a good idea to add too many questions, since it could

generate rejection from user’s side. At the end, 14 questions were selected covering user interface,

privacy, functionalities and mobility.

Regarding the obligatory nature of the questions, most of them (the rating ones) were set as mandatory,

while the ones which request text opinions were optional, but the last one where it is asked a general

opinion about what they would change/improve for future implementations.

Following, it is presented the questionnaire in English and the possible responses that users could select.

In addition, in the third column, the KPIs and metrics (defined in [3]) related to the questions are

presented.

• Questionnaire in English and possible response + KPIs & Metrics related to the questions

Table 2. Questionnaire for Rich Parking pilot

# Question Answer options Covered KPIs & metrics

1 How do you value the look and feel of the application?

Rating from 1 (Really bad) to 5 (Very good)

KPI_05 SP_005


20

2 Does it seem intuitive and easy to navigate?

Rating from 1 (Little intuitive and difficult) to 5 (Intuitive and easy)

KPI_05 SP_005

3 How easy is to …?

• Calculate a route

• Analyse a route

• Provide feedback

Rating for each point from Very difficult to Very easy

KPI_05 SP_005

4 Please, indicate the difficulties that you have found when managing the application (if it proceeds)

Text (not mandatory) SP_005

5 How would you value the quality of the routes and parking information provided by the application? (use can provide a different value for each element)

Rating from Really bad to Very good

KPI_04

6 If you want, you can add a comment about the proposed routes or about the parking spots recommended

Text (not mandatory)

N/A

7 Do you think that it is useful to use the same application in different countries?

Rating from 1 (useless) to 5 (very useful)

KPI_04 SP_007

8 Do you think that it can be useful for you or other citizens?

Rating from 1 (useless) to 5 (very useful)

KPI_04

9 Why? Text (not mandatory) N/A

10 Please, value the following functionalities:

• Show parking spots

• Routes calculation

• Walking calculation

• Route analysis

• Feedback provision

• Parking statistics

• Remember car position

• Time expiration notification

Rating from 1 (little useful) to 5 (very useful)

SP_001

11 Do you feel that your privacy is preserved when using the app?

Yes, probably yes, I don’t know, Probably not, No

KPI_04

12 Do you think that the data provided (parking spots and statistics) are reliable?

Yes, probably yes, I don’t know, Probably not, No

KPI_04

13 What percentage of times the application has been force closed because an error attributable to the application?

Less than 10%, Between 10% and 30%, Between 30% and 40%, More than 50%

SP_004 SP_009

14 What changes or improvements would you like to find in the application?

Text (mandatory) N/A

• Questionnaire results

About the participation of users in the pilot, it can be extracted these final numbers:

o During the pilot 41 consent were signed. Although a bigger number of people were approached

as described in Section 2.2.1.1 , finally a total of 41 people signed the consent for participation.

o From the Google Play statistics, it can be extracted that 30 different people installed the app.

o Regarding the number of questionnaires answered, it has been received a total of 27.


21

Following, Table 3 presents the summary of the results from the questionnaire.

Table 3. Results of Rich Parking Questionnaire

• How do you value the look and feel of the application?

*Value 1 indicates really bad look and feel, and value 5 very good look and feel.

• Does it seem intuitive and easy to navigate?

*Value 1 indicates little easy and intuitive, and value 5 very easy and intuitive.

• How easy is to …? o Calculate a route o Analyse a route o Provide feedback

*Results are in the same order from left to right, and from 1- really bad (blue) to 5- very good (purple)

• Please, indicate the difficulties that you have found when managing the application (if it proceeds) → Responses are in Spanish, here is the translation of all of them

- At first it was complicated, but later with the help menu I could use it without problems. - You have to memorize the functionalities of all icons, the sequence of actions is not clear, the actions

over the map are not intuitives.


22

- The navigation through the app is not intuitive nor usable. Even knowing to use it, if a few weeks pass you have to check again, which is not productive.

- The route is not always calculated. - Slow response time. - Although the idea is very good and the appearance is cared, the app is not very intuitive at first, it takes

some time to calculate the route and without route you cannot access the evaluation questionnaire. - The application resets when executing it, without possibility of use. - The icons are little intuitive and I cannot see the cartography ander the grid. It is very difficult to use

for who drives a car, althoug it is not done under way. - It takes time for giving the route.

• How would you value the quality of the routes and parking information provided by the application?

*Results are in the same order from left to right, and from 1- really bad (blue) to 5- very good (purple)

• If you want, you can add a comment about the proposed routes or about the parking spots recommended. → Responses are in Spanish, here is the translation of all of them

- Parking spots are occupied a lot of times and the sensor mark them as empty or the street is under works

- The times I have use it I have not managed to park: or there was not an empty parking spot or there were not enough space

- It does not calculate well. There are not empty spaces where the app says. - They could not be evaluated - The parking spots are not delimited so it misleads

• Do you think that it is useful to use the same application in different countries?

*Value 1 indicates little useful and value 5 very useful.

• Do you think that it can be useful for you or other citizens?


23

*Value 1 indicates little useful and value 5 very useful.

• Why? → Added in () the value reported by the respondent

- It could be, but now, with the low usability and the low reliability of the sensors, I would not use it (1) - I think that in the future it could be useful if it would work correctly, if it would warn you and recalculate

the route when someone gets the spot that has been recommended to you. Also, it would be useful if it could avoid that two people could select the same parking spot (1)

- Sometimes it is difficult to decide the use of the vehicle by parking probability (5) - It is convenience (5) - It would be more useful if it would provide information about the parking spots in all the city, not only

in one area which is also under work and with closed or semi-closed streets. (3) - The idea is good. Bad execution. (5) - The functionality is not new. Or at least this is not evidenced (3) - Especifically if it is extended to other cities, to find more easily places to park (5) - It is difficult to use and little intuitive (2) - It reduces the time to find a parking spot (5) - Because it easy the management (4)

• Please, evaluate the following functionalities o Show parking spots o Routes calculation o Walking calculation o Route analysis o Feedback provision o Parking statistics o Remember car position o Time expiration notification

*Results are in the same order from left to right, and from 1-little useful (blue) to 5- very useful (purple)

• Do you feel that your privacy is preserved when using the app?


24

*Blue = Yes, Red = Probably yes, Yellow = I do not know, Green = Probably not, Purple = No

• Do you think that the data provided (parking spots and statistics) are reliable?

*Blue = Yes, Red = Probably yes, Yellow = I do not know, Green = Probably not, Purple = No

• What percentage of times the application has been force closed because an error attributable to the application?

• What changes or improvements would you like to find in the application? → Responses are in Spanish,

here is the translation of all of them

- I do not have any - It is ok - Information of more parking areas - Stability - More parking areas - Nothing remarkable - It is correct, simple and easy to use - A better design - A graphic user manual with simple icons - A faster route calculation and to extend the parking information to a larger area of the city - The area does not have individual (marked) spaces of parking so it reflects unreal numbers, also the

area is under works so the routes are wrong and the sensors are not there or they are broken - To enlarge the area of functioning - Nothing (x2) - The usability maybe is the most important, it is not intuitive, and it is tedious to remember/guess what

is the next step Refresh the parking spot if it is occupied while we are on route The reliability of the sensors, this is not an app fault, but it has not sense if the sensors are not reliable or if the they do not have a proper maintenance

- First, it should work correctly, secondly it is necessary to rethink the navigation of the app, because for example: it is not usable, you must check three options in a concrete order, in addition when you use the mobile in horizontal it does not rotate, and in a city as Santander which is wide it would be a good idea, or if you use it as a navigator in the car


25

Third, when you select a parking spot to go with the car, other person should not be able to select the same spot. It would be good if when the parking spot gets occupied when you are on route, it would warn you and the route would be recalculate

- More parking spots covered - To start, the parking spots which are marked as empty should be empty. Possibility of changing the

proposed route. - A correct functioning and to be part of a permanent service - Cross data with public parking - Fewer functionalities but they should work alone or very easy - I would do it simpler and I would show only routes with the highest success percentage to find parking - Increase the spots offered - I like everything - Nothing concrete - Show routes more quickly

2.2.2.2 Smart Parking evaluation (KR pilot)

• Evaluation phase

Before the application gets released through Google Play Store, it was tested by members working on

Smart Parking application and WISE-IoT system integration. After the first launch, users provided

feedback by the link accessible through the application, and statistics (e.g. application crach counter)

were automatically gathered by the application itself.

• Form distribution

Feedback form was distributed via the parking application. When the user starts the application, there

is a pop-up having a link to the feedback form made by the Google Forms service. Not to annoy the user,

the user can disable this pop-up which lasts one day. The reason for having the link on the application

was to restrict the feedback participants to the real application users. At the first launch of the

application, but sometimes later, when users see the pop-up they would decide to leave their feedback.

It was selected Google Form service because it offers easy building of questionnaires with user friendly

interfaces and tools for result analysis including bar graphs.


The questionnaire consists of mainly two parts: one is service satisfaction, and the other is user

information. Satisfaction related ones are user’s subjective assessment. This is the other way of KPIs

measuring methodology complementing machine measurement (e.g. response time).

While service related questions are mandatory to answer for application improvement later, user

information remained optional. For both Korean and European users, each question is written in Korean

and English.

The Google Form questionnaire for user feedback can be found at [10].

In the table below service related questions and the possible responses that users could select are

presented in English. In the last column, the relevant KPIs and metrics defined in D4.1 [3] are listed per

question.


26

• Questionnaire in English and possible response + KPIs & Metrics related to the questions

Table 4. Questionnaire for Smart Parking pilot

# Question? Answer options Covered KPIs & metrics

1 Are you satisfied the parking service? Rating from 1 (Not at all) to 5 (Absolutely)

SP_003, SP_004, SP_005, KPI_04

2 Is the data shown on the application trustworthy?

Rating from 1 (Not at all) to 5 (Absolutely)

SP_001, KPI_04

3 Is the app fast enough to show parking information?

Rating from 1 (Not at all) to 5 (Absolutely)

SP_006, KPI_05

4 Has the parking service been unavailable?

Rating from 1 (Mostly unavailable) to 5 (Mostly available)

SP_009

5 Has the app been stopped unexpectedly?

Rating from 1 (So many times) to 5 (Not at all)

SP_004 SP_009, KPI_05

• Questionnaire results

There are 35 participants for the service feedback. Table 5 shows the summary of the results from the

questionnaire generated by Google Forms service.

Table 5. Results of Smart Parking Questionnaire

• Are you satisfied the parking service?

*(1) Not at all, (5) Absolutely.

• Is the data shown on the application trustworthy?


• Is the app fast enough to show parking information?


27


• Has the parking service been unavailable?

*(1) Mostly unavailable, (5) Mostly available.

• Has the app been stopped unexpectedly?

*(1) So many times, (5) Not at all

• Where do you live?

• Where do you use this service?


28

2.2.2.3 Parking Use Case Evaluation (EU & KR pilot results)

Tables below present the selected metrics for the Smart Parking use case, defined in D4.1 [3]. These

indicators are based on some of the use case requirements defined during task T1.1, and cover different

aspects such as data availability, information provision to the user, user experience, process

performance time, mobility support, etc.

Metric Id SP_001 Metric title Diversity of IoT data provided through the app to the user

Definition: This metric measures the diversity of smart parking information that it is offered to

the user through the mobile applications. The following categories must be covered

to fulfil this indicator although more information could be also presented:

• Parking areas: parking areas will be highlighted in the application screen allowing users to click on them and obtain parking information related to that area.

• Parking status: parking status (empty/occupied) information will be provided.

• Parking statistics: this information will be presented when the user click on parking areas. These statistics provide information related to the rotation of free parking spots in the area.

• Crowd info: areas with crowd detectors deployed in the vicinity will provide information related to the number of people around the area

Criteria: The aforementioned information will be made available to the user through the

application screen in different ways of representation. The provision of these data is

crucial to accomplish the main objectives of the use case and to achieve a high grade

of user satisfaction. The initial desired data sets refer to the ones mentioned in the

definition of the metric.

Target Number of different data sets provided to the user >=4

RESULTS Number of different data sets provided to the user = 7

• Parking areas from Santander ✓

• Parking status from Santander ✓

• Parking statistics from Santander ✓

• Crowd information from Santander X

• Parking areas form Busan ✓

• Parking status from Busan ✓

• Parking areas from KETI premises ✓

• Parking status from KETI premises ✓

COMMENTS OF THE RESULTS IF NEEDED

The purpose of the provision of crowd information through the application was to inform users about crowds in some areas where there are no parking spots. This way they could guess the availability of parking spots looking at how many people is in the area. As crowd detectors have not been finally deployed in the city, as reflected in the lessons learnt in Section 2.2.1.1, it has not been possible to provide this info to the users.

Metric Id SP_002 Metric title IoT data for third party developers

Definition: This metric measures the diversity of smart parking information provided by the WISE-IoT platform to third party developers. The following data will be required to be available through the information access layer of the WISE-IoT platform to fulfil this indicator (although more information could be also presented):

- Parking sensor information from Santander and Busan

- Parking areas/lot information from Santander and Busan


29

- Parking statistics from Santander and Busan

- Traffic information from Santander

Criteria: This information will be provided by different IoT platforms involved in the project,

translated thanks to the Morphing Mediation Gateway, and made available following

the common WISE-IoT data model into the information access layer. The initial

desired data sets refer to the ones mentioned in the definition of the metric.

Target Number of data sets provided for third party developers >=4

RESULTS Number of data sets provided through the WISE-IoT platform = 8

• Parking areas from Santander ✓

• Parking status from Santander ✓

• Parking statistics from Santander ✓

• Traffic information from Santander ✓

• Parking lot information from Busan ✓

• Parking status from Busan ✓

• Parking lot information from KETI premise ✓

• Parking status from KETI premise ✓

COMMENTS The aforementioned information is provided through both FIWARE and oneM2M interfaces.

Metric Id SP_003 Metric title Number of monitored parking spots

Definition: This metric measures the total number of smart parking spots monitored in the cities of Santander and Busan and available to the end user through the application.

Criteria: A parking application only makes sense if it can provide meaningful information to

the user. If the information provided to the user only covers a small quantity of

parking spots it makes no difference to the user.

Target >=300 parking spots monitored in Santander

RESULTS • 297 parking spots monitored in Santander

• 203 parking spots monitored in Busan

• 35 parking spots monitored in KETI premises

COMMENTS In the case of Santander, in addition to the legacy parking deployment, during the project a new set of LoRa parking sensors have been deployed. As reflected in the lessons learnt, these new sensors will be replaced by other technology due to technical issues. In addition, due to the works on parking areas, it has been necessary to replace some of the legacy parking spots.

Metric Id SP_004 Metric title Number of application crashes

Definition: Percentage of application crashes during its usage directly related to bugs in the

application or due to problems related with the recommendation systems.

Criteria: During concrete testing actions it will be gathered the number of crashes out of the

total of testing actions. A high percentage of application crashes will result in a low

performance. Also, users will be consulted about their perception about the

percentage of crashes through concrete surveys shared with them. Although this

number will not be too accurate it can give an idea about their experience.

Target <= 10 % of application crashes

RESULTS


30

From questionnaires, in the case of the Rich parking application, 66.7% of people said that the app crashed less than 10% of times. In the case of the Smart parking app 29.4% of people reported not crashes at all, and the 50% some crashes (4 out of 5). So we can say that this metric has been satisfied.

COMMENTS For Rich Parking app: internal testing of the app provided very good results, without crashing events. However, some people reported some problems on the textual responses on the questionnaire pointing that they were not able to open the app (a concrete person, it should be due to the Android version), or that they could not see the map under the grid (due to the permissions that have not been accepted by the user even when they have been warned about it). Some people probably have reported a crash on the app when the route took too much time for being provided.

Metric Id SP_005 Metric title User experience

Definition: This metric aims to rate the user experience when performing the different actions

allowed by the application. Different aspects such as look and feel, or usability of

different functionalities will be considered.

Criteria: The users will be requested to provide feedback related to the appearance and

usability of the application when using different functionalities such as request

parking area info, request of routes, feedback provision, etc.

They will provide a grade of satisfaction from 1 to 10 being 1 no satisfied at all and 10

very satisfied. Value 0 means not feedback provision. The higher levels of satisfaction

will reflect greater success on the application design.

Target User’s satisfaction >=6 (scale 0-10)

RESULTS

In summary, 77.2 % of the respondents agreed on that they were satisfied >=3, and the average mark would be 3.53/5 which is a 7.06/10, so it can be said that the metric has been satisfied.

As a result, 3.81 out of 5.0 (7.62/10) is the overall service satisfaction score, so it can be said that the metric has been satisfied.


31

COMMENTS In order not to provide too complicated questionnaires to the users, it was agreed to change the scale from (0-10) to (1-5). Thus, the target has been scaled to >3 instead of >=6. Checking some textual answers of the Rich Parking app questionnaire, some people argue that some improvements in terms of usability should be applied. They ask for simplicity, even if the application has less functionalities.

Metric Id SP_006 Metric title Response time

Definition: Time the application takes to perform the different actions requested by the user.

Times related to the following parameters will be measured:

- Parking area information request - Route request - Feedback delivery

Criteria: Several tests will be performed internally by developers to measure the time required

by the aforementioned functionalities. As the processing time while using the

application can differ depending on the performance of the mobile phones, there will

be measured the time the system takes to calculate the response but not the one

taken by mobile phones, although some tests can be also provided. In addition, users

will be requested through surveys about their feeling about the response time,

capturing this way the user’s perception.

Target Response time <=10 seconds

RESULTS For Rich Parking app: - Parking area info request → < 1 sec - Route delivery → average 6.65 sec - Feedback delivery → < 1 sec

For Smart Parking app: - Parking area info request → < 1 sec

From the values reported, it can be said that the metric has been satisfied.

COMMENTS These results have been obtained from internal testing. Regarding the Rich Parking, the route provision takes more or less time depending on the distance between the starting point and the parking area. This is because when the distance increase, it also increase the quantity of information to be taken into account, such as the traffic information. Although there was not a specific question about the response time, some of the comments agreed on that it took too much time to calculate the routes. For Smart Parking the value for route delivery has not been reported because it depends on an external routing tool.

Metric Id SP_007 Metric title User mobility support

Definition: This metric validates the user mobility support by ensuring that both applications

(European and Korean approaches) are able to work in both countries and provide a

common set of functionalities independently of the location (Santander or Busan). As

the characteristics of the parking services are different in both cities, there will be

some functionalities which will be different but the following minimum set is required

to fulfil the indicator:

- Getting parking spots status information - Getting route to the selected/recommended parking space - Checking remaining parking time or current parking fee - Providing feedback of parking service


32

Criteria: Basic mentioned services are required to be available in the two applications for both

cities. However, the number of common services can increase so the greater number

of services supporting mobility the greater the success of the user mobility support.

Target Number of services available through both applications and working in both use case sites >=4

RESULTS Number of services available through both applications and working in both use case sites = 5

• Get parking spots status ✓

• Getting route to selected recommended parking space ✓

• Set time ✓

• Receive notification ✓

• Provide feedback ✓

COMMENTS In order to enrich these numbers and add the point of view of the users, the answers to the question available in the Rich Parking questionnaire: do you think that it is useful to use the same application in different countries? are provided

The results show that 92.6% of the users think that the application is useful or very useful to use the same application in different countries.

Metric Id SP_008 Metric title Number of users

Definition: Total number of users using the parking applications

Criteria: Number of applications installations. A higher number of users testing the application

will provide better feedback results and higher values of penetration, allowing more

realistic and accurate conclusions.

Target From WISE-IoT description of work, the target number of users for Santander use cases (including parking applications) is >= 1500 and for Busan >= 4000.

RESULTS For Rich Parking app: - 41 people signed the consent - 30 people installed the app

For Smart Parking app: - 83 people installed the app

COMMENTS In the lessons learnt different reasons for these numbers have been provided. In summary, in Santander the means to safeguard the privacy of users required us to use a paper consent so we could get the email of the users (so they could download the app). In addition, during the pilot it just happened that some main streets on the parking area, where most of the parking sensor are, were under works, which discourage people to participate. In the case of Korea, the far distance from the KETI premises to the parking lots in Busan made difficult to meet people, so posters were used instead of meetings. In addition, in some places the number of sensors were small, so people were not too interested in download an app for a little number of parking spots.


33

Metric Id SP_009 Metric title Service availability

Definition: This metric measures the service availability, which means the percentage of the time that the service is up and running.

Criteria: The service should be available as much time as possible in order to provide a continuous service to the user and do not favour the churning.

Target >=95% of the total time

RESULTS For Smart Parking app, from the fourth question asking about parking service availability:

Most of users 73.6% of users report that the service has been available or mostly available. It is important to make service up and running even with the small number of parking lots provided by application. Regarding the Rich parking app, from internal analysis the service was available most of the time.

COMMENTS For Rich parking app, from the point of view of the users, only once a problem of service availability was reported by email. This issue was solved in less than 10 minutes from when it was reported.


34

2.42.3 Bus Information System

As bus remains one of the most common means of transportation worldwide, this Bus Information

System use case is developed to improve user experience in using and managing buses. To achieve this

goal, both static and real-time information is collected and presented to various stakeholders involved.

In this use case, we collect information from the cities of Santander and Busan and present it to bus

users. The outcomes of the use case are an Android smart phone application, and the evaluation of

review questions from users about the application. In the application, we have developed various

necessary functionalities for bus users including display bus stop information, display bus line

information, display real-time bus information, display remaining time until arrival information, display

current position & follow position, and display path from current position to a bus stop. The details of

these functionalities are described in deliverable D4.1 [3].

To evaluate the application and the use case, we have conducted a pilot to reach bus users and have

them answer review questions about the pilot. The review questions are designed to cover key KPIs and

metrics; initial descriptions about the pilot, the evaluation, and the KPIs and metrics are presented in

both deliverables D4.1 [3] and D4.2 [4].

2.4.12.3.1 Pilot report

• Pilot phases overview

The pilot has three main steps. Firstly, the Android application was developed, then the second step

includes testing and improvements of functionalities, and the last step includes presenting the

application to users and collecting feedback from them. The users’ feedback together with other

necessary evaluations of the use case, pilot, and application are put together for this deliverable.

After development and testing, the application was distributed to users through direct installation of

Android executable file. Even though the main functionalities of the application were already developed

before giving them to the users, sometimes certain bugs and problems were identified by users and

were fixed by us. The pilot were primarily conducted in Korea for this project.

• Engagement activities

To gather users for the application, we primarily utilize social network and chatting tool (Facebook,

KakaoTalk). We usually organize small-scale meetings (Figure 8) with a small group of people once at a

time since it is not easy to gather a large number of people. Also, in total we estimate around 250 users

were involved in the test, but not all of them participated in the questionnaire (207 people answered

the questionnaire). Thus, several members of our laboratory were involved in organizing meetings with

different number of people in different locations. The pilot was conducted from the beginning of

February 1st until the middle of May 15th. Also, for users’ privacy, users’ names and email addresses

were only used for distribution of the forms and were not kept.

In each meetup, we firstly introduce about the WISE-IoT project, then explain about the purpose of the

use case, and finally explanations about the application. To help with our application presentation, we

also use a poster (Figure 6) with figures and texts explaining our use case and application, and a tutorial

video (Figure 7) explaining the functionalities of the application.


35

Figure 6. Poster of Bus Information System Use Case

Figure 7. Screenshots of tutorial video

Due to Android constraint of the application, we informed people who can participate in meetups

beforehand that an Android phones are required. In a few cases, since our laboratory has several

Android phones available, these phones were given to users for some periods of time so that they can

experience with the application. We also received some direct feedback during these meetups that were


36

later reflected in the application (via fixes and improvements) since most of these feedbacks are

relatively small and easy to implement.

Figure 8. Meetups with users

With all of this, there were a total of 207 people participating in the questionnaire.

• Lessons learnt, regulation issues

During the development and testing of the application, the pilot, and the collection of the

feedback/questionnaire, there were not many regulation issues for our members. The law and

regulations in Korea do not really put too much restriction for our pilot; still, we do not keep users’

personal information (names, email addresses) for users’ privacy.

Initially, there were plans on installing crowd detection devices (from NEC) in buses in Busan. However,

after extended discussions, these devices were not adopted by the city’s officials due to the existence

of card readers already in buses. Both card readers and crowd detection devices have an overlapping

purpose of providing an estimation, not an exact number, of the number of people inside the bus at a

time. Furthermore, 3G and 4G internet technologies are quite popular in Korea, especially in public

spaces and on vehicles like buses where WiFi is not available. Thus, the officials were not convinced by

the crowd detection devices’ approach that depends on users’ mobile phones having WiFi turned on at

the time of detection.

As Busan is a popular city in Korea, there were some users for our application that do not currently

reside in Busan but travel to Busan sometimes and are familiar with both the city and its bus system.

This is partly because the bus application covers the entire range of buses inside the city, thus it is

familiar to most citizens who live in or travel frequently to Busan, which provides us with a great

advantage in gathering users for the pilot.

2.4.22.3.2 Evaluation report

• Forms distribution

As there were many meetings with users in different locations, and due to different schedules of users,

we have distributed feedback/questionnaire forms and collected them as we conducted the pilot. Users

were provided the feedback forms and asked to provide their feedback at their convenient time. This is

because we expect a higher proportion of users participating in the feedback for use case evaluation

than if we decide on a small period of feedback collection. As a result, the feedback collection phase was

conducted in the same period as the pilot, from beginning of February to middle of May.

Users were informed beforehand about the collection of feedback, and the feedback forms were

distributed by email or social network (link to the Google questionnaire form), or paper depending on

each user’s specific preferences. We decided to not include the feedback/questionnaire form inside the


37

application in order to separate the two and avoid any problems with the application that can negatively

influence the form.


We have tried to limit the number of questions to be included in our questionnaire in order to promote

users’ participation, as a lengthy form is often unappealing to users. The questionnaire, which contains

8 questions, was also designed to cover all the relevant KPIs and metrics mentioned in deliverables D4.1

[3] and D4.2 [4]; one or more questions can contribute to the evaluation of one KPI or metric. The list of

KPIs is covered in Section 4, and the final list of metrics for our Bus Information System use case is shown

in Table 6. There were updates to BUS_002 and BUS_004 metrics as compared to deliverable D4.2 [4] as

there are questions in the questionnaire that are relevant to these metrics; we decided that it is better

to include user reports for evaluation of BUS_002 and BUS_004 instead of putting many questions

together to evaluate only BUS_005 metric.

Table 6. Summary of metrics for Bus Information System

Metric ID Metric title Evaluation tool Evaluation owner

BUS_001 Data availability Self-assessment Bus use case owner

BUS_002 Diversity of information to users Self-assessment & user reports Bus use case owner

BUS_003 Number of application crashes Self-assessment & user reports Bus use case owner

BUS_004 Processing delay Self-assessment & user reports Bus use case owner

BUS_005 User experience User reports Bus use case owner

BUS_006 User mobility support Self-assessment Bus use case owner

As the KPIs and metrics are clearly defined, the questions are quite straightforward. Table 7 shows the

design of our questionnaire, including all the questions, answer options, and the covered KPIs and

metrics.

Table 7. Questionnaire for Bus Information System


1 Does the application provide sufficient information about bus lines, bus stops, and buses?

Very bad, bad, average, good, very good

BUS_002 & KPI_05

2 What percentages of time does the application crash or suddenly close?

>40%, 30-40%, 20-30%, 10-20%, <10%

BUS_003 & KPI_05

3 How would you rate the responsiveness of the application?


BUS_004 & KPI_05

4 How would you rate the functionalities provided by the application?


BUS_005 & KPI_05

5 How would you rate the difficulties of using the application?

Very difficult, difficult, average, easy, very easy

BUS_005 & KPI_05

6 Do you think that this application is useful to you or other citizens?

Not useful at all, somewhat useful, average, useful, very useful

BUS_005 & KPI_04


38

7 Do you feel that your privacy is preserved when using the application?

Strongly disagree, disagree, normal, agree, strongly agree

KPI_04

8 Do you trust the application and the data provided by it?

Strongly disagree, disagree, normal, agree, strongly agree

KPI_04

The link to the Google form of the questionnaire (both English & Korean) is available at [8]

• Evaluation results

In addition to users’ convenience, the questionnaire was also designed for easy evaluation. Every single

question in the questionnaire has 5 answer options, ranked from the first question being the worst to

the last question being the best. Thus, to display the evaluation results, we assign to each answer option

a number between 1 and 5, the first option being 1 and the last option being 5. The higher the number

is, the better the chosen option is (or the better the related functionality/KPI/metric is). The evaluation

results of each question is summarized in Table 8.

Table 8. Results for Bus Information System questionnaire

Question 1: Does the application provide sufficient information about bus lines, bus stops, and buses?

Average: 4.04

Question 2: What percentages of time does the application crash or suddenly close?

Average: 4.27

Question 3: How would you rate the responsiveness of the application?


39

Average: 4.05

Question 4: How would you rate the functionalities provided by the application?

Average: 3.87

Question 5: How would you rate the difficulties of using the application?

Average: 4.33

Question 6: How would you rate the difficulties of using the application?

Average: 4.19

Question 7: Do you feel that your privacy is preserved when using the application?


40

Average: 4.10

Question 8: Do you trust the application and the data provided by it?

Average: 4.03

• Evaluation of related KPIs & metrics

KPI Id BUS_001 KPI title Data availability

Definition: This KPI verifies the availability of required data for the correct development and execution of the use case. The following data will have to be available:

- Bus information from Santander - Bus information from Busan

Criteria: The availability of the aforementioned information is crucial for the development and execution of the application. More detailed bus information means higher achievement of the KPI objective

Tool: Self-assessment

RESULTS: This KPI is satisfied as bus information from both Santander and Busan is provided.

KPI Id BUS_002 KPI title Diversity of information to users

Definition: This KPI will verify that all information initially agreed is offered to the user: - Bus stop information - Bus line information - Real-time bus information - Remaining time until arrival information

Criteria: The described information will be made available to the user through the application interface.

Tool: Self-assessment & user reports

RESULTS: Except the approximated number of passengers on the bus information that was not available due to the aforementioned overlapping issue with card readers already discussed with the city’s officials, other information is properly provided.


41

Also, the result from users’ feedback of question 1 was 4.04/5, which is sufficiently high for satisfying the KPI. As a result, this KPI is satisfied.

KPI Id BUS_003 KPI title Number of application crashes

Definition: Percentage of application crashes during its usages (related to bugs in the application)

Criteria: This is crucial in maintaining high usability of the application. A failure ratio of over 10% may result in coding reconsideration.


RESULTS: During our testing of the application, the crash ratio was well below 10%. Also, the result of users’ feedback of question 2 was 4.27/5, which is sufficiently high to satisfy the KPI. As a result, this KPI is satisfied.

KPI Id BUS_004 KPI title Processing delay

Definition: The amount of time the application takes to process a functionality requested by a user.

Criteria: The application should be able to complete any functionalities requested by a user with processing delay of no more than 5 seconds.


RESULTS: During our testing of the application, there was no experience of any functionalities taking 5 seconds or more. Also, the result of users’ feedback of question 3 was 4.05/5, which is sufficiently high to satisfy the KPI. One detail worth noting is that the question was designed in relative terms instead of exact seconds as it is not easy for users to measure the delay in exact terms all the time.

KPI Id BUS_005 KPI title User experience

Definition: This KPI guarantees a satisfactory level of user experience when using the system. This includes functional and easy to distinguish application interface and display of information.

Criteria: The application should provide an interface with necessary signs for notable spots inside the cities. Also, bus information should be provided together with a map for high usability.

Tool: User reports

RESULTS: The average result of users’ feedback for questions 4, 5, and 6 was 4.13/5, which is sufficiently high to satisfy the KPI. As a result, this KPI is satisfied.

KPI Id BUS_006 KPI title User mobility support

Definition: This KPI will verify that the application will function in both Korea and Europe using data from the two areas, respectively.

Criteria: The application will function properly in both Korea and Europe, more specifically all the KPIs above will be satisfied for the application deployment in both areas.

Tool: Self-assessment

RESULTS: The application has been tested by us to be able to work in both Korea and Europe. As a result, this KPI is satisfied.


42

2.52.4 Smart Skiing

The Smart Skiing use case aims at setting up a testbed in Chamrousse, a ski resort close to Grenoble in

France. The main idea of the use case to be tested is to gather data related to the skiing performance of

the skiers by using connected sensors carried by the users and to use the information for various

purposes, such as providing recommendations to improve the skiers performance, to compare their

performance with others, provide the location information of the user and/or friends and family, provide

the slopes’ crowdness information to the ski resort, etc. The use case has been detailed in D1.1 [1].

Thanks to the roaming function provided by the LORA tracker devices developed in the project, the

devices can be used seamlessly both in Korea and in France. In addition, the interoperability layer

provided at the application level by the OneM2M standard allows reusing the same ski monitoring

application both in Chamrousse, France and in Alpensia, Korea. The interoperability has thus been

ensured both at the device and application level.


The pilot phase of smart ski use case consisted in 4 main steps:

1. Development of the smart ski application and the backend

For the smart ski application, the first step was to prepare the IoT platform sensiNact for the trial. It

consisted in developing the necessary bridges for the Wise-IoT project, in particular building the

OneM2M bridge and LoRaWAN protocol bridge. In addition, integration of the Solu-M devices with the

sensiNact platform was part of the development phase. The development phase has been terminated

by October 2017.

Figure 9. The overall architecture of the smart ski application


43

Figure 10 Screenshots from the smart ski application

2. Deployment of the LORA Gateway to the Chamrousse station

The second step was the deployment of the LORA gateway to the Chamrousse station. Firstly, CEA’s legal

department has prepared an agreement document to be signed between the parties: CEA and

Chamrousse station operator. The document was for the objective of clarifying several issues such as

responsibilities in cases of damages, malfunctioning of the gateway, insurance, failures, confidentiality,

data ownership, etc.). CEA also needed to do an internal verification to validate various potential risks

related to the deployment (electro-magnetic risks, data protection, interference of radio with third-

party installations, etc.). This legal procedure took about 4 months to be finalized, which caused an

important delay in effectively deploying the LORA antenna to the station. Due to this delay, the gateway

could only be deployed at the beginning of February.

The gateway has been installed at the top of the Chamrousse Mountain. It is at the 2253 m of altitude.

With only one antenna, we could cover a large part of the station. Figure 11 shows the location and

coverage of the antenna (in red). Thanks to its high altitude, it could also cover an important part of the

Grenoble city, at the skirts of the mountain. We could also observe that the antenna coverage could

reach until Lyon international airport which is about 100 km away from Chamrousse.

During the installation, we have encountered some technical issues (network firewall, internet

connection, etc.), as well as environmental issues such as the cold and stormy weather. The installation

has been finalized at the end of February.

Figure 11 LoRa Gateway deployed at the Chamrousse station, 2253 m of altitude


44

3. Organisation of a hackathon to test the application and allow developers building new

application ideas.

In order to test the smart ski application by end users, we organized a special hackathon event on 18 –

20 January 2018. We have collocated the event with the Eclipse IoT Days in Grenoble1. We have

presented the Wise-IoT project and gave the information about the hackathon during the plenary

session of the event (See Figure 12 and Figure 13). On the 18th we have organized two training sessions

to the developers to show the usage of the application as well as the usage of sensiNact APIs. About 10

developers have joined to the training sessions.

Figure 12 Wise-IoT project presentation at the plenary

session Figure 13 Morning session of the training event

After the training sessions, on the 20th, we have invited the participants to test the smart ski application

in real environment at the Chamrousse station and asked them to think about new use cases and

implement them by using the Wise-IoT platform, with sensiNact platform as the entry point. See Figure

14 and Figure 15 for some pictures from the event. Since the installation of the gateway at the top of

the mountain were not finalized due to the legal issues rather than technical issues, we have installed 2

mobile LORA gateways at the bottom of the slopes, which could also cover the main slopes of the station.

During this 1-day event, the idea was to test the application in Chamrousse and receive feedback for

possible improvements in the same day. Thus, the participants had access to the devices, could wear

them and use the application. They could also access directly the data on the platform to create new

services. They have brainstormed with new interesting application cases.

The winning idea was “dynamic slope ranking”, which was about ranking the slopes from 1 to 10 based

on some dynamic difficulty parameters such as snow quality, number of people on the slope, weather,

etc, instead of having only the usual 4 levels: green, blue, red and black. The winning team won a 1-

season ski pass from the Chamrousse station.

1 https://wiki.eclipse.org/Eclipse_IoT_Day_Grenoble_2018

https://wiki.eclipse.org/Eclipse_IoT_Day_Grenoble_2018


45

Figure 14. Presentation of the Chamrousse experimentation

to the participants Figure 15. Presentation of the Chamrousse experimentation

during the hackathon

4. Evaluation of the results (explained in the next sub-section)


The hackathon provided us valuable and encouraging feedback. The participants have filled up a

questionnaire, provided at the Table 9 with some answers, as well as the list of related use case KPIs

and metrics (See D4.1 [3] for more details on those metrics and KPIs).

Table 9. Questionnaire for smart ski hackathon evaluation

# Question and answer options Received answers Covered KPIs & metrics

1 How would you rate the idea of using IoT in the smart ski domain, from 1 (really bad) to 5 (very good)?

3, 5, 4, 5, 5 SK_01, SK_02, KPI_02, KPI_04

2 How was your experience with the smart ski application, from 1 (really bad) to 5 (very good)?

1, 5, 5, 4, 5 SK_01, SK_02, KPI_01, KPI_05

3 Were the APIs you used intuitive and easy to use, from 1 (really bad to 5 (very good)?

5, 5, 5, 4, 4 KPI_01

4 Please, indicate the difficulties that you have found when using the APIs (optional, free text).

Sometimes we got inconsistent data and we were not sure if it was because of the bad weather or not; Except for position data, the unit of measures and their meaning were not very detailed (speed, angle...); It was hard to collect data because of the storm but it’s not the APIs fault.

N/A

5 Do you think that it is useful to use the same smart ski application in different countries (yes, probably, no)?

Yes, yes, yes, yes, yes SK_06, KPI_02, KPI_03, KPI_05


46

6 Do you feel that your privacy is preserved when using the smart ski app (yes, probably, no)?

Probably, yes, probably, probably, yes

SK_01, KPI_04

7 What changes or improvements would you like to find in the application (free text)?

Ideas from hackathon can be applied; extensions such as access to historical or statistical data; details and description about measures provided by sensors; Adding a mean to check if the device is transmitting relevant data or not;

KPI_05

We have received 5 responses over 10 participants. The replies have been provided above. We can

conclude that the participants were convinced about the utility of the IoT in the smart ski domain. They

found the application and APIs easy to use. Some users have encountered few issues, probably related

to temporary issues such as weather conditions or unreliable nature of the LORA protocols. They believe

that the roaming and interoperability features provided by the Wise-IoT is important for cross-border

applications. They were probably not sure about the privacy issues. And they provided some interesting

extension ideas for the application.

After the hackathon, once we have deployed the gateway, by the beginning of March, we continued our

user engagement phase by inviting some CEA colleagues to use and test the application. We have

prepared necessary consent forms, material lending procedure, etc. However, to recruit users was a

challenging task since Grenoble is surrounded by ski stations so there are many alternatives for the

skiers, which reduces the number of potential testers in Chamrousse. Given that we had a limited

number of weekends, in addition to some electrical problems of the installations to which the gateways

was plugged, we could not have concluding results from this second phase of the trial.

However, Chamrousse station showed a great interest to continue with the smart ski experimentation,

in particular use cases related to asset tracking and management in the station. Once the ski season has

ended (so that the resort staff could have more free time to discuss) we have organised 2 meetings with

them. During the first one we have presented them the smart ski resort management use case from our

Korean partners. They showed a great interest for the use case.

In consequence, we have organised a technical workshop in Grenoble by inviting our Samsung SDS

colleagues to organise an integration hands-on workshop. We did the installation of the Brightics

platform of Samsung SDS in Grenoble and done the interoperability tests between the sensiNact

platform to Brightics via our OneM2M bridge. We have successfully integrated the two platforms via

OneM2M.

A picture from the meeting between CEA, Samsung SDS and Chamrousse is given at the Figure 16. We

have presented a demo to Chamrousse and demonstrated the features of the application. They showed

great interest and now we are looking some further collaboration opportunities after the end of the

Wise-IoT project.


47

Figure 16 Smart ski resort management application being presented to Chamrousse staff at CEA premises

Thanks to the Wise-IoT project, the deployment of the LORA gateway at the Chamrousse station has

provide to CEA a very good visibility in the IoT and LORA community in Grenoble. A very important ski

lift management company is now using our LORA gateway to collect data from their sensors deployed

on the ski lifts to do predictive analysis for the maintenance of the ski lifts. Also a startup company from

Grenoble providing LORA-based location tracking devices, is using our gateway to do some experiments

in the Grenoble city.

As a result Wise-IoT project has provided to CEA an excellent opportunity to further exploit the project

outcomes in the context of future collaborations.


48

2.62.5 Smart Resort Management

Basically, this use case is materialized in a web application to enhancing the traceability of resort assets,

valuable belongings, children, disable or senior people location for the resort visitors and managers at

Alpensia, which is one of the ski resorts in Pyongchang Olympic area. The main objective of this pilot is

to assess the feasibility of the WISE-IoT developments from WP2 and WP3 related to interoperability

among the oneM2M platform (Brightics IoT - SDS), sensiNact platform (CEA) and LoRa protocol. Once

the IoT tracker collect the data on the LoRa protocol, the interworking proxy entity will transform the

data into oneM2M standard for Brightics IoT.

The IoT Trackers allow the collection of location and status data from visitors, equipment and facilities.

Deployed in Alpensia area, IoT trackers enable resort managers and visitors to have the rapid access to

the location information on the resort application. Based on that information, the resort managers

increase their controllability to perform actions such as sending patrol, rescue teams, and maintenance

staffs so that they can make right decisions on time, even earlier. On the other hands the visitors utilize

the location information to enhance the resort experience in the ski resort.


As previously mentioned, Smart Resort use case aims to enhance the resort experience of resort

managers and visitors in a ski resort. The application provides functionalities of location monitoring,

location history, lend management, tracker management, statistics, etc.

At the beginning, first target users of the resort application were resort managers. However, since we

wanted to achieve not only feasibility of the technical interoperability, but also a meaningful service for

real users, we focused on how to make the pilot sustainable and reasonable for users, so we spent a lot

of time and efforts to achieve it.

Since ski resorts in Pyongchang area were partially prohibited because of the security policy of the 2018

Pyongchang winter Olympic game, we had strong concerns regarding finding a resort partner who will

participate in the project. We started to develop a technical solution for IoT world but, what if everyone

does not want to participate in the pilot because of the security concerns? That is the reason why we

put life into the resort management pilot.

In order to engage the local resort partners, we contacted the local resort institutes to ask for assistance.

The first meeting with resort managers in Alpensia was set on the 23th of Dec. through several phone

calls to ski resorts in Pyongchang area. First contact point was the rescue team among Alpensia resort

managers (Figure 17). Even though we developed a couple of use cases and user scenario, we needed

to validate the use cases and find out a really valuable solution for the resort industry, so we wanted to

get some domain knowledge by the field specialists to launch the pilot. For this purpose, we prepared a

questionnaire to fill including a lot of basic questions. Some resort managers were worried about the

privacy issues regarding user’s private data that might be collected for the resort application. We

changed the database design and portal processes to avoid storing user’s data at all.


49

Figure 17. The first meeting with resort managers in Alpensia (Rescue team)

Based on a demo presentation and a lot of efforts, the head of Alpensia allows Wise IoT team to launch

the pilot during 2018 Pyongchang Paralympic. Thanks to the WISE-IoT components, the demo

successfully showed them the interworking between different IoT platforms. Also, the resort

management application was designed and developed by using real user’s feedback and iterative

development methods. The reason why we decided to schedule the pilot for the Paralympic game was

that we designed the application to provide the traceability of resort assets, valuable belongings,

children, disable or senior people for the resort visitors and managers in Alpensia (Table 10).

Table 10. Schedule of the resort management use case pilot

Milestone Schedules Tasks

Kick off 27th and 28th of June of 2016 Vision Sharing

Use Case workshop 9th to 15th 2016 Scenario Ideations

Pilot workshop 6th to 13th October 2016 IITP R&D conferences, IoT Week Korea Define scenarios, Define architectures

Development planning November to January 2016 EU / South Korea

PoC / demo to resort managers (iterative approach)

February to July 2017 LoRa Gateway test

Development August to December 2017 Brightics IoT platform (oneM2M) + IoT Tracking resort application


50

Deploy January 2018 IoT Tracker, LoRA gateway, IPE(Interworking Proxy Entity), Brightics IoT integration tests

Pilot 9th to 18th March 2018 Smart Resort (IoT tracker service) Pilot

Interworking b/w France and South Korean

April and May 2018 Alpensia-Grenoble application exchange test

• Engagement activities

To gather users during the pilot, we primarily utilize X-banner and concierge services at a front desk in

Alpensia Hotel (Figure 18).

Figure 18. Hotel concierge service developers stayed and X-banner at the front gate of Alpensia

In order to explain the purpose of the project and resort application to users (resort managers and

visitors both) some of people needed to stay during 2018 Pyongchang Paralympic so that they can teach

how to use the system to improve the operational service quality. Thus, several members of our

laboratory were involved in the field pilot of Alpensia. The pilot was conducted from the 9th of March

until 18th of March in 2018 and the project members who were involved in the project were visited

during the period to collaborate with the concierge service of Alpensia (Figure 19). The main goal was

to manage the resort application and Wise IoT system but sometimes they needed to help visitors by

directing them to location of resort buildings and points of interests. For users’ privacy (the Alpensia

resort raised an issue regarding the data privacy), the users’ information was not included in the

questionnaire forms.


51

Figure 19. Pilot activities during the 2018 Pyongchang Paralympic

With all of this, there were a total of 32 people participating in the questionnaire.

• Lessons learnt, regulation issues

A lot of meetings with resort managers were held in Alpensia. Facility management team, IT

infrastructure and network team, and accommodation service team as well. But all of them could not

make the final Go/NoGo decision for the IoT tracker pilot. They were worried about participating the

project because they were clearly on their way to the global big events (2018 winter Olympic game). The

way we took was an iterative approach to convince the resort managers and CEO of Alpensia. We

prepared concept arts, a lot of demos, physical device samples and successful references related with

resort industry use cases, among others. We reproduced the IoT trackers following the user’s

requirements and redesigned the resort applications several times after the meeting with resort

managers to provide the user-friendly experience. Here are enumerated the final features on the smart

resort management application we delivered for Alpensia resort:

- Users management

- IoT device management

- Show IoT trackers location and lending status on a map

- Show an IoT tracker location history

- Show the person info. Who borrowed the IoT tracker

- Record IoT trackers usage history

In order to avoid the privacy regulations and issues on the pilot, the idea has been to, from the beginning,

not storing user’s data. We consider that this way of handling the location data of IoT trackers has

balanced the gathering of user’s location data which might be critical.


52


• Forms distribution

We had many meetings with resort managers in different locations, we decided to have distributed

questionnaire forms and collected them when we conducted the pilot. Resort managers were provided

the feedback forms and asked to provide their feedback at their convenient time. This is because we

expect a higher proportion of users participating in the feedback for use case evaluation than if we

decide on a small period of feedback collection. As a result, the feedback collection phase was conducted

in the expanded period compared to the pilot, from 9th of March to first week of April in 2018.


We have tried to limit the number of questions to be included in our questionnaire in order to promote

users’ participation, as a lengthy form is often unappealing to users. The questionnaire, which contains

8 questions, was also designed to cover all the relevant KPIs and metrics mentioned in deliverables D4.1

[3] and D4.2 [4]; one or more questions can contribute to the evaluation of one KPI or metric. Once the

evaluation started, they were sent by email a link to a Google form with 8 questions covering KPIs and

metrics in deliverable D4.1. For answering this questionnaire, they had 1 week. Google Form was

selected because of the good looking appearance, free, and also because it provides a good analysis of

the results.

As the KPIs and metrics are clearly defined, the questions are quite straightforward. Table 11 shows the

design of our questionnaire, including all the questions, answer options, and the covered KPIs and

metrics.

Table 11. Questionnaire for Smart Resort Management pilot


1 Are you satisfied the IoT location service?

Rating from 1(Not at all) to 5 (Absolutely)

SRM_001 & SRM_002 KPI_05

2 Is the data shown on the application trustworthy?


SRM_001 & SRM_002 KPI_05

3 Is the app fast enough to show location information?


SRM_001 & SRM_002 KPI_05

4 Has the IoT location service been unavailable?

Rating from 1(Mostly unavailable) to 5 (Mostly available)

SRM_001 & KPI_05

5 Has the app been stopped unexpectedly?

Rating from 1(So many times) to 5 (Not at all)

SRM_002 & KPI_05

6 What type of users are you? Resort managers / visitors N/A

7 Where do you live? South Korea, EU, USA, etc. N/A

8 Where do you use this service?

South Korea, EU, USA, etc. N/A

The link to the Google form of the questionnaire (both English & Korean) is at [9]

• Evaluation results

In addition to users’ convenience, the questionnaire was also designed for easy evaluation. Every single

question in the questionnaire has 5 answer options, ranked from the first question being the worst the

last question being the best. Thus, to display the evaluation results, we assign to each answer option a


53

number between 1 and 5, from the first option being 1 and the last option being 5. The higher the

number is, the better the chosen option is (or the better the related functionality/KPI/metric is). The

evaluation results of each question is summarized in Table 12.

Table 12. Results for Smart Resort Management questionnaire

Question 1: Are you satisfied the IoT location service?

Question 2: Is the data shown on the application trustworthy?

Question 3: Is the app fast enough to show location information?

Question 4: Has the IoT location service been unavailable?

Question 5: Has the app been stopped unexpectedly?


54

Question 6: What type of users are you?

Question 7: Where do you live?

Question 8: Where do you use this service?

• Evaluation of related KPIs & metrics

KPI Id SRM_001 KPI title Data availability

Definition: This KPI verifies the availability of required data for the correct development and execution of the use case. The following data will have to be available:

- location information of tracker - device status (accessibility, battery level) - emergency notification

Unit: Boolean

Criteria: The availability of the aforementioned information is crucial for the development and execution of the application.


55

RESULTS: This KPI is satisfied as location, device status, emergency information of IoT trackers in Alpensia is available

KPI Id SRM_002 KPI title Diversity of information provided to the user

Definition: This KPI will verify that all information initially agreed is offered to the user: - group member information - the number of IoT devices in the area - location information of same group members - device status (accessibility, battery level) - emergency information

Unit: Percentage of information available to the users through the application interface

Criteria: The described information will be made available to the user through the application interface

RESULTS: In order to accept the resort manager’s requirements, we revised the group management concept of skiers to lending IoT trackers, since the resort managers asked us to provide that kinds of functionalities. Thus we delivered users management, IoT device management (the number of IoT trackers in Alpensia, emergency situation info, etc.), IoT trackers rental service, IoT tracker status (accessibility, battery level, lend information, etc.) on a map, and location history & statistics. We provided well-targeted and sufficient information that the resort managers asked us for. As a result, this KPI is satisfied.


56

2.72.6 Lifestyle Use Case

Smart fitness use case has been chosen for Lifestyle use case. Specifically, tennis sport was selected

using the PIQ device for the tennis. The objective of the smart fitness use case is to assess the feasibility

of the smart fitness using PIQ device and IoT-based healthcare platform.

For the use case, it was created a tennis show room at the Daegu global healthcare centre living lab

(Daegu GHC living lab). The PIQ device was installed at the show room and communicated with the

MAPHIS (Most Advanced Personalized Healthcare Intelligent Service) platform. In order to support the

smart fitness use case, the tennis activity data is sent to the PIQ server and the data is then stored at

MAPHIS healthcare platform.


The PIQ tennis service evaluates tennis skill by measuring information about swing activity, and

improves style, speed and spin based on measured data. The test of PIQ tennis service was mainly

conducted on weekday afternoon with people who had visited GHC living lab. The visitors include both

university students and tennis clubs members in Daegu area.

Figure 20. GHC living lab tennis show room

Following, test procedure is described. A tennis player first accesses the PIQ device, creates an event,

registers, and selects the Record Video tab to select themselves from the list. As the player presses the

Record button, the measurement of the data for the swing begins. Three types of data, i.e., style, speed,

and spin, are afterwards displayed on the screen. For example, the style is 1667, the speed being 2532,

and the spin is 3333. In addition to these data, a lot of additional data are collected by measuring tennis

data through various swings such as forehand, backhand, and smashing. After the recording finishes,

the measured data can be checked in the Video Gallery or the PIQ Score tab. The video gallery outputs

the highest data and highest PIQ score in the recorded image.


57

Figure 21. Display data for swing in Record Video

Stored measurement data includes style, speed, spin, total score and PIQ score. Each player checks their

own measured data, and tests again in the same way. Table 13 shows the PIQ tennis data chart collected

for 3 males and 2 females who are university students.

Table 13. PIQ tennis data

M=Male, F=Female

STYLE SPEED SPIN TOTA L PIQScore

A(M) 1667 2532 3333 7532 5564

B(M) 1667 2147 3333 7147 4882

C(M) 1667 1751 1923 5341 3171

D(F) 667 1535 2844 5046 2923

E(F) 667 3334 3333 7334 2891


The questionnaire form was created based on opinions obtained from living lab employees and AIN

researchers after a week using PIQ devices. As an initial methodology, an attempt to send an email with

a Google Form like questionnaire was envisioned. However, a more direct approach was carried out as

many of the participating users were not interested on providing personal information and fulfilling

questionnaires. In this sense, a quick interview with the users using a predefined survey was the chosen

option.

In order to get feedback from users about PIQ tennis service, we prepared questionnaires and received

feedback from users. Testing and surveys were conducted from October 2017 to February 2018. We

conducted a survey of about 100 people who visited the GHC living Lab and about 200 students and

tennis club users. Almost all of them answered the provided survey.

Most of the received feedback was related to how the app (DemoDay PIQ App) represent the

information. As an example, some of the users complained about the lack of historical information (i.e:

the measured data of the last month was not shown to the player). As the application offers the

possibility to send direct feedback to its developer, some other suggestions such as showing information

when the speed is slow, the style needs to be changed or the angle of swing activity is not good were

also provided.

The PIQ questionnaire is presented in Table 14 and the results of the questionnaire, which included

around 300 people, are presented in Table 15.


58

Table 14. PIQ questionnaire

No. Questions to ask Bad Not Good Good Very Good Excellent

1 What about your tennis skills ?

2 Do you enjoy playing tennis?

3 Is tennis lessons from a coach helpful?

4 Is the tennis pose good?

5 How is the network connection quality of PIQ Device and App ?

6 How easy to the PIQ to add a new player ?

7 How is the video gallery ?

8 Does PIQScore appear fine on the Leaderboard ?

9 Is it awkward to wear a PIQ device?

10 Do you accuracy of the desired results of the PIQ App ?

11 How is the UI of PIQ ?

12 Do your satisfaction is using the PIQ ?

13 Is the data from PIQ helpful ?

14 Does the PIQ device seem to improve tennis play ?

Table 15. Results for PIQ device questionnaire


59


60


61


62

Evaluation of the Self-Adaptive Recommender Key Performance Indicators

63

3 Evaluation of the Self-Adaptive Recommender Key

Performance Indicators

This section evaluates the Key Performance Indicators (KPIs) that are considered by the Self-Adaptive

Recommender (SAR) and that are related to the application of the SAR in the use cases. These KPIs have

been defined in Section 3.1.3 of the Deliverable D4.1 [3] and are summarized here for the convenience

of the reader.

KPI Id SAR.KPI_001 KPI title Users’ Trust in IoT Data

Definition: The end-users develop and maintain trust in IoT data by understanding the quality

of information provided by IoT devices. Such provision of trust is necessary for the

uptake of IoT systems.

Unit: Mean Opinion Score using the Likert scale 1-5.

Target: >=4

KPI Id SAR.KPI_002 KPI title Users’ Adherence to Recommendations

Definition: The IoT system technicians and application developers want to develop and

maintain trust in end-user’s adherence to recommended user journeys. Such

adherence trust reflects the confidence that the IoT system offers effective services.

Unit: Ratio R of accepted recommendations over total number of recommendations

Target: The target is an improvement of the users’ adherence to recommendation as the

system is being used and evolved. Such an improvement may be measured with:

average R for second half of measurements >> average R for first half of

measurements.

KPI Id SAR.KPI_003 KPI title Adherence monitoring acceptance

Definition: How many times has the monitoring been allowed by the users (if they want to turn

on the GPS or not)

Unit: Percentage of users

Target: >= 20%

KPI Id SAR.KPI_004 KPI title Users’ Quality of Experience (QoE) from the

Recommendations

Definition: The users’ quality of experience with the recommendations

Unit: Mean Opinion Score (MOS) using the Likert scale 1-5.

Target: The target is an improvement of the users’ adherence to recommendation as the

system is being used and evolved. Such an improvement may be measured with:

MOS for second half of measurements >> MOS for first half.

KPI Id SAR.KPI_005 KPI title Feedback questionnaires completed

Definition: How many times has at least one feedback questionnaire been filled out by a user

(to offer ratings and feedbacks)

Unit: Percentage of users

Target: >= 50%


64

3.1 SAR in the EU Rich Parking Use Case

This subsection evaluates the KPI of the SAR in the field trials in Santander.

3.1.1 Amount of data gathered

Table 16 provides an overview of the gathered amount of data by the SAR in the Rich Parking pilot.

During the pilot in Santander, a total of 303 SAR sessions have been created, which means that 303

recommendations for a route inclusive parking spot have been computed and issued (as the SAR creates

a session for each recommendation). In 68 out of the 303 cases, users have started the monitoring after

receiving a recommendation. In 26 out of the 68 monitoring sessions, users have allowed the mobile

app to send their GPS positions to the SAR system, enabling the system’s adherence monitoring

functionality.

Users who agreed to send their GPS data in a session were presented with a short Supersede feedback

form at the end of the session. For each session, the SAR system chose one of three different feedback

forms (see Annex 7.8), depending on the adherence monitoring results:

- type 1 – a form asking about the user’s satisfaction with the recommended route and the reason

for deviating from it (if at least 50% of the user GPS points were not located on the route)

- type 2 – a form asking about the user’s satisfaction with the proposed parking spot, as well as

the recommended route (if at least 50% of the user GPS points were located on the route, and

the user parked on the proposed spot)

- type 3 – the same as type 2, but also asking about reasons for not using the proposed parking

spot (if at least 50% of the user GPS points were located on the route, but the user parked on a

different spot than the proposed one).

In 16 out of the 26 adherence-monitoring-enabled sessions, users have submitted the feedback form.

Seven of these forms were of type 1 (asking about the route), four of type 2 (asking about the parking

spot), and five of type 3 (asking about reasons for taking a different parking spot).

Table 16: Amount of data gathered in the SAR.

Number of issued recommendations 303

Number of started monitoring sessions 68

Number of sessions with gathered user GPS data 26

Number of submitted feedback forms about route deviations (type 1) 7

Number of submitted feedback forms about parking spot satisfaction (type 2) 4

Number of submitted feedback forms about parking spot deviations (type 3) 5

3.1.2 KPI results

In this subsection, we evaluate and discuss each SAR-related KPI. To calculate the KPI, one person from

FHNW analysed the log files that were generated by the SAR between the 23rd of February 2018 and the

30th of May 2018 (with one exception: to calculate SAR.PKI_001, we used data from the Google

questionnaire). The SAR logs contained information about created sessions, the session monitoring, and

Supersede user feedback. The logs listed all session that were created, sessions where the monitoring

was started, and sessions in which the user agreed to send GPS data. The information from the logs was

manually extracted and put in an Excel file to enable the computation of the performance indicators.


65

SAR.KPI_001: Users’ Trust in IoT Data. When the Supersede feedback forms for the use case were

created, it was decided that questions related to the users’ trust in the overall IoT system should not be

part of the feedback forms in order to keep them as short as possible. Instead, these questions were

asked in the final Google questionnaire provided by the use case owner. The idea was that short

feedback forms motivate more users to submit feedback, which leads to more data points for the

evaluation. Since app users do not want to spend much effort on giving feedback, the probability that

we get a lot of user feedback is larger if the forms are kept as short as possible.

Therefore, we evaluate this KPI with the help of question 12 from the Rich Parking questionnaire, which

asked “Do you think that the data provided (parking spots and statistics) are reliable?” The answers to

this question (provided in Table 3) were measured with a Likert scale from 1 to 5 (no, probably not, I

don’t know, probably yes, yes). 14.8% of respondents thought that the values are reliable and 55.6%

thought that they probably are. 14.8% were not sure whether they are trustworthy or not, and the rest

of them (14.8%) thought that the data provided is (probably or totally) not reliable.

This results in a mean opinion score of 3.67, which is slightly lower than the targeted 4.00.

In the following, we analyse the Supersede user feedback for reasons why the score is not higher. The

feedback forms of type 2 and 3 used star ratings (1 to 5 stars) to ask about the user’s satisfaction with

the proposed parking spot. The answers to this question were used by the Trust Monitor to compute

trust values between the users and the parking spots. This information was in turn used by the SAR to

adjust future recommendations. These three form types used a Likert scale to ask about the user’s

satisfaction with the recommended route. These questions resulted in a total of eight parking spot

ratings and 14 route ratings. The average satisfaction rating of the parking spots was 2.125 stars, while

the average satisfaction rating of the recommended routes was 2.071 stars. To find out the reasons for

these values, we looked more closely at the user feedbacks and coded the free-text answers. Figure 22

shows the resulting categories and number of answers in each category.

Figure 22: Categorized user feedback from the free-text answers.

The majority of the free-text feedback revealed two main problems: i) often, the parking spot was

occupied by another vehicle when the user arrived at the proposed spot (6 answers), or ii) there was not

enough space to park the car because of other cars or obstacles located right next to (and potentially

penetrating) the proposed parking space (5 answers). Furthermore, some users said that they could

either not follow the recommended route or use the proposed parking spot due to ongoing construction

work. Indeed, it happened during the pilot phase that there was quite some construction work going on

in the regions where the parking sensors were located.

For route deviations, the user feedback forms also provided reasons in the form of multiple-choice

options. Two users did not select any option, while five users selected the option “There was a more

direct or faster route”. Some users provided reasons for that selection in their free-text answers:

6

5

2 2

1

TheparkingspacewasoccupiedTheparkingspace wastoosmall/Thesensorwasinabadlocation

TherouteorparkingspacewasblockedbyconstructionworkTheappwastooslow/hangedFoundafreeparkingspotbeforearrivingattheproposedone


66

• One user said that she found a free parking spot before arriving at the proposed destination.

• Another user said that she encountered a blocked street due to construction work while

following the recommended route.

Apart from these reasons, there could be another factor that influenced users to choose the

aforementioned option: although they were informed about it in a briefing at the beginning of the trial,

they might have overlooked that the IoT Recommender takes into account the different traffic loads of

street segments, as well as construction works reported to the external routing framework. The IoT

Recommender can decide to discard routes that include high traffic load or construction work. To the

users, it could then look like the SAR is not proposing the fastest route to the destination.

We can conclude that the Supersede feedback mechanism of the SAR has successfully given insights into

what can be improved in the future.

• First, the current hardware solution related to the parking sensors has some drawbacks: if

cars/obstacles are placed right next to sensors without occluding the sensors, the sensors will

say that there is free space even though the space is not big enough for an additional vehicle.

Also, the current solution cannot cope with the amount of fluctuation in the parking spots. A

free parking spot that gets recommended to a user may already be occupied when the user

arrives at the location.

• Second, the reporting of construction work information to the external routing framework (that

was used by the IoT Recommender) was not well synchronised with the real-world situation,

leading to some recommendations that included blocked streets.

Table 17 summarizes the observed phenomena and rationales that could be extracted from the

Supersede feedback.

Table 17: Summary of observed issues and possible solutions, based on the Supersede feedback provided by users.

Observed phenomena Rationales for the observations Improvement recommendations

Recommender proposed occupied parking spots

• Imprecisely parked vehicles did not trigger the parking sensors

• Empty parking spots became occupied before users arrived at the proposed spots

• Use sensors that are more advanced technologically

• Could introduce a parking spot reservation system

• Recommendation algorithm could be improved to favour locations with bigger concentrations of free parking spots.

Recommender proposed blocked streets

• New construction sites were not reported to the routing framework in a timely fashion

• Better communication with the city authorities responsible for construction work

Even before seeing the user feedback, the pilot owners noticed an increase in construction work sites

during the trials. As a reaction, we updated the routing framework with the information about blocked

streets several times during the trial phase, and parking spots located within construction sites were

removed from the database. If new construction work had gone unnoticed, the Supersede user feedback

would have pointed out the blocked street issues, and the developers could have reacted in the same

way. This result shows a real-world example of a control loop for system evolution (collect-analyze-

decide-act loop, based on [11]). Unfortunately, the issue with the sensors (as pointed out by the insight

stream of the SAR) could not be solved during the trial phase, as upgrading the hardware would have

required a significant amount of work and coordination between many authorities (such a change would

have been impossible to address in such a short amount of time). Nonetheless, assuming that the


67

solution will be improved in the future according to the user feedback we gathered, we have successfully

shown a further example of a control loop for system evolution.

SAR.KPI_002: Users’ Adherence to Recommendations. In total, users have agreed to send their GPS

data in 26 sessions. In ten out of these sessions (38.5%), the users have adhered to the recommended

route. We defined that a user adhered to a recommended route if at least 50% of the user coordinates

sent during the session were located on the recommended route (a user GPS point is considered to be

on the route if the minimum distance from the point to the path of the route is not bigger than eight

meters). We were interested to see if the system can improve its recommendations and increase the

users’ adherence to recommendations over time. Therefore, we have computed the accumulated

percentage of user adherence over time, as shown in Figure 23.

Figure 23: Percentage of sessions in which the users adhered to the recommended route, accumulated over time.

The curve does not show any trend; the percentage is neither increasing nor decreasing. The ratio R1

for accepted recommendations over all recommendations for the first half of measurements is 0.5. For

the second half of measurements, R2 is 0.38. The target (R2 > R1) is not achieved.

Regarding the number of 26 sessions, statistical conclusions how the percentage changes over time

should not be made. There is a use-case specific factor that possibly had a significant effect on the results

and has worked against increasing user adherence: construction work that blocked streets during the

field trial phase. The recommender has used an external street routing algorithm framework that could

have not been updated with the construction work information in a timely fashion. Therefore, the

recommender sometimes proposed routes with blocked segments. The effect of increasing construction

work during the trial phase appeared to be larger than other effects that could have led to improved

route recommendations over time, and is a possible reason for the stagnating graph. Apart from this,

the value of 38.5% for route adherence (versus 61.5% for non-adherence) seems to be quite good, given

the construction work problem as well as possible cases where users might have decided not to follow

the proposed route for other reasons. However, this result must be taken with care due to the low

number of sessions and the fact that the users were aware of participating in a study, which could have

biased them to follow the proposed routes eagerly.

SAR.KPI_003: Adherence monitoring acceptance. This KPI served two purposes. First, a sufficient

number of users who accept the monitoring was required for the measuring of SAR.KPI_002. Second, a

high number of users who accept to send their GPS data shows that the users have trust in our system.

The target value for the KPI was 20% of users. Since we cannot count users due to data privacy reasons,

0

10

20

30

40

50

60

70

Accumulatedovertime:percentageofroute

recommendationsthathavebeenadheredto

Datenreihe118-02-23 18-05-30


68

we count individual sessions instead. Thus, we were interested to see if the users turned on the GPS

transmission in more than 20% of all monitoring sessions.

During the trials, 68 monitoring sessions were started. Out of them, the users accepted to send their

GPS data in 26 sessions, which leads to a value of 38.2%. We therefore reached the target of SAR.KPI_003

(>=20%).

Furthermore, it could be that some of the 68 monitoring sessions were only started because the users

wanted to test the app functionality, and not because they wanted to go somewhere. In this case, the

percentage would be even higher. If we measure the percentage in relation to the overall total number

of recommendations produced by the system, we would end up with 8.5% (26 sessions out of 303

computed recommendations). However, this is not meaningful because it includes many cases in which

the user has asked for multiple recommendations before starting a session, or cases in which the user

just wanted to try out the app without actually driving to a parking spot. Figure 24 shows a route taken

by a user, reconstructed from the transmitted GPS coordinates.

Figure 24: The route taken by a user during the field trials.

SAR.KPI_004: Users’ Quality of Experience (QoE) from the Recommendations. In addition to the

average parking spot and route ratings (star ratings from 1 to 5), we were also interested to see whether

the averages changed over time (SAR.KPI_004) since the SAR adapts its recommendations according to

the user feedback. Figure 25 shows the aggregated average ratings over time.


69

Figure 25: Average ratings of parking spots and routes, aggregated over time

Due to the small number of user ratings (8 for parking spots and 14 for recommended routes), it is not

possible to draw statistical conclusions. If we treat the first data points in both graphs as outliers, we

could approximate the resulting curve with a horizontal line.

There is no visible trend that the ratings improve over time. Regarding route ratings, the mean opinion

score MOS2 for the second half of measurements (2.57) is larger than the mean opinion score MOS1 for

the first half (1.57). Regarding parking spot ratings, MOS2 (2.00) is smaller than MOS1 (2.25). For both

parking spot and route ratings, the target was MOS2 > MOS1.

In the following, we briefly discuss some pilot specific factors that have possibly influenced the results

and thereby obscured the impact of the SAR:

• Ratio between user ratings and available parking spots: one of the core mechanisms for self-adaptation in the Rich Parking pilot is the utilisation of parking spot ratings when computing recommendations. The SAR system less likely proposes parking spots that have previously received low ratings. For this, the SAR takes into account both the overall rating of a parking spot from all users and the ratings of the particular user who is asking for a new recommendation. However, the Santander area contains hundreds of parking spot sensors. Even if the system receives a hundred bad parking spot ratings, the probability that the system selects one of these parking spots and then drops it because of the bad rating is relatively low – because of the total number of available parking spots, it’s likely that the system chooses another spot anyway. There were too few user ratings in relation to too many parking spots in order to perceive the self-adaptation effect.

• Street construction work: due to emerging construction work during the trial, whole streets became unusable. If a user gave a bad rating to a parking spot because it was not reachable, there was still a chance that next time, the system would propose one of the parking spots that are close to the badly rated spot and that are still located within the (same) construction zone. Additionally, the external street routing framework used by the SAR seemed to have not been updated in a timely fashion with the construction work. Therefore, the SAR sometimes proposed routes that passed through construction zones.

• Sensors and fluctuation in parking spot availabilities: the system proposes parking spots to users – parking spots that are unoccupied at the time when the user requests a recommendation. When there is high traffic, there is a good chance that another vehicle parks on the proposed spot before the user arrives. As a result, the user sees an occupied spot and gives a bad rating. This effect puts a significant layer of noise over the self-adaptation effect of the system. Furthermore, vehicles and other objects are sometimes inaccurately placed – for example, in the middle of two parking spots – and might not trigger the parking sensors correctly.

0

1

2

3

4

5

Averageparkingspotratingsovertime

Datenreihe1

0

1

2

3

4

5

Accumulatedrouteratingsovertime

Datenreihe118-02-23 18-05-30 18-02-23 18-05-30

Averageparkingspotratings,aggregatedover time Averagerouteratings,aggregatedover time


70

While the discussed factors obscured the impact of the self-adaptation mechanism, the user feedback

and the insights generated by the SAR have exposed these issues to the pilot and IoT system developers.

The successful exposition and detection of these issues by the developers have set the collect-analyse-

decide-act control loop for system evolution in motion, which allows developers to come up with

improved solutions. Indeed, the pilot app has received an update during the study. However, the effect

of the increasing construction work in the city and the high parking spot fluctuations prevented the

measurement of a higher quality of experience score over time. Thus, while we successfully

implemented the self-adaptation control loop, it did not reflect itself in the user feedback.

SAR.KPI_005: Feedback questionnaires completed. Out of the 26 sessions in which the users agreed to

send their GPS data to the SAR, we received 16 submitted Supersede feedback forms.

This means that 61.5% of users who were presented with a feedback form decided to fill it out and

submit it. The ratio of completed feedback questionnaires is thus higher than the target KPI of 50%.

If we consider all sessions in which the monitoring has started, the ratio would be 68/16 = 23.5%.

However, we cannot assume that the app’s functionality was seriously used in all these 68 cases. Maybe

some of these users have just tried out the app’s functionality to see what happens, and it would not

make sense if they afterwards fill out a form where they rate a route and a parking spot. Since the users

did not want to share their GPS data in 42 out of the 68 cases, we cannot say for sure what happened in

those cases. However, even if we use a pessimistic approach and take the 23.5% as the feedback ratio,

we can say that our use of Supersede competes very well with other feedback approaches. One reason

for this could be that we kept the Supersede forms small and simple. Of course, a possible bias could be

that the trial users had a motivation to send feedback just because they knew that they were part of a

study.

Summary of the answered engineering questions

In this subsection, we briefly summarise the answers that we were able to gather with the SAR to the

engineering questions presented in Table 11 of deliverable D4.1 [3]. Table 18 is an extended version of

that table containing the answers to each question.


71

Table 18: Answers to the engineering questions addressed by the SAR.

Beneficiary Question addressed by SAR

Application Developer

EQ.Users.QoE: How satisfied are the users with the application and the information it offers? Answer: The users provided mediocre ratings for recommended routes and proposed parking spots. The overall rating for the whole application is elicited in the pilot questionnaire whose results are described in Section 2.2.2.1 EQ.Users.Issues: What are the issues that generate user churn? Answer: Proposed parking spots that get occupied before the user arrives. Sensors telling that parking spots are free although there is not enough space to park the car. Blocked streets because of construction work (construction work blocks the recommended route or the parking spot). EQ.Users.Preferences: What are the preferences (likes, dislikes) of users? Answer: The SAR contains ratings for parking spots and route recommendations. Therefore, it is possible to tell what parking spots and routes received very good or very bad ratings. EQ.Streetmap.Correctness: What segments of the streetmap are incorrect? Answer: The biggest issue was that the external street routing framework did not receive timely updates with blocked street information due to construction work. The segments with ongoing construction work that do not contain this information can be seen as “incorrect”. EQ.POI.Correctness: What POI information is incorrect? Answer: Some spaces between cars are too small to park a car. Sometimes the sensors are between two cars and report an empty parking space. Blocked spaces due to construction work have not always been taken into account.

IoT System Engineer

EQ.Entity.Trustworthiness: What is the users’ trust rating in the entity? Answer: The users provided mediocre ratings for the parking spots. EQ.Entity.Rationale: What are the users’ reasons for the trust ratings in the entity? Answer: The submitted Supersede feedback forms contain the users’ reasons. In most cases, the proposed parking spot was occupied or there was not enough space to park the car. EQ.Entity.Plausibility: How plausible are context information offered by the entity? Answer: This is difficult to answer by looking at the SAR data because of the reasons above. In general, the sensors themselves seem to work fine. To get better context information, the functionality of the sensors would need to be extended such that they could sense the size of available spaces (i.e., such that they could see cars and obstacles even if they are not placed directly on top of the sensors).

3.2 SAR in the Smart Skiing Use Case

Initial tests and ongoing discussions with Chamrousse have led to a shift in the focus of the use case

after both the ski resort and the test users have shown relatively little interest in recommendation

functionality.

Ski resort: the ski resort is more interested in resort management functionality similar to what is

provided by the Smart Resort Management use case, while it showed less interest in recommendation

functionalities for skiers.

Test users: in an early test phase before the start of the main field trials, skiers have used a mobile app

that contained three main features: i) collecting and showing statistics of the skier (speed, airtime,

maximum carving angle, and traveled height difference), ii) showing the locations of the skier and assets

that are tracked by LoRa trackers, and iii) slope recommendations. The skiers were mostly interested in

the statistics part of the application and had neglected the recommendation functionality.

Therefore, the Self-Adaptive Recommender has not been actively used in the Smart Skiing Use Case.

Evaluation of General Key Performance Indicators for Use Cases

72

4 Evaluation of General Key Performance Indicators

for Use Cases

Following, from the data gathered from all use cases, the results for the general KPIs described in

deliverable D4.1 are provided.

KPI Id 01 KPI title Diversity of IoT data provided through the WISE-IoT

ecosystem to end users (citizens and developers)

Definition: This KPI measures the diversity of IoT data sets that are offered to the user through

the WISE-IoT ecosystem. The generic term “user” involves both developers, who are

interested in the creation of new added value services, and citizens that benefits from

the project through applications and/or services. It is important to highlight the

importance of the diversity, quantity and quality of data provided to the users. The

interest of those users, both citizens and developers, will rely to a great extent in the

information available on the platform that will allow the creation and use of more and

better services. Throughout the different use cases developed and deployed in WISE-

IoT, diverse categories will be covered. Smart city use cases will provide information

related to parking spots and areas availability, traffic, bus stops and lines, bus arrival

estimations, etc. On the other hand, skiing use cases will provide data regarding

performance of skiers, traffic, assets location, emergency information, etc.

Target Number of data sets available through WISE-IoT >= 10

RESULTS Number of data sets available through WISE-IoT = 15

Parking spots in Santander

Parking areas in Santander

Parking statistics in Santander

Traffic in Santander

Parking lot information (Busan and KETI premises)

Parking spot information (Busan and KETI premises)

Parking spot occupancy (Busan and KETI premises)

Bus stops in Santander and Busan

Bus Lines in Santander and Busan

Bus arrival estimations in Santander

Buses information in Santander and Busan

IoT tracker (GPS location, lending status)

IoT tracker location history

IoT tracker usage history

PIQ data (style, speed, spin, etc.)

KPI Id 02 KPI title Application and service portability in complex IoT

scenarios

Definition: This KPI provides insights about the success of the application and service portability within the use cases developed as part of WISE-IoT project. It is not pleasant for citizens to use different application for same services when they move to different places. However, the heterogeneity of actual IoT scenarios, where data is available through different interfaces/end points and following different data models, obstructs developers to create services which can be used in different scenarios without recoding and adaptations.


73

By providing data interoperability, developers can collect IoT data from different end points through common interfaces and common data modelling, allowing and easing the development of applications that can be portable to different sites.

Target Number of portable application/services >=5

RESULTS: Number of portable application/services = 5

• Smart parking application

• Rich parking application

• Bus information system application

• LoRa tracker interoperability

• Ski monitoring

KPI Id 03 KPI title Reuse and improvement of existing platforms and

deployments in Smart Cities

Definition: This KPI represents the success in the adoption of the different WISE-IoT components by the involved Smart Cities, aiming at improving their already existing IoT ecosystems and transforming them into interoperable ones. One of the key objectives of the WISE-IoT project is to concentrate the efforts in the development of data interoperability building upon existing platforms, hence avoiding reinventing the wheel. Within the current IoT environment there are multiple IoT platforms that have built their architecture upon different premises and following diverse principles. The result is a big number of IoT platforms that provide data and services through many different interfaces and following different data models. One of the most important steps to be followed by researchers within the IoT environment is to seamless integrate these heterogeneous platforms and offer common interfaces to end users without strong changes in their systems.

Target Number of platforms involved in WISE-IoT use cases >= 5 Number of deployments involved in WISE-IoT use cases >= 5

RESULTS: Number of platforms involved in WISE-IoT use cases = 6 Number of deployments involved in WISE-IoT use cases = 7

Platforms Deployments

• FIWARE platform

• Mobius oneM2M

• Oliot (GS1)

• Brightics IoT

• sensiNact

• MAPHIS

• SmartSantander legacy deployment

• Legacy parking deployment in Busan

• Legacy bus deployment in Busan

• LoRa deployment in Santander, KETI premises, Alpensia, Chamrousse

KPI Id 04 KPI title Penetration of added value IoT services in Smart

environments

Definition: This KPI measures the increase of the citizenship acceptance of new IoT services. IoT is starting to attract more attention from citizens since it is starting to have its place in different aspects of their daily activity (e.g. buildings, health, and of course cities). They can see how their cities are being transformed into Smart ones, providing new benefits for the society, environment and economy of the city. The success of a smart city has different legs, and even though the infrastructure is of great importance, the engagement of the smart city services in the citizenship is also crucial. This have to be done through the provision of interesting services which contribute with valuable solutions for their daily challenges. It is known that users can be sceptic to this new


74

IoT paradigm, however, our work is to provide them with interesting services in secure environments and introduce them smoothly to this new framework. The value of this KPI will be calculated through the compendium of different parameters obtained through the number of users of the applications (downloads + installations) in addition to the results of some of the questions in the surveys provided to the users at the end of the pilots phase. Some examples of the questions to be answered could be:

- Would you recommend this service to other users? - Would you pay for this service? - Do you think that your privacy is preserved? - Would you like to have more roaming services available? - Do you think that this service improve your quality of life in some way? - Etc.

Target Penetration of added value IoT services in Smart environments >=7 in a scale (1-10)

RESULTS: Penetration of added value IoT services in Smart environments=7.9 in scale (1-10)

Penetration information extracted from pilots:

• 8.2/10 (Rich parking, from questions 5, 7, 8, 11 and 12 in Rich Parking form)

• 7.85/10 (Bus Information system from questions 6, 7 and 8 in Bus Info form)

• 6.48/10 (Smart Parking results, from questions 1-2 in Smart Parking form)

• 8.8/10 (Smart Skiing results, from question 1 in Smart Skiing form)

• 9.31/10 (From Smart Resort Management results)

• 7.33/10 (Smart Fitness from questions 9,13,14 in Smart Fitness form)

KPI Id 05 KPI title Quality of experience in IoT scenarios

Definition: Although related to the previous KPI, this parameter provides understanding about the quality of the experience of the users when participating in IoT scenarios. In order to make users participant of the IoT environment, and penetrate smoothly within different aspects of their life, the quality of experience of the users (both citizens and developers) must be satisfactory in different facets such as ease of use, delays, interest and quality of information, etc. End users, citizens in this case, will be involved on the use cases deployed within WISE-IoT by using the applications and services developed in the project. These applications and services will be the visible face of the developments of the project, as those non-skilled users will not probably be able to see the benefits provided by the project from a more technical point of view. The value of this KPI will be calculated as a compendium of different parameters related to the performance of the application and services and feedback provided to the users

Target Quality of experience >=6 in a scale (1-10)

RESULTS: Quality of experience >=7.8 in a scale (1-10)

Quality of experience extracted from pilots:

• 8.22/10 (Bus Information system from questions 1-5 in Bus Info form)

• 7.06/10 (Rich Parking from questions 1-3 in Rich Parking form)

• 7.62/10 (Smart Parking from question 3 and 5 in Smart Parking form)

• 8/10 (Smart Skiing from question 2 in Smart Skiing form)

• 9.16/10 (Smart Resort management from question 1-5 in SRM form)

• 6.83/10 (Smart Fitness from questions 5-8 and 11-12 in Smart Fitness form)

Conclusions

75

5 Conclusions

This deliverable has presented the evaluation of field trials aiming at the verification of the success of

the developments of WISE-IoT project in real field deployments. With this objective, this deliverable

provides pilot reports and the evaluation results obtained from these pilots.

For each pilot, a set of reports has been provided. These include how the pilot has been organized, how

participants have been engaged, explanations about the different decisions taken by pilots owners

during the deployment, and the lessons learnt and regulations issues which have been found throughout

its duration. The different evaluation reports have been used to build the results of the evaluation of

general KPIs as well as the use cases specific metrics that were defined in deliverable D4.1 [3]. Each one

of them covers the design of the evaluation phase, how the users were approached to gather feedback,

how the questionnaires forms were designed so the different KPIs and metrics were involved, and how

they were distributed. Finally, per-pilot evaluation, results and statistics have been presented. In this

sense, the information gathered to validate the metrics defined for the pilot has been provided.

Moreover, the results for the general KPIs are offered as a validation of general objectives of the use

cases within the WISE-IoT project. In this regard, most of the KPIs and metrics have been satisfied.

It is important to highlight the value of the deployments and services carried out during the pilots. They

not only have allowed the improvement of the city/industrial infrastructures from the different partners,

but also the interoperability of them.

Users participating in the pilot have been really involved in the pilots, and although the number of them

has been lower than expected due to some reasons already mentioned, they do have showed interest

in the pilot and have allowed us to see their perspective and get feedback about the different services

offered. Through these users, it has been possible to, on one hand, assure that the technical

developments in terms of interoperability has been reached and, on the other hand, to see until what

extent the services offered have been of their interest, and how they can be improved to provide better

services in future implementations.

The results obtained during the evaluation process of the pilots have clearly showed that the WISE-IoT

developments have allowed the creation of new added value services, in which it has been possible to

gather, process and present information from heterogeneous data platforms thanks to the automatic

mediation components developed within the project. Consequently, the main objectives of the WISE-

IoT use cases in particular and the project in general have been achieved.

References

76

6 References

[1] WISE-IoT D1.1 – “WISE-IoT Pilot Use Case Technical Description, Business Requirements, and

Draft High-Level Architecture”, 2017.

[2] WISE-IoT D1.2 – “WISE-IoT High-level Architecture, Reference Technologies and Standards”,

2017.

[3] WISE-IoT D4.1 – “Service requirements and KPIs”, 2017.

[4] WISE-IoT D4.2 –“Field trials with end users”, 2017.

[5] WISE-IoT D4.4 – “Field trials with end users (update)”, 2018.

[6] IoT Santander Meetup. https://www.meetup.com/es-ES/IoT-Santander/ [Accessed 11 June

2018]

[7] Rich Parking Questionnaire. https://docs.google.com/forms/d/e/1FAIpQLSe8nS10LwelW8qOB-

CUJHLugtISgcRHhc2PiHXFSi4XeS0z9Q/viewform?usp=sf_link [Accessed 11 June 2018]

[8] Bus Information System Questionnaire.

https://docs.google.com/forms/d/13HCVQ71x5sSXsprWekpnzM2FfmCowttZaiK5ditXT2o/viewfo

rm?edit_requested=true [Accessed 11 June 2018]

[9] Smart Resort Management Questionnaire.

https://docs.google.com/forms/d/e/1FAIpQLSdhRs7He_AFuh1Q5PeujCNTqjMjLRraQ-s2U9Q-

dcKaoBYo0A/viewform [Accessed 11 June 2018]

[10] Smart Parking Questionnaire.

https://docs.google.com/forms/d/e/1FAIpQLSdu9vLVHZ07ghTVml5PcnV_y74Ql_x6W_clooiaiEw

mQRw3Mw/viewform [Accessed 13 June 2018]

[11] Betty H. C. Cheng et al., "Software Engineering for Self-Adaptive Systems: A Research

Roadmap," in LNCS 5525 - Software Engineering for Self-Adaptive Systems. Berlin, Heidelberg:

Springer, 2009, pp. 1-26.

https://docs.google.com/forms/d/e/1FAIpQLSe8nS10LwelW8qOB-CUJHLugtISgcRHhc2PiHXFSi4XeS0z9Q/viewform?usp=sf_link

https://docs.google.com/forms/d/e/1FAIpQLSe8nS10LwelW8qOB-CUJHLugtISgcRHhc2PiHXFSi4XeS0z9Q/viewform?usp=sf_link

https://docs.google.com/forms/d/13HCVQ71x5sSXsprWekpnzM2FfmCowttZaiK5ditXT2o/viewform?edit_requested=true

https://docs.google.com/forms/d/13HCVQ71x5sSXsprWekpnzM2FfmCowttZaiK5ditXT2o/viewform?edit_requested=true

https://docs.google.com/forms/d/e/1FAIpQLSdhRs7He_AFuh1Q5PeujCNTqjMjLRraQ-s2U9Q-dcKaoBYo0A/viewform

https://docs.google.com/forms/d/e/1FAIpQLSdhRs7He_AFuh1Q5PeujCNTqjMjLRraQ-s2U9Q-dcKaoBYo0A/viewform

https://docs.google.com/forms/d/e/1FAIpQLSdu9vLVHZ07ghTVml5PcnV_y74Ql_x6W_clooiaiEwmQRw3Mw/viewform

https://docs.google.com/forms/d/e/1FAIpQLSdu9vLVHZ07ghTVml5PcnV_y74Ql_x6W_clooiaiEwmQRw3Mw/viewform

Annexes

77

7 Annexes

7.1 Rich Parking PIA

Data Collection

What data is collected for the experiment?

- Direct personal data (user name, emails, address, etc…)

In order to participate in the pilot users have to share their name (or nickname, this is not important)

and their email. They have to sign a consent document with this information.

The email will be used for registering them within the beta group to be able to download the application

on their phones and also for being sent an email with the evaluation form at the end of the experiment.

- Sensitive personal data (sexual orientation, political opinion, etc...)

No sensitive personal data is required.

- Location information (of a user and/or a device)?

GPS data from the mobile is collected, however it will not be related to the user. The user will be able

to enable or not the use of GPS data, however if it is not enabled some of the functionalities will not

work.

- Other users related data

There will be requested personal opinions related to the user’s experience in the use of the application

(i.e. If the recommendations are satisfactory or not and why).

- Data unrelated with the users

Data from parking sensors and traffic sensors deployed in the city.

How is the data collected?

- What sensors are you using?

GPS data from the mobile is collected from the mobile, however it will not be related to the user. The

user will be able to enable or not the use of GPS data, however if it is not enabled some of the

functionalities will not work.

- Are you requesting the data directly from the user?

The data is automatically gathered (if the GPS is enabled) once the app is open to get the initial position

of the user, every time the user requests their position and when the user enables the feedback

gathering until the user stops it.

- Are you using external data sources?

Data from parking sensors and traffic sensors deployed in the city.

Describe security measures used in the data collection phase

Annexes

78

The information gathered from the users when they consent to participate in the pilot (name and email)

is kept secure under a data protection file at the University of Cantabria.

GPS information from user’s phone is gathered (if they agree) when they allow to analyse the route

followed. In order to perform this analysis, the same identifier has to be sent with the GPS coordinates

so they can be related to the same route. However, for each route analysis the identifier is different, so

it is not possible to relate the routes followed with a user. The selection of the identifier is random so it

is not related at all with any information from the user.

When the opinion of the user is gathered, there will be two options. If the user provides an opinion on

a parking spot and it needs to be recorded for future actions (not to be recommended a concrete parking

spot) this information is gathered in the phone, so this information can be sent when asking for future

recommendations. In addition, this information is sent to the SAR, but without identifier. When the user

provides an opinion about the recommendations, but it is not needed for future parking spots

recommendation, the information is accompanied again by a random identifier changed every time

feedback is provided.

Data Storage

How and where are the data stored?


is kept secure under a data protection file at the University of Cantabria.

For storing the GPS data when analysing the route, it is accompanied by a random identifier (which is

changed each time the user wants to analyse a route) and kept in MySQL databases on a virtual server

(running Ubuntu) during around 6 hour in Swiss Datacentres (Zurich & Lausanne), under Swiss

Dataprotection jurisdiction.

For storing feedback of users related to concrete parking spots, that will be use in the future for

recommendations, it is stored at the users phone in a SQLite database, so this information can be sent

when asking for future recommendations. In addition, this information is sent to the SAR, but without

identifier.

For storing feedback related to recommendations but which will not directly used for future

recommendations to the user, the info is kept in MySQL databases on a virtual server (running Ubuntu)

in Swiss Datacentres (Zurich & Lausanne), under Swiss Dataprotection jurisdiction.

Describe security measures used in the data storage


is kept secure under a data protection file at the University of Cantabria. This file has an authorization

access system. Only three people belonging to the project have credentials to access this data file.

Data Usage

What are the data used for?

The application aims to provide the user with routes from one point in the city to a free parking spot to

park the car. The initial point can be provided by the user by moving a marker to the precise point or by

enabling the GPS to gets its actual position. With this information (the initial point), the application

requests a route to the IoT Recommender component through the SAR façade.

Annexes

79

Once the route have been provided, the user can enable the monitoring of the route in order to, at the

end of the monitoring, provide feedback to the system regarding the satisfaction of the route or parking

spot proposed. When the monitoring starts, the GPS position of the mobile is gathered, and when the

user stops the monitoring the feedback mechanisms presents the user different questionnaires

depending on the behaviour of the user during the route. If the user follows the route proposed and

arrives to the proposed parking spot, the feedback system will ask about the satisfaction of the parking

spot. In the case that the user does not follow the route the feedback system will ask about the reason

why the user deviated from the route. This way, the satisfaction with the parking spot will be kept at the

phone and taken into account for future recommendations.

In addition, the data gathered is used for KPIs (related to SAR) calculation and defined in deliverable

D4.1

Are you using profiling techniques?

No, we are not.

Are you verifying the data?

No, we are not

Are you considering secondary/future use?

The only information that could be used in the future are the opinions of users regarding the satisfaction

of the routes proposed, but no personal data from users is gathered. This information can be used in

order to improve the recommendation systems.

Data Sharing

Are you sending/sharing the collecting data with a third party or

publishing the data?

The data is not shared with third parties nor published.

How are you sharing / sending / publishing the data?

The data is not made available to third parties nor open data. The data is exchanged between the

application and the WISE-IoT system. And it is done through https connections

Data Destruction

How long is data stored?

Parking spots scores on the user’s phone are not deleted but the user can delete the app data in the

device settings.

GPS data and route analysis remain stored for a specified amount of time, the default is 6 hours.

Feedback is not destroyed, since it is used for future analysis and improvement of the system, but there

is no personal data nor user identifier on this information.

Data Management

Annexes

80

What regulation / legislation is followed by the experiment to

protect data and user privacy?

For personal data gathered for the participation on the pilot (name and email) GDPR is followed.

The anonymous user opinions and GPS data are stored in Swiss Datacentres (Zurich & Lausanne), under

Swiss Data protection jurisdiction

Who has access to the data for management purpose?

Three people involved in the WISE-IoT project from the University of Cantabria manage the information

gathered from the users when they consent to participate in the pilot (name and email). As it is under a

protected file, only these three people are able to access this information.

Data archive of the SAR logs is stored in cloud system with encryption, and only authorized persons have access.

Describe security measures used in the data management?

For managing the information, provided by users through the consent, the files need authorization for

access them.

All the user feedback gathered by the SAR is anonymous. A developer from FHNW has checked that no

user entered his/her name, address, or other personal information into the feedback sent to the SAR.

Annexes

81

7.2 Smart Parking PIA

Data Collection



To manage usage history, minimum user information is required by the application. User name (does

not have to be real name, sort of a nickname), phone number and car license plate number are required.

The numbers are just used to uniquely identify the users but they are not exposed. This has been

consented when user firstly start the application.

The other gathering of user information (i.e. email and phone number) is through the user feedback

form. This feedback is linked from the application and to provide the information or not is a user’s

choice.




GPS data from the mobile phone is collected, however it does not get stored into the server. The device

location is just sent to the service server to get nearest parking lot(s). The user can enable or disable the

use of GPS data on the user application. If it is disabled, the user needs to search for the parking lot on

the search menu manually and cannot use the navigation service to the selected parking lot.


Parking check-in time is gathered to show elapsed time and estimated parking fee on the application.

However, it is not reported to the server. Parking service history of the user also stored but just locally

on the phone not to the server.

A region of living (i.e. South Korea or Europe) and a region of service usage (i.e. South Korea or Europe)

are collected. Also, personal opinions on service usage are reported during feedback.


Occupancy data from parking sensors are collected to the server. Parking spot number from the BLE

sensor is gathered to the mobile phone, when it is equipped on the parking sensor.



GPS location data of the mobile phone can be collected depending on user’s preference. BLE sensor data

having spot number is sent to the mobile phone application but not to the server.


Personal information is gathered as described above. However, that information is not used for parking

service (e.g. getting nearest parking lot availability). Check-in and check-out timestamp is recorded when

user select relevant menu on the UI. No other data is requested directly to the user.


Annexes

82

Parking sensor occupancy data from the city deployment is used. Also, parking lot information (e.g. used

to estimate the parking fee) is fetched from the IoT server.


The personal information gathered at the first time of application start has been done with user consent.

Geo-location data from GPS sensor on the phone is sent to the service server but without any personal

identifiers (name, phone number). Parking lot availability information is fetched anonymously.

Parking spot number information that user checked-in is stored only to the mobile application to show

the exact parking location to the user. When the parking event is stored on the server, user information

is not associated and stored together.

Data Storage


Personal information with user consent is secured stored on the service server securely which does not

get exposed to any of the service API.

User location data from GPS does not get stored on the server but just used to query the nearest parking

lot information (i.e. parking lot location and occupancy).

When satisfactory level to the parking lot is rated by a user, there’s no user identifier stored together.

The feedback information with parking lot is only stored on the user application locally.


Personal information is secured by credentials and fetching this is secured with credentials which are

allowed only to the person in charge of the database.

Data Usage


GPS geo-location data is used to fetch parking lots near the user (i.e. smart phone). However, no user

identifier is sent to the server for this query. Also this location information is used to call local navigation

application to provide routing service to the selected parking lot, but it gets done locally without sending

the location to any other.

Parking lot rating is sent to the server but there is no user information. And this is used to generate

average rating from users to the parking lot.

Personal information is used to get the number of users only for KPIs measurement. Feedback on service

is used also for the KPIs.


No profiling is used.


No, there is no verification.

Annexes

83


No, it is not considered to do so.

Data Sharing



No data is shared to a third party or published.


Not applicable.

Data Destruction


Locally stored personal and service usage information on the mobile phone will be deleted when the

application gets deleted as well as its local storage.

User (account) information will be immediately removed after KPIs measurement.

User feedback will be removed once KPIs measurement and project reporting gets done.

Data Management



Personal data is protected under Personal Information Protection Act in South Korea.


Only two people, from the SME which built the parking application as outsourcing from KETI, have access

to personal information gathered from the parking application.

Only one person from KETI has access to personal information from service feedback form.


Personal information is gathered under user’s consent and stored into secured database. When the

information is collected there’s no verification with external authority services, so it is not exposed to a

third party.

Annexes

84

7.3 Bus Information system PIA

Data Collection



For the pilot, the users share their names, and some users share their email addresses depending on

what means (link to Google form or paper) they want to use to answer the questionnaire.




GPS data from the mobile is collected, however it will not be related to the user. The user will be able

to enable or not the use of GPS data, however if it is not enabled some of the functionalities will not

work.


There were requested personal opinions related to the user’s experience in the use of the application

(i.e. If the recommendations are satisfactory or not and why)


Data from buses moving around in the city.



GPS data from the mobile is collected from the mobile, however it will not be related to the user. The

user will be able to enable or not the use of GPS data, however if it is not enabled some of the

functionalities will not work.


The GPS information is automatically gathered (if the GPS is enabled) for functionalities that require

them (e.g. display current position).


Data from buses moving around in the city.


The personal data from users (names and email addresses) was kept in a secure desktop at Korea

Advanced Institute of Science and Technology (KAIST) to evaluate the questionnaire results and were

not kept for privacy reason. Only the final evaluation results (e.g. the number of users choosing each

answer option) were kept.

Aside from the personal data for the questionnaire, all other data (including GPS) is not gathered and

only available to users’ personal devices and was not collected by us.

Annexes

85

Data Storage


The information gathered from the users participating in the pilot (names and email addresses) was kept

in a secure desktop at Korea Advanced Institute of Science and Technology (KAIST) at the time of

evaluation, and discarded after evaluation.


The information gathered from the users participating in the pilot (names and email addresses) was kept

in a secure desktop at Korea Advanced Institute of Science and Technology (KAIST). This desktop has an

authorization access system, and only two people belonging to the project have credentials to access

this desktop.

Data Usage


The personal data was used only for evaluation of questionnaire. Other type of data was not collected

by us.


No, we are not.


No, we are not.


We only consider using the final evaluation results (e.g. the number of users choosing each answer

option), which do not contain any specific personal information for possible future use (e.g. to improve

the use case and application).

Data Sharing





The data is not made available to third parties nor open data.

Data Destruction


Personal data is discarded after the evaluation of the questionnaire, while the final evaluation results

(e.g. the number of users choosing each answer option) is kept, which do not contain any specific

personal information.

Annexes

86

Data Management



There were no specific regulations or legislation from Korea that we follow; still, we maintain necessary

data protection and user privacy that can satisfy regulations and legislation that are used in other use

cases (e.g. Rich Parking).


Two people involved in the WISE-IoT project from Korea Advanced Institute of Science and Technology

(KAIST) manage the personal information gathered from users (names and email addresses) to evaluate

the questionnaire results. As it is inside a desktop with authorization access system, only these two

people can access users’ personal information. The personal data is discarded after the evaluation of

the questionnaire.


To access users’ personal information during evaluation of the questionnaire, authorization for the

desktop that contains the files of personal information is required. Users’ personal information is

discarded after the evaluation of questionnaire.

Annexes

87

7.4 Smart Skiing PIA

Data Collection


Asset tracking:

- Location of the asset, e.g., skis, shoes.

Conquer the slope:

- Username

- Password

- User location

- Motion, i.e., acceleration, rotation.


Data Sensor Asking user

Username Form Yes

Password Form Yes

User location LoRa band No

Phone name Phone properties No

Asset location LoRa hooker No

Motion PIQ Robot No

No external data sources are used.


The data are collected using either a private LoRaWAN network or a private Ethernet network.

The data specific to a user, i.e., username and password, are collected using a dedicated form.

Data Storage


The data are stored on one of the platform provided by the project, i.e., sensiNact.

The data are stored in the CEA, on a dedicated server.

Describe security measures used in the data storage?

To access to the data, access rights, through authentication, are required.

Data Usage


The data are used by understating the skiing performance of the skier, as well to monitor the location

of the users and their assets. See D1.1 to learn more about the usage of the data.

Annexes

88


No. We have planned to use the recommendation system provided by the project, however finally we

did not use it. See D2.6 to learn more about the profiling techniques.


No data verification.


No, the data will be deleted by the end of the project.

Data Sharing



The data are shared with the others partners of the project but not with third party.


The data are shared with the others partners with the MQTT protocol.

The data are shared with the final user using the HTTP protocol.

Data Destruction


The data will be stored as long as the project is active.

At the end of the project, May 2018, the database containing the data will be erased.

Data Management



We follow the European regulation about user privacy. We have prepared consent forms to be validated

by the participants


Only the owner of the experiment site has access to the data.


To access to the data, access right mechanism, through authentication, has been put in place.

Annexes

89

7.5 Smart Resort Management PIA

Data Collection


- Location information of assets (latitude, longitude, battery, speed)

- Location information of visitors (latitude, longitude, battery, speed, email)

In order to participate in the pilot users for a resort manager scenario have to get a predefined

id/password from the system administrator in Alpensia

In order to participate in the pilot users for a visitor scenario have to share their email to get the access

link (the email address will be not stored on the resort application system)


In order to participate in the pilot users for a visitor scenario have to share their room number of

Alpensia hotels where they are staying as an identifier



GPS data from the IoT tracker is collected, however it will not be related to the user. The user will be

able to enable or not the use of IoT tracker, however if it is not enabled some of the functionalities will

not work.


The data is automatically gathered (if the GPS is enabled) once the IoT tracker is moving to some places,

the IoT tracker sends their position


No


GPS information from user’s IoT trackers which resort managers attached on their assets (corporate

vehicles and snow canon and etc.) and the visitors borrowed during the stay is gathered. For location

analysis, only the GPS location data stored without any personal identifiers, so it is not possible to relate

the routes followed with a user. In order to find the IoT trackers we used the room number in Alpensia

which is time limited identifier for the stay and it is not related at all with any information from the user.

Data Storage


For storing the GPS data to find the route and position, it kept in PostgreSQL databases on a virtual

server (running Ubuntu).


Annexes

90

GPS information from user’s IoT trackers which resort managers attached on their assets (corporate

vehicles and snow canon and etc.) and the visitors borrowed during the stay is gathered for location

analysis. The GPS data of IoT trackers and the room number in Alpensia which is time limited identifier

for the stay stored in the application we delivered and it is not related at all with any information from

the user (visitor’s information stored in the hotel reservation management system operated by Alpensia

themselves)

Data Usage


The application aims to provide the location data of IoT trackers which resort managers attached on

their assets (corporate vehicles and snow canon and etc.) and the visitors borrowed during the stay.

Once the location information have been provided, the user can enable the monitoring of the location

data to find current location of IoT tracker.


No, we are not.


No, we are not


No, we are not

Data Sharing





The data is not made available to third parties nor open data. The data is used for between the

application and the WISE-IoT system only. And it is done through https connections.

Data Destruction


We do not need to capture the GPS data. We deleted all the location data of IoT trackers after the trial.

Feedback is not destroyed, since it is used for future analysis and improvement of the system, but there

is no personal data nor user identifier on this information.

Data Management



Annexes

91

Personal data not gathered for the pilot.

The anonymous user opinions stored in Google server, under Korea Personal Information Protection Act.


Three people involved in the WISE-IoT project from Samsung SDS manage the information gathered

from IoT trackers. As it is under a protected file, only these three people are able to access this

information.


For managing the GPS information, provided by IoT trackers, all the data have been already removed.

All the user feedback is anonymous. A developer from Samsung SDS has checked that no user entered

his/her name, address, or other personal information into the feedback.

Annexes

92

7.6 Lifestyle PIA

The data collected for the Lifestyle user case includes direct personal data such as user name, emails,

phone numbers, and country. In addition, the data of right-hand or left-hand style should be collected.

PIQ device number is optionally selected to turn on the Bluetooth communication device. To register

with the PIQ service, the pilot must select the sensor direction (left or right) and create a user name

(which can be a nickname) and an email. Then enter the country and mobile phone number (country

code selection) and touch “Save” to complete the player creation. When you create a player, you can

also include a profile picture.

In order to create an event and register the player, the PIQ device and the app must be linked to each

other via Bluetooth. To do so, turn on the device and select the device's ID from the app. Sensitive

personal data (sexual orientation, political opinion, etc...) and location information are not required.

The data is collected directly from the user. The information (name and emails, etc.) gathered from the

users is sent to PIQ server in Europe and kept secure under a data protection. The data is not made

available to third parties nor open data. The data is exchanged between the application and the WISE-

IoT system. And it is done through https connections.

Annexes

93

7.7 Engagement documents

7.7.1 Engagement documents (Santander Use Cases)

Annexes

94

Annexes

95

Annexes

96

7.7.2 Leaflet Rich Parking Use Case

Annexes

97

7.8 Self-Adaptive Recommender Feedback Forms

The following screenshots show the final versions of the three Supersede feedback forms as used by the

Self-Adaptive Recommender in the Rich Parking use case. For the field trials, a Spanish translation of the

forms has been provided.

The main goal of the feedback forms was to receive rationales from users for not following

recommendations. Combining the users’ rationales with the measured deviations from

recommendations allows developers to reason about the causes, to come up with possible system

improvements, and to actually evolve the systems.

Feedback form type 1. When more than half of the user’s GPS data points were located further away

from the recommended route than a certain threshold, this form was presented at the end of the session

(Figure 26). It focuses on the reasons for the user’s deviation from the route.

Figure 26. Feedback form of type 1.

Annexes

98

Feedback form type 2. When the user mainly followed the recommended route and parked on the

proposed parking spot, this form was presented at the end of the session (Figure 27). It focuses on the

user’s satisfaction with the route and parking spot.

Figure 27. Feedback form of type 2.

Feedback form type 3. When the user mainly followed the recommended route but parked on a

different parking spot than the proposed one, this form was presented at the end of the session (Figure

28). It focuses on the reasons why the user took a different parking spot.

Figure 28. Feedback form of type 3

field trials evaluation -...

Documents