d7.2 evaluation of users, results and...

The sole responsibility for the content of this publication lies with the authors. It does not necessarily reflect the opinion of the European Union. Neither the EACI nor the European Commission are responsible for any use that may be made of the information contained therein.

Title: Document Version:

D7.2 Evaluation of users, results and conclusions 1.0

Project Number: Project Acronym: Project Title:

621041 SIMON aSsIsted Mobility for Older aNd impaired users

Contractual Delivery Date: Actual Delivery Date: Deliverable Nature-Dissemination level:

M39 M39+27 R (Report) – PU (Public)

Responsible: Organisation: Contributing WP:

Eva Muñoz ETRA I+D WP7

Authors (organisation):

Eva Muñoz (ETRA)

Alberto Ferreras (IBV)

Marcus Handte (LOC)

Abstract:

The evaluation of SIMON project results consists of a qualitative and quantitative analysis of the data collected during the demonstrations that have been performed during the large scale pilot phase, in order to extract conclusions for the performance of the system not only on fraud reduction, but also on efficiency, reliability, robustness and user acceptance.

Keywords:

Pilot, demonstration, parking card, mobile app, accessibility, disabled

D7.2 Evaluation of users, results and conclusions 2

SIMON. ASSISTED MOBILITY FOR OLDER AND IMPAIRED USERS

Revision History

Revision Date Description Author (Organisation)

V0.1 07/02/2017 Table of Contents Eva Muñoz (ETRA I+D)

V0.2 15/03/2017 Section 2 and section 4.3 Eva Muñoz (ETRA I+D)

Alberto Ferreras (IBV)

V0.3 17/03/2017 Section 3 Marcus Handte (LOC)

V0.4 11/04/2017 Section 4.1 Manuel Vivó (ETRA I+D)

V0.5 25/04/2017 Inputs from cities Joana Ribeiro (EMEL)

Eugenia Capone (PAR)

V1.0 02/05/2017 Final review. Document ready for submission

Eva Muñoz (ETRA I+D)



TABLE OF CONTENTS

1. INTRODUCTION .............................................................................................................5

1.1. Purpose of the Document .............................................................................................. 5

1.2. Scope of the Document .................................................................................................. 5

1.3. Structure of the Document ............................................................................................ 5

2. Preliminary considerations ............................................................................................6

3. Impact indicators...........................................................................................................7

3.1. System performance indicators ..................................................................................... 7

3.2. Application-based Metrics ............................................................................................ 17

3.2.1. Daily Usage .....................................................................................................................17

3.2.2. Monthly Usage ...............................................................................................................18

3.2.3. Weekly Usage .................................................................................................................19

3.2.4. Usage Duration ...............................................................................................................21

3.2.5. Screen Usage ..................................................................................................................22

3.2.6. Screen Navigation ...........................................................................................................23

3.2.7. Server Usage ...................................................................................................................25

3.2.8. Application Actions .........................................................................................................26

3.2.9. Badge Usage ...................................................................................................................27

3.2.10. Popup Questions ....................................................................................................28

3.3. Authority Operator Tool- based metrics for users ....................................................... 30

3.3.1. Activity of citizens regarding the parking badge ............................................................30

3.3.2. Activity of pilot cities controllers ....................................................................................33

3.3.3. Activity of Public Authority .............................................................................................37

3.4. Project performance indicators .................................................................................... 37

3.5. User experience indicators ........................................................................................... 46

3.5.1. Online questionnaire ......................................................................................................46

3.5.2. Other feedback from citizens .........................................................................................53

3.5.3. Feedback from Parking Controllers ................................................................................55

3.5.4. Feedback from Public Authority .....................................................................................56

4. Indicators vs Use Cases ................................................................................................ 58

5. Coverage analysis ........................................................................................................ 62

6. CONCLUSIONS ............................................................................................................. 66

7. REFERENCES AND ACRONYMS ..................................................................................... 67

7.1. References .................................................................................................................... 67



7.2. Acronyms ...................................................................................................................... 67



1. INTRODUCTION

1.1. PURPOSE OF THE DOCUMENT

This deliverable presents the results of the tasks in the evaluation of SIMON project results, which have to do with:

Cross evaluation of project results: As a result of the cross-testing pilot and activities carried out, some evaluation has been further needed. Qualitative and quantitative analysis of the data collected has been performed in order to extract conclusions for the performance of the system.

Evaluation of the user satisfaction, measuring the acceptance and satisfaction with the pilot technologies.

Evaluation of project results: Qualitative and quantitative analysis of the data collected dur-ing the demonstrations has been performed in this task, in order to extract conclusions for the performance of the system not only on fraud reduction, but also on efficiency, reliability, robustness, cost-effectiveness and user acceptance.

Performance of cost-benefit and cost-effectiveness analyses

1.2. SCOPE OF THE DOCUMENT

This document is aimed to show in a descriptive manner the results of the pilot activities in SIMON and to extract the main conclusions that serve as a basis for further deployment and exploitation of the SIMON system, thus justifying the possible decisions taken by a city interested in SIMON services or policy makers interested in the new ICT model for the EU parking card.

1.3. STRUCTURE OF THE DOCUMENT

The document includes the following contents:

Section 2 is the starting point, with a review of the evaluation plan

Section 3 describes the impact indicators that have been evaluated: system performance indicators, project performance indicators and user experience indicators

Section 4 is the link between indicators and the SIMON use cases that had to be demonstrated

A coverage analysis is provided in Section 5, to match fulfilment of requirements with use cases

Finally, some conclusions and lessons learnt are provided.



2. Preliminary considerations

The main objectives of work package 7 are:

The drafting of a detailed evaluation plan to be used as guideline for the tests

Analysis of data collected during the pilots in the four pilot sites.

Evaluation of the system’s performance and the different ICT services in the pilot sites.

Identification of strengths and weaknesses and conclusions over the efficiency of the system as well as its potential impact

The SIMON services and applications have been demonstrated and different metrics have been registered according to a specific evaluation plan detailed in D7.1 Evaluation Plan (1). The data gathered during the pilots have been used to assess the results of the project.

A set of different scenarios was defined in order to describe how several user profiles can demonstrate the use cases that have been implemented at each of the pilot cities. The description of the monitoring procedure as well as the evaluation focuses not only on the system operation but also in the user acceptance and satisfaction; this is why several questionnaires and templates were defined in order to gather the user feedback.

Deliverable D7.1 also proposed a list of impact indicators to be measured during SIMON evaluation phase. Three big groups of indicators were defined in order to have a clear picture on how (a) the project is performing (b) the system is really as expected and results are of impact, and (c) the user is really experiencing an improvement in his /her quality of life. Three main groups are defined in terms of indicators to assess the impact of SIMON project: the system performance indicators, the project performance indicators and the user experience indicators.

Indicators related with the system performance are quantitative and calculations/aggregations must be done from the parameters registered or collected. Project performance indicators can be classi-fied as quantitative (“number of”) or qualitative (“index of”). Finally, indicators related with the eval-uation of the user experience will be usually defined as qualitative.

Collected indicators are evaluated on a monthly basis when possible, which helps to understand the evolution of the users activities and also to detect if there is an “abandon” effect probably due to a lack of interest of the user participating, or on the contrary how effective is a possible event in the attraction of new users for the system. On the other hand, this alsos allow to have some useful information regarding scalability of the system, if users of the system are increasing while their operations are maintained. Indicators are also analyzed per pilot site.



3. Impact indicators

According to the SIMON evaluation plan (1), three main groups are defined in terms of indicators to assess the impact of SIMON project: the system performance indicators, the project performance indicators and the user experience indicators.

3.1. SYSTEM PERFORMANCE INDICATORS

The following tables describe the results obtained for the indicators proposed in the evaluation plan, quantified upon operations by user and month. The method of measurement is described also specifying the parameters registered or collected. Finally, and assessment is included for each indicator. Figures are shown for 2016, when users of the parking badge had activity at some moment in any of the four pilot cities. To complement these measures, also the period from January to March 2017 is available since some additional operations have taken place.

Indicator/Denominator Number of service requests (per month and per user)

Required Measures Data base request

Target 12

Method of measurement The information for this indicator was obtained using the Authority Operator Tool, from the “Avg # of user validations per user” dash-board. In the system “user validations” are all those operations done by the user for: login, parking validation, restricted areas validation and queries, all of them meaning a user service request.

Value See Figure 1 – Average # of user service requests, 2016 and Figure 2 – Average # of user service requests, 2017

Assessment Information is differentiated by pilot city. According to the activities reported in D6.1 Large Scale Demonstration (2) it can be observed how:

- Madrid was the first city to start pilot activities, Parma had some activity from February to June. Both of them had pre-pared some parking badges and were meeting face to face end users to explain how to use the system. Some of those users also registered in the system or were trained in the use of the app.

- Activity in Madrid from January to June is quite relevant, it is due to users that were requested to actively use the app, test different functionalities and provide feedback to the SIMON team.

- In October Lisbon produced a large deployment of the new complementary parking badge.

- Reading trialed the parking badge schema in January 2017.

- There is a decrease in activity during the summer months, but starting in September, Madrid increases again.

- Parma stopped activity until privacy issues and data protec-



tion statements were agreed in November 2016.

- Overall we can say that the average of user service requests is fulfilling the proposed target of 12 by month and user

Figure 1 – Average # of user service requests, 2016

Figure 2 – Average # of user service requests, 2017

0

10

20

30

40

50

60

70

80

90

100

Jan

uar

y

Feb

ruar

y

Mar

ch

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

2016

Average of user services requests

Lisbon

Madrid

Parma

Reading

0

5

10

15

20

25

30

35

40

45

50

January February March

2017

Average of user services requests

Lisbon

Madrid

Parma

Reading



Indicator/Denominator Number of mobile APP downloads

Required Measures Data base request – App Stores

Target As defined in D5.1 (2)

Method of measurement Figures for app downloads have been obtained from Google Play Store and Apple store

Value See Figure 3, Figure 4, Figure 6 and Figure 7

Assessment Figure 3 and Figure 6 show the total installations of SIMON Mo-bile apps since they were released (November 2015 for Android devices, May 2016 for iPhone) until end of March (finalization of SIMON project). SIMON Mobile has reached 1000 downloads for Android and 350 for iOS. It has to be highlighted that the pro-gression for the iOS version is quite similar to the one for An-droid, since in only 11 months it has achieved 350 installations, which is quite relevant and make us realize that actually iPhone devices are often used by disabled users due to their good per-formance regarding accessibility.

Some increases can be observed in the graphs at certain point in time, which can be better explained with the analysis of the indi-vidual graphs for each country

Android version (Figure 4, Figure 5): the highest number of downloads takes place in Madrid, with more than 400 by the end of the project. One big increase occurred in Febru-ary 2016, when face to face meetings started with end user associations. In Lisbon and Parma evolution was quite simi-lar, a first increase took place in Italy in April 2016, due to the promotion activities carried out. Also in Lisbon, a cam-paign in social networks was done by the end of August 2016, which resulted in more app downloads. Finally, the SIMON Final event, hold in Madrid on March 2017, has also caused the last big increase in the number of installations in Madrid. In the UK, a news report (including a radio inter-view) in the BBC had a great impact in the number of instal-lations by the end of November 2016.

iOS version (Figure 7 and Figure 8): information for iOS in-stallations per country is not complete, since Apple privacy policies are not considering in this account those devices that did not share their position. In any case, trends can also be analyzed. The higher number of downloads seems to happen also in Spain. Dissemination and promotion activi-ties have an impact in all the countries in the same line as for the Android app.



Figure 3 – Total installation of SIMON Mobile app, Android version

Figure 4 – Total installation of SIMON Mobile app per country, Android version

0

200

400

600

800

1000

1200

Total installsGoogle Play Store

0

50

100

150

200

250

300

350

400

450

Total installs per countryGoogle Play Store

ES GB IT PT



Figure 5 – Total installation of SIMON Mobile app per country (cumulative), Android version

Figure 6 – Total installation of SIMON Mobile app, iOS version

0

100

200

300

400

500

600

700

800

900

Total installs per countryGoogle Play Store

Spain UK Italy Portugal

0

50

100

150

200

250

300

350

400

Total installsApple Store

Total installs



Figure 7 – Total installation of SIMON Mobile app per country, iOS version

Figure 8 – Total installation of SIMON Mobile app per country (cumulative), iOS version

05

101520253035404550

Total installs per countryApple Store

Italy UK Portugal Spain

020406080

100120140160

Total installs per countryApple store

Italy UK Portugal Spain



Indicator/Denominator Number of accesses of SIMON Website / platform


Target At least one per user to access tutorials (learning use cases)

Method of measurement Number of accesses to the YouTube channel where the video-tutorials are hosted, using the Analytics tool provided by Google.

Value See Table 1 – Indicators about video-tutorials visualization

Assessment Users in Madrid were active from January to June in a first stage of the pilot activities. The indicator “Active users” in the AOT dash-boards means those badge holders that have generated any event which has registered in the SIMON backoffice. Lisbon and Parma us-ers were active starting November 2016 until March 2017, and Par-ma users have been mainly active in January and February 2017. Ac-cording to this registered activity, there is also a correlation with visualizations of the video-tutorials in the same periods, as shown in Figure 10:

In February 2016, a high activity is registered when Spanish asso-ciations were asked to recruit users and to prepare face to face meetings.

From March 2016, Spanish video tutorials had many visualiza-tions and the English video for Navigation routing was also one of the most accessed.

After summer 2016, when deployment of the badges started in Madrid, number of active users was increased in Madrid an also Spanish video tutorials related with the use of the parking card were visited. Also to remark the increase in visualizations in March 2017, when the SIMON Final event took place.

Figure 9 – Active users in 2016 and 2017(all cities)

0

5

10

15

20

25

30

Jan

uar

y

Feb

ruar

y

Mar

ch

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Jan

uar

y

Feb

ruar

y

Mar

ch

2016 2017

Active users

Lisbon

Madrid

Parma

Reading



Figure 10 – Video-tutorials view in SIMON Youtube channel

Figure 11 – Legend for Video-tutorials view in SIMON Youtube channel



Name of the video Duration of visualization (minutes)

Number of visualiza-tions

Average duration of views

SIMON Mobile (Español) Tutorial 1: Estructura y funciones 154 (4,6%) 118 (5,4%) 1:18

SIMON Mobile (Español) Tutorial 2: Explorar mapas 94 (2,8%) 68 (3,1%) 1:22

SIMON Mobile (Español) Tutorial 6: Uso de la tarjeta de aparcamiento

143 (4,3%) 67 (3,1%) 2:07

SIMON Mobile (Español) Tutorial 3: Rutas 127 (3,8%) 52 (2,4%) 2:26

SIMON Mobile (Español) Tutorial 8: Uso de la tarjeta de aparcamiento (QR)

43 (1,3%) 50 (2,3%) 0:51

SIMON Mobile (English) Tutorial 1: Structure and features 62 (1,9%) 49 (2,2%) 1:15

SIMON Mobile (Español) Tutorial 7: Uso de la tarjeta de aparcamiento (NFC)

32 (1,0%) 42 (1,9%) 0:45

SIMON Mobile (Español) Tutorial 4: NAVEGACION HORA-RIOS

51 (1,5%) 41 (1,9%) 1:14

SIMON Mobile (English) Tutorial 6: Disabled badge (Valida-tion and access)

39 (1,2%) 22 (1,0%) 1:45

SIMON Mobile (English) Tutorial 7: Disabled badge: Valida-tion and access with NFC

23 (0,7%) 18 (0,8%) 1:15

SIMON Mobile (English) Tutorial 3: Navigation (routes) 50 (1,5%) 17 (0,8%) 2:56

Table 1 – Indicators about video-tutorials visualization



Indicator/Denominator Number of validations with SIMON services (per month and per user)


Target 8

Method of measurement SIMON validation services are those related to the use of the parking badge to validate in a reserved parking place and to access a restrict-ed area. The SIMON backoffice has registered all these validations that can be consulted through the AOT, through the metrics Avg # of parking validations per user and Avg # of access control validations per user.

Value See ¡Error! No se encuentra el origen de la referencia. and Figure 12

Assessment The proposed target of validations to be made by users in the system has not been met in most of the cases, only in February in Madrid the activity was significant at this regard (11,6 validations in average). With the exception of the months in summer, we can say that most of the users in Madrid tried the validation functionality and produced some events related to badge validations. This same behavior can be inferred for users in Lisbon (from October) and Parma (from Decem-ber).

Figure 12 – Avrg Number of validations with SIMON services per month and user

0

2

4

6

8

10

12

14

Jan

uar

y

Feb

ruar

y

Mar

ch

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Jan

uar

y

Feb

ruar

y

Mar

ch

2016 2017

Average of validations per active user

Lisbon

Madrid

Parma

Reading



3.2. APPLICATION-BASED METRICS

To evaluate the usage of the Simon Mobile application, we have instrumented it with a light-weight logging mechanism that captures different actions performed by the users in a database. The logging mechanism regularly synchronizes newly logged events with a Locoslab service that stores the logs of different users and makes them available for a later analysis. The events logged by the instrumentation include the following categories:

Server interaction (GET_DIRECTION, UPDATE_DIRECTION, GET_LOCATION, GET_OVERLAY, GET_SCHEDULE)

Scans of badge (NFC_SCAN, QR_SCAN)

Validation/access process (STEP_REQUEST, SUCCESS_VALIDATE, SUCCESS_RELEASE, SUC-CESS_ACCESS)

Screens displayed (ACCESS, BADGE, BOOKMARK, DIRECTION, EXPLORE, HOME, LICENSE, NAVIGATION, PRIVACY, ROUTE, SETTINGS, STARTUP, STOPTIME, TRACK, TUTORIAL, USAGE, VALIDATION)

Help displayed (OPEN, CLOSE)

Dialogs displayed (BOOKMARK, CREDENTIALS, DATE, FEEDBACK, NFC, OPTIONS, PLATE, SEARCH, TIME)

Actions accessed (SHOW_MAP, SET_SOURCE, SET_TARGET, SET_NAVIGATE, SET_STOPTIME, DISCARD_BOOKMARK, RENAME_BOOKMARK, ADD_BOOKMARK, CLEAR_SELECTION, RE-SET_TIME, ZOOM_IN, ZOOM_OUT, REFRESH_DATA, EXPAND_VIEW, COLLAPSE_VIEW, RE-PEAT_VOICE, SHOW_TRACK, SHOW_DIRECTION, START_NAVIGATION, END_NAVIGATION)

For each event, we log a sequence number, the time, the application version as well as a globally unique identifier that represents a specific app installation. In addition, we log data that is specific to the type of event, for example, to differentiate menu clicks from drawer clicks or to identify the movement modality for a route request. This allows us to analyze the sequence in which events oc-curred as well as the duration that a user was interacting with a particular application function.

To differentiate users belonging to the different target cities, we analyze the event logs of each user for their language settings. These settings are transferred when a user interacts with the navigation services. Based on the languages supported by the application, we then categorize the users into Spanish, English, Italian, Portuguese and others.

To filter out the early development stages of the Simon Mobile application, which could distort the evaluation results, we limit the analysis of the event logs to application versions that have been made available publicly. This means that we only consider events from Android apps whose version is above V0.33. This effectively sets the start of the evaluation period to the beginning of August 2015. Since there was no beta phase for the iOS application, a similar filtering does not have to be per-formed for this platform.

3.2.1. Daily Usage

As a first metric, we look at the number of daily active users. To compute this, we compute the num-ber of individual users who have been active on a particular day. From a computational perspective, this means that their event log contains at least one entry whose time lies between 0.00h and 23.59h of that day.



Figure 13 – Daily Active Users

As indicated by Figure 13, the total number of daily active users starts low in August 2015 with 3-5 users and keeps on growing over the whole evaluation period. Two noticeable peaks are at the end of August 2016 and between November and December 2016. Which are likely to be related to the project activities (i.e. a workshop in August and the participation in the transport conference in No-vember 2016 and the organization of the take-up seminars in December 2016).

3.2.2. Monthly Usage

When looking at the daily usage numbers, the question arises whether the users are repeatedly using the application over several days or whether they are different individuals. To determine this, we ex-tend the aggregation period from one day to 30 days which approximates a monthly period.

Figure 14 – Monthly Active Users

When looking at the total numbers of monthly active users show in Figure 14, the daily users are ac-tually both, individuals that are using the application repeatedly as well as different users. Similar to the daily numbers, the monthly numbers show the trend of an increasing number of users that peaks



in November 2016. The drop in February 2017 can be explained by the fact that this monthly period is incomplete and thus shorter than the other periods.

3.2.3. Weekly Usage

In order to analyze the participation at the different testing sites, we take a closer look at the weekly numbers. To start off, we first take a look at the total numbers – which show a similar slope than the monthly numbers. As discussed previously, Figure 15 does not exhibit the noticeable drop in Febru-ary that is visible in Figure 14 which is a clear indicator for the fact that the low numbers are a result of the aggregation and not the actual usage.

Figure 15 – Weekly Active Users

Figure 16 – Weekly Active Users (Spanish)

When looking at the number of Spanish users shown in Figure 16, it becomes apparent that the main usage period starts in February 2016. The usage remains high over the whole evaluation period but it exhibits a noticeable drop between end of July and end of September 2016. We speculate that this



might be a result of the summer period, in which a significant number of users might be on vacation and thus, not be using the Simon Mobile application.

Figure 17 – Weekly Active Users (Italian)

When looking at the number of active users in Italy shown in Figure 17, we can see that the total number of users is about 1/3rd of the number of Spanish users. Similar to the numbers in Spain, the number of users ramp up at the beginning of 2016 and keeps on being equally high. A noticeable drop seems to be present around the Christmas time of 2016.

Figure 18 – Weekly Active Users (Portuguese)

When looking at the number of active users in Portugal shown in Figure 18, we can see that the users seem to become more active around August 2016. However, the total number of active users is less than the Italian or Spanish counterpart.



Figure 19 – Weekly Active Users (English)

When looking at the number of active English speaking users shown in Figure 19, we can see that there is a continuous usage throughout the project period. A noticeable peak is in November 2016 which corresponds to a media report in the UK that highlights the Simon Mobile application.

3.2.4. Usage Duration

Apart from looking at the activeness of the users in aggregate, it is also interesting too look at the pe-riods of time the individual users remain active. To compute this, we take a look at the timestamps of the first and last event produced by a particular installation and group the durations by the number of weeks. As shown in Figure 20, the usage duration exhibits a long-tail distribution in the sense that about 50% of the users have been using the application for 7 days or less. The remaining 50% keeps using the application for a varying amount of time with about 20% of users that have been using the application for 12 weeks or more.

Figure 20 – Usage Duration

Given the fact that Simon Mobile is a free application on both, Android and iOS, this user behavior can be expected, i.e. a significant number of users will download the application and then quickly un-



install it, if it does not meet their expectations. As indicated by the data, this affects about 50% of the installations. Interestingly, the remaining installations are kept and used for a considerable amount of time.

3.2.5. Screen Usage

Besides from the pure usage numbers, it is also interesting to look at how the users are interacting with the application. As a basic metric, we count the number of times a particular application screen has been accessed by the users. Figure 21 and Figure 22 depict the numbers for the four user groups.

Figure 21 – Screen Accesses (Spanish and Portuguese)

Intuitively, as an artifact of the application structure, the home screen (Home) is the most accessed screen of the application. Independent from the language, this is followed by the number of accesses to the navigation menu (Navigation) and the badge menu (Badge). Depending on the user group, the exact order between these menus may differ.

For Italian, Spanish and Portuguese users, the menus are closely followed by the startup screen (Startup) which is shown when the app launches. This indicates, that most users seem to start the application, go to the desired function and close it again with little browsing.

More interestingly, Spanish and Portuguese users seem to be using the validation screen (Validation) more frequently than any other application function. For English and Italian users, the route compu-tation (Routing) is the most frequently accessed function.

For Italian users, the access request screen (Access) is almost equally important than the validation whereas for all other user groups, the validation is the most frequently used badge function. The his-tory function (Usage) which shows the issued validation and access requests has a usage that is close to the access screen in all countries.



Figure 22 – Screen Accesses (Italian and English)

The least used screens are the screens related to the privacy policy (Privacy), the application license (License), the bookmarked places (Boomark) and the application tutorial (Tutorial).

With respect to navigation functions, the map-based exploration (Explore) and the route computa-tion (Route) tend to be much more frequently accessed than the timetable view (Stoptime). Similarly, since the step-by-step navigation is result of using the routing computation, the amount of accesses to the map-based navigation view (Track) is lower than to the routing view. Interestingly, the textual navigation view (Direction) is accessed significantly less frequently than the map-based alternative.

3.2.6. Screen Navigation

Besides from looking at the accesses, another interesting view on the application usage is to look at the way how people are navigating the application. Figure 23 shows this in aggregate for all users. The nodes in the graph represent the individual screens and the edges represent the transitions be-tween them. The thickness of the edges represents the relative transition probability and the size of the nodes represents the average usage duration.



Figure 23 – Transition Probability and Relative Stay Duration

As indicated by Figure 23, in aggregate users are spending most of the time with the home screen, exploration, routing and validation. To get to these functions, they usually go through the corre-sponding sub menu. However, the thinner edges between validation, access and usage also indicate that people are jumping directly between the validation functions. Similarly, some users are directly jumping to the route view from the exploration view, which can be done by selecting a destination and selecting the route option from the action bar.

As also indicated by the raw access numbers, when performing a step-by-step navigation, users are spending more time on the map-based track view than the text-based direction view. However, some users seem to be switching directly between these views (which is supported by the navigation bar during step-by-step navigation).



Figure 24 – Transitions and Duration (left: Spanish, right: Italian)

When comparing the graphs for each user group, the results closely mimic the results from the num-ber of accesses. For example, as shown in Figure 24, the Italian users are much more likely to use the access control functionality than the Spanish users. This is an immediate result of the deployment, since app-based access control is a major benefit in the Parma deployment.

3.2.7. Server Usage

Besides from the navigation through the application, another set of interesting observations can be made by looking at the number of accesses to the navigation services depicted in Figure 25. As indi-cated by the data, people are frequently tapping the map to set a marker. This then results in an ac-cess to the address resolution function (Get_location). In addition, when the users are displaying the map, they will be frequently downloading the markers with stops, elevators, etc. for a particular loca-tion (Get_overlay).

Figure 25 – Navigation Service Usage



Comparing the number of requests for routes (Get_route) with the number of requests for timeta-bles at stops (Get_schedule) shows the same picture that was already apparent when looking at the accesses to screens, namely, that the users are much more likely to compute routes than to lookup individual schedules.

3.2.8. Application Actions

Finally, in order to understand which functions users are (not) using, Figure 26 shows the number of actions accessed by the users. As one might guess, the most commonly used actions are the back button and the menu button which are needed to navigate through the application.

These two actions are closely followed by the zoom in and zoom out function of the map. Interesting-ly, the zoom in button is used much more frequently, which could indicate that the initial viewport of the map is too much zoomed out. Another interesting point is that the double tap zoom in is used less frequently than the zoom in button on the top menu bar. This could mean that this function is not very visible to the users or that they have trouble to access it.

Figure 26 – Application Actions



As already indicated by other statistics, the bookmark related functionality is used least often. Con-sequently, one might consider the complete removal of this function or consider a redesign that makes it more visible and provides better explanations for its benefits.

3.2.9. Badge Usage

Another interesting piece of data gathered by the instrumentation shows how the end users are us-ing their parking card for disabled persons. As indicated by Figure 27, there have been tests with dis-abled badges in all pilot sites. However, the number of usages in Madrid clearly exceeds the usage in other cities.

Figure 27 – Badge Usage (Android and iOS)

As shown in Figure 27, Spanish and Portuguese users are more likely to use QR codes in order to scan their parking cards, whereas Italian and English users seem to prefer NFC. However, this result is par-tially skewed by the fact that the numbers include iOS devices which cannot scan NFC codes due to limitations of the underlying hardware. When looking at the numbers gathered from Android-based devices that are shown in Figure 28, the difference between NFC and QR codes for Spanish and Por-tuguese users becomes smaller. However, given that there are still a considerable number of users that are performing QR code scans (rather than NFC code scans), providing both options seems to be required.



Figure 28 – Badge Usage (Android only)

3.2.10. Popup Questions

In addition to implicit data capturing, we used a simple form of experience sampling in an attempt to determine the satisfaction of our users. For this, we implemented a 5-star rating dialog which we used to pose a small number of questions.

Four general questions about the application could show up when the user moved back to the home menu. These questions were asked in a specific order based on their logical relation and using timeouts we ensured that a user was asked only one question per week. A navigation and a badge re-lated question was shown when the user finished the step-by-step navigation or the validation pro-cess, respectively. In order to avoid annoying the user, the latter two questions were limited to every 3rd attempt.

Independent from the type of question, we allowed the user to quickly cancel the dialog to minimize negative feedback from users annoyed by questions. For each of the questions, the return rate ranged between 30-35%. Meaning that the number of questions that were deliberately canceled by the users was about twice as high as the number of submitted responses depicted in the figures be-low.



Figure 29 - Application Questions

The first question that we asked (Function) the users was related to the proper functioning of the application. Here was asked “Please rate your experience with the application. Do you think the ap-plication works reliably?” and specified as extremes “No, it’s not working well” and “Yes, it works as it should”. Here the response was mostly positive with 96 answers at or above 3 stars and only 20 answers below. This indicates that overall most users found the application to be working reliable.

The second question that we asked (Interface) was about the suitability of the user interface. Here we asked “Please rate your experience with Simon Mobile. Is the user interface of the application easy to use?” with the extremes specified as “No, very complicated” and “Yes, very easy”. The over-whelming majority of responses with respect to this question is neutral (i.e. 3 stars) or positive (i.e. more than 3 stars). However, there are 10 users which report a rather negative experience.

The third question that we asked (Recommendation) was asking the user to consider the application for a recommendation. Here we asked “Please rate your experience with the application. Would you recommend the application to other persons?” With the extremes specified as “No, never” and “Yes, definitively”. Here we got mostly positive responses, however, since this question was asked only af-ter 3 weeks, we believe that most users that did not like the application would have uninstalled it by then. Consequently, we think that this might be a biased response.

The forth question that we asked was on the monetary value (Value) of the application. Here we asked “Please rate the value of the application. Would you pay a small amount of money for an ap-plication like Simon Mobile?” with the extremes given as “No, never” and “Yes, a small amount”. De-spite the fact that this question was asked after the recommendation question, here, we received a fairly mixed response with 11 negative, 16 positive and 19 neutral responses. Consequently, we think that most end users would not be willing to pay for the developments in Simon Mobile directly. Thus, we think that it is necessary to consider alternative ways of monetization.



Figure 30 – Usage Questions

When looking at the responses to the questions related to step-by-step navigation and badge usage shown in Figure 30 we again received mostly neutral and positive feedback.

In order to determine the user satisfaction with the step-by-step navigation (Navigation), we asked the users “Please rate your experience during navigation. Was the application useful for you?” with the extremes defined as “Not useful” and “Very useful”. Out of the 35 responses to this question, on-ly 5 were negative and 20 were clearly positive which seems to indicate the usefulness of the step-by-step navigation.

To determine the satisfaction with the badge (Badge), we asked the users “Please rate your experi-ence with the Simon badge. Is the parking validation easy to do?” with the extremes set to “No, it’s too complicated” and “Yes, it is easy to do”. Here we received more than 100 responses with only 8 negative ones. As a result, we think that the user experience with the validation and access control process is rather positive for most users.

3.3. AUTHORITY OPERATOR TOOL- BASED METRICS FOR USERS

In this section we extract some metrics related to the different user profiles that have generated events and registered in the SIMON backoffice. All these events are related to the Validation services (SIMON SAYS and SIMON OPENS). The Authority Operator Tool allows to extract the following infor-mation for citizens (holders of the parking card), controllers (in charge of verifying the parking valida-tions) and public administrators (those in charge of managing the system).

3.3.1. Activity of citizens regarding the parking badge

Parking validations evolution

The Authority Operator Tool allows the visualization of the parking validations by month. Data have been exported to visualize the evolution of activity in the pilot sites. It is clear how the first stage in the pilot demonstrations implied some users in Madrid using the parking card for validation purpos-es. Activity also started in Lisbon in October. Nevertheless, Parma users have barely used their park-ing badge in the period represented (2016 – 2017).



Figure 31 - Parking validations evolution

Parking validations per token type (NFC / QR code)

Parking validations can be performed by users by scanning the NFC tag with their mobile or by scan-ning the QR code. Both option are analyzed in the figure below, where it can be seen that Madrid us-ers have tested both options but they tend to prefer QR code. This is line with the metrics gathered from the use of the SIMON Mobile app (see section 3.2.9). Lisbon users have also preferred the QRcode scanning as an option.

0

10

20

30

40

50

60

70

80

90

100Ja

nu

ary

Feb

ruar

y

Mar

ch

Ap

ril

May

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Jan

uar

y

Feb

ruar

y

Mar

ch

2016 2017

Parking validations evolution

Lisbon

Madrid

Parma

Reading



Figure 32 - Parking validations by token type

Parking validations by end mode

When users have to leave the parking place they have to notify this operation to the system. If this is not the case, SIMON will register this as a “delay”. A third option for finalizing a parking validation is the notification of a controller that verifies the car is not in that place any more. Those are the three metrics in Figure 33. There are some users that tend to forget they have to notify they leave. SIMON app is also able to warn the user about this fact when he/she tries to do the following validation. Thus, it is expected the driver will finally do this operation correctly, although the learning phase can be confusing at this regard.

0

10

20

30

40

50

60

70

Jan

uar

y

Feb

ruar

y

Mar

ch

Ap

ril

May

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Feb

ruar

y

Mar

ch

Ap

ril

May

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Jan

uar

y

Feb

ruar

y

Mar

ch

Jan

uar

y

Feb

ruar

y

Mar

ch

NFC/RFID QR Code NFC/RFID QR Code

2016 2017

Parking validations by token type

Lisbon Madrid Parma Reading



Figure 33 - Parking validations by end mode

3.3.2. Activity of pilot cities controllers

The figures regarding the parking controllers in the SIMON system are as follows:

Total controllers in the system

Active Controllers

Madrid 8 8

Parma 32 7

Reading 1 1

Lisbon 196 75

Total general 237 91

According to the pilot implementation plans, Lisbon has clearly as an objective to include their en-forcement officers in the system in order to test the SIMON CONTROLS functionality that has been integrated in their OPERA app. The rest of cities are demonstrating the use of the SIMON CONTROLS app involving partially some of their officers.

Activities performed by controllers have taken place quite late in the project, but they are very valu-able and must be integrated in this evaluation. These are the verifications done by parking control-lers, all of them registered during March 2017.

0

10

20

30

40

50

60

70

80

90

100

Feb

ruar

y

Feb

ruar

y

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Jan

uar

y

Feb

ruar

y

Mar

ch

Ap

ril

May

Jun

e

July

Au

gust

Sep

tem

ber

Oct

ob

er

No

vem

ber

Dec

em

ber

Jan

uar

y

Feb

ruar

y

Mar

ch

Jan

uar

y

Feb

ruar

y

Mar

ch

Controller Delayed Normal Delayed Normal

2016 2017

Parking validation by end mode

Lisbon

Madrid

Parma

Reading



Figure 34 – Number of verifications done by controllers

As shown in Figure 35, many enforcement officer were active in Lisbon and performed activities at least onec, to check SIMON CONTROLS functionalities. Only one of them dedicated some extra effort for a more exhaustive use of the app. A similar case occurs in Madrid.

Figure 35 – Distribution of verifications by controller ID

The token types used by controllers are mainly QR code in Madrid and it seems Lisbon finds more useful the direct typing of the user ID. Apparently, the NFC scanning is not used for verification pur-poses.

98

57

15

Marzo

Lisbon Madrid Parma

1 1 1 1 2 2 2 3

26

2 14

1 2 2 1 15 5

3 3 2 1 14

1 24

1 15

79

31

11

62

6 7

87

16

2

26

3

36

9

50

7

55

1

58

9

66

0

78

9

80

7

81

6

85

5

85

9

89

3

90

2

91

1

Cas

iello

MA

DSC

00

4

MA

DSC

00

6

test

e

Marzo

Activity per controller

Lisbon Madrid Parma



Figure 36 – Secure access mode used by controllers

Finally, we have obtained the areas in each city where controllers carried out their activity.

Figure 37 – Verifications in Madrid

3

34

61

47

10

2

13

NFC RFID QR code User ID

Marzo

Activity per token type

Lisbon Madrid Parma



Figure 38 – Verifications in Lisbon

Figure 39 – Verifications in Parma

Regarding the notification of infractions, this functionality is not yet very used, only one Mobility Of-ficer in Madrid reported 7 notifications. Given that the sample of users who are actually validating the parking is really slow, by now the only possible notification has been that the user did not regis-



tered a validation. Many of the parked cars that were showing a SIMON parking card model in Ma-drid did not activate it, thus the event registered by SIMON CONTROLS is “ErrorNoActiveValidation”. Since there is not an active validation in the system, GPS location is not shown in the map but in any case, the list of users is registered in the data base and the public authority could decide to contact them in order to offer any possible incentive to activate the card whenever they park. During March, 30 parking cards were detected that were not using the validation in SIMON system.

3.3.3. Activity of Public Authority

Four user IDs have been created under the profile of “Public authority”, one for each pilot city: MAD, LIS, PAR and RED. They have their own credentials and can access individually to manage the infor-mation about their users in the system.

Only Madrid, Lisbon and Parma have carried out an effective management of their users (Reading just did a small scale trial), by uploading additional credentials when a new batch of badges started to be deployed. Scalability of the system has been tested in this way, since number of badges in the system has been gradually increasing.

The number of citizens (Blue Badge holders) currently managed is as shown in the table.

Citizen Controller

Madrid 18542 8

Lisbon 700 196

Parma 779 32

3.4. PROJECT PERFORMANCE INDICATORS

As defined in D7.1 (1), project performance indicators have been selected to be as relevant as possible to measure to what extent the project is successfully achieving its objectives and the impact at each of the pilot sites. A rationale will be provided according the values assessed.

Measurable indicator 1 Method of measurement: reported from…

Target / expected impact

VALUE

Number of services adapted in open architecture and instantiated at pilots (summation of services)

WP5 (D5.3) 14 14

D5.3 Adapted existing ICT infrastructure (3) reports about the adaptation of SIMON services at each of the pilot cities. Thus, in Madrid the following services have been adapted: SIMON SAYS, SIMON BOOKS, SIMON OPENS and SIMON ANSWERS. In Lisbon, the services are: SIMON SAYS, SIMON BOOKS, and SIMON ANSWERS; In Parma, the same three services and also SIMON OPENS. Finally, in Reading SIMON SAYS, SIMON BOOKS, and SIMON ANSWERS are adapted. The addition of all of them results in 14 services, which is reaching the target.



VALUE

Minimum number of services deployed in each site (Madrid, Lisbon, Parma, Reading)

WP5 (D5.3)

WP6 (D6.1)

4, 3, 4, 3 3, 3, 4, 3



Those services are also listed in the previous indicator, “Number of services adapted in open architecture and instantiated at pilots”, also including SIMON OPENS that was adapted in Madrid in order to be later deployed for the pilot. Nevertheless, this was not possible due to some technical limitations, since the whole platform to manage the access to restricted areas in Madrid has been completely renewed in the last months. ETRA prepared some developments according to the previous system and the model to communicate with the platform managing the white lists was ready, when the whole department in charge of this system was changed and the platform was modified. There was no time enough to start the whole process again. Thus, this city is one service below target.



VALUE

Minimum number of applications deployed in each site (Madrid, Lisbon, Parma, Reading)

WP5 (D5.3)

WP6 (D6.1)

3, 3, 3, 2 3, 3, 3, 3

SIMON Mobile app, the application for citizens, is available for free in the platforms for Android and iOS, and it integrates the information for the four pilot cities. SIMON CONTROLS has also been deployed in the four cities, in Madrid, Lisbon and Parma to be used by enforcement officers and controllers and in Reading at trial level. Finally, the Authority Operator Tool is available to be used by the four pilot cities and the list of authorized SIMON users is managed through this tool in the four cities.



VALUE

Number of third bodies which whom SIMON op-erates (summation of bod-ies)

WP6 (D6.1)

WP8 (D8.2)

8 (2 each) 17

The project will have interacted with the following public bodies and companies operating in the cities:

- Municipalities: Madrid City Council – General Direction of Traffic Management and Control, for the deployment of the parking cards and the validation service, and also to involve the Mobility officers in charge of controlling parking validations. Reading Borough Council – to manage the trials, the deployment of parking sensors, the meetings with users. Besides those two, which are already part of the SIMON Consortium, both Comune di Parma and Camara Municipal de Lisboa have been interacted by the SIMON partners Infomobility s.P.a. and EMEL.

- Transport operators have been engaged in Lisbon, where initially was not foreseen to have the information of public transport and finally an agreement was achieved: Carris (bus and tram) and Metropolitano de Lisboa (metro). In Parma also the bus and the metro operators have been contacted to provide the data sources to be integrated in SIMON. In Madrid, the information about the whole network of urban buses (EMT) is offered as open data. Furthermore, EMT was also contacted to provide support in the promotion of the SIMON app –now recommended in the EMT Open Data Portal-. Metro Madrid has been interacted to deploy the indoor navigation system in one of its stations.

- Finally, some other associations and public bodies have been interacted. The most important ones are highlighted:



o In Lisbon: Secretary of State for inclusion of people with disabilities, the National Institute for Rehabilitation (INR, I.P.) and The Institute for Mobility and Transport (IMT, I.P.) (this is the entity issuing the parking cards)

o In Madrid: the associations FAMMA-Cocemfe and ONCE have actively collaborated to evaluate the system and to disseminate information about the project through their channels.

o In Parma, the University of Parma has been involved since the beginning.

o In Reading, trials have been supported by the Physical Disabilities and Sensory Needs Forum



VALUE

Coverage of accessible metro stations WP4 (D4.2)

WP5 (D5.3)

2 1

The purpose of this indicator is to measure the number of accessible metro stations in the city of Madrid that are modelled in SIMON system. Only Moncloa metro station has finally been included and all the related effort has been focused to deploy an indoor location system that fulfils the transport operator expectations, instead of modelling additional stations that would not be later on included in this deployment.



VALUE

Number of Public transport modes considered at each city (Madrid, Lisbon, Parma, Reading)

WP5 (D5.3) 4, 2, 2, 1 4, 5, 2, 0

The following public transport modes can be found for each of the cities:

- Madrid: urban buses, metro, cercanías (light train) and interurban buses.

- Lisbon: urban buses, metro, tram, funicular, ferry

- Parma: urban buses, regional buses

- Reading does not integrate public transport due to the lack of time and resources to be allocated to develop the adaptors for the UK standard data format.



VALUE

Number of interfaced external ICT infrastruc-ture (Madrid, Lisbon, Parma, Reading)

WP5 (D5.3) 2, 2, 2, 2 5, 3, 5, 4

SIMON interacts with multiple databases and existing services, building on-top of already existing ICT infrastructure. In the four pilot sites services related to mobility and public transport are interfaced.

The specific infrastructure tackled at each city is as follows: integrated mobility platform (park meters system) in Madrid, restricted access areas (ZTL system) in Parma and disabled bays monitoring system in Reading. Regarding data bases, the following ones are successfully integrated in all the pilot sites: Outdoor geometry database, Transit information database and Parking information



database. Also in Madrid: Indoor geometry database; finally, also in Parma: Access controlled zones database.



VALUE

a. User satisfaction index (controllers) WP7 >8, in a rank 0 to 10

See Section 3.5.3

b. User satisfaction index (citizens) WP7 >7, in a rank 0 to 10

3,68 in a rank 1 to 5.

See section 3.5.1

c. User satisfaction index (public authority) WP7 >8, in a rank 0 to 10

See section 0



VALUE

a. Number of SIMON users WP6 As defined in D5.1

b. Number of SIMON disable badges WP6 As defined in D5.1

SIMON users comprise all the citizens that have downloaded and installed the SIMON Mobile app, who will use both the navigation and the access-right management functionalities. Enforcement of-ficers at each of the pilot sites, as well as public administrations are also included as part of the SI-MON users. The indicator SIMON disabled badges refers exclusively to the parking disable badges.

The target in the number of SIMON users was revised in September 2016 and modified according to the Amendment #2 to the SIMON Grant Agreement. The new targets are as follows:

Madrid: 1400 users in the Madrid pilot, 3000 ICT badges issued

Lisbon: 1050 users in the Lisbon pilot, 1700 ICT badges issued

Parma: 400 users in the Parma pilot, 3300 SIMON ICT badges issued

Reading: 300 users in the Reading pilot (no badges issued)

According to the figures provided in section 3.1 for the app total installations and for the users and badges registered in the SIMON TSM, the number of SIMON users until March 2016 was as indicated in the following table:

App installations

Android/iOS*

Badges issued / active users Controllers

Madrid 383 / 43* 6000 / 56 8 / 8

Lisbon 174 / 33* 700 / 61 196 / 75

Parma 184 / 42* 779 / 22 32 / 7

Reading 74 /31* - -



* iOS figures are underestimated, since Apple provides the information only if the users has shared

the information about location. Thus, and according to the total installations, this numbers might be about a 30% higher.

The number of users is below the target proposed, even if this target was already decreased to be more realistic after the first phase of the large scale demonstrations. The problem to effectively engage users was already detected in the second reporting period, and some mitigation plans were proposed and carried out in the different pilot cities.

If we look at the deployment of badges in Madrid, they have started a large scale deployment which indicates that they are confident with the result of SIMON parking card as a tool to fight fraud and to enhance the mobility of disabled users. Nevertheless, engaging users to activate their cards is resulting a hard process. During 2016, mobility officers have carried out a campaign for controlling fraudulent use of parking cards1, which has very successful in terms of results: more than 2000 drivers were fined for this reason. In March 2017 the full operation of mobility officers started to verify SIMON validations, and they have detected that the parking cards that have the SIMON security tokens have not been validated. It is now the decision of the Madrid City Council to take the decision of regulating about this, because probably this is the only way to make the users do their part. Madrid City Council has already declared that a among the municipal proposals for these cards they can pose the question of elevating their regulation to decree, in addition to establishing a sanctioning regime that allows the City Council to fine the improper uses.

Parma has also the purpose to deploy at large scale their badges, even if they already did not registered in the system the full amount of the 3500 estimated badges, they have already been produced integrating the SIMON QR code (together with the Parma RFID). The main problems Parma is facing regarding the involvement of users is the age (a high percentaje of them are >70 year old) and they are not used to technology. Parma Municipality might address this problem by promoting technology literacy and training for their elderly.

Finally, in Lisbon they have used a complementary card which is associated to a valid parking badge. Lisbon authorities are very interested in this new model of parking badge, even thinking in exporting the model to some other areas related to mobility. Thus, this interest is reflected in the higher relation (badges issued/active users) among the three cities. The main issue for Lisbon authorities when it comes to recruit users is not the age of them but in this case, the lack of access to smartphones. It seems users in Lisbon are not very used to this kind of devices.



VALUE

Number of types of access tokens WP6 4 4

The following access tokens are used: park meters (Madrid pilot), plate recognition systems (Parma), NFC smartphones (all cities), QR code readers (smartphones of controllers in Madrid, Lisbon and Parma).



VALUE

Number of types of access challenging strate-gies

WP6 3 3

1 http://www.elmundo.es/madrid/2017/03/14/58c7da78468aeb81078b4618.html

http://www.elmundo.es/madrid/2017/03/14/58c7da78468aeb81078b4618.html



Three access challenging strategies for validating the parking card have been implemented: the first one is the most robust of them, since requires the NFC smartphone to scan the NFC tag in the parking card; the second one, for those devices that are not provided of NFC consists of scanning the QR code also with the smartphone (it is assumed all smartphones incorporate a camera); finally, if the user does not have a smart phone, the validation can be done through a smart park meter, by scanning the parking card (NFC reader of the park meter) or directly introducing the user ID. For this operation, a standard mobile phone is required because a sms will be sent with the challenge and the user will have to type a code in the park meter.



VALUE

Creation of Roadmap for Deployment WP8 M39 M39

Deliverable D8.2 Guidelines for SIMON deployment and Roadmap for the long-term SIMON deployment is submitted at the end of the project.



VALUE

Use of relevant standards WP8 As de-fined in D3.2 (4)

See below

Number of standards bodies and public legis-lators to which results are sent

WP8 5 5

SIMON project uses extensively adopted standards when possible:

in the pilots involving the use of NFC (Madrid, Lisbon, Reading) and RFID (Parma)

for some data sets in the adopted information model: OpenStreetMaps (OSM) for outdoor geometry and GTFS for transit information.

In Madrid, the local regulation has been modified in 2016 in order to allow the use of tech-nology in the parking cards for people with reduced mobility (PRM). This modification has al-lowed the large scale deployments of the new model of the parking card.

SIMON project has as one of the main outcomes the proposal of a specification of the new model for the EU IT-parking card for disabled people, which is delivered as D8.3.

This outcome is presented to the public authorities of the four pilot sites: Furthermore, both Lisbon and Reading does not have a local regulation for the parking card but depend on a national entity for issuing the badges instead. Lisbon has the intention to prepare its own recommendation –based on the specification for the SIMON IT card- to be proposed to the national body. Reading will explore this possibility too.



VALUE

Reduce economic negative impact of abuse WP7 15% See below

The negative impact of abuse in the use of parking badge is a key economic indicator. Economic costs associated to the non-detection of the infraction can be estimated depending on themobility policies applied by the public authority in the city.



In Madrid, as in the rest of cities, specific conditions apply for parking badges of disabled users. In particular, for owners of a Disabled Card, current regulation also indicates that they are allowed to: a) Parking on the specifically marked places for that purpose. b) Parking for unlimited time in reserved zones for loading and unloading. c) Parking for free and unlimited time in controlled parking zones. d) Parking in places where parking is prohibited by signs, adjusting in all cases to the indications of Mobility Officers and Municipal Police. For a big city as Madrid, where also mobility policies are focusing sustainability and reduction of emissions, it is important to promote and encourage the use of public transport and on the other hand, try to dissuade drivers of getting the car to move in certain areas, that is why on street parking is regulated in an extended area and mainly in the city center. The same approach is followed by most of European cities, this also including the rest of SIMON pilot cities. In this analysis it is important to describe the situations in which SIMNON system is effective to fight fraud in the use of the EU parking card.

Fraud situation 1: a user has duplicated / produced a copy of his/her parking badge so it can be used in additional car by some other person when parking (varius copies could be pro-duced). Both the original card and the copy (Ies) will be placed simultaneously in several cars when parking.

SIMON solution: the use of the ICT parking card together with the validation functionality of SIMON Mobile app will register as “validated” the User ID associated to the parking card. In the system, only a valid authentication is allowed so in case that a forged card would be cir-culating, two simultaneous validations would not be possible, in consequence this situation can be certainly avoided.

Fraud situation 2: a non-authorized user is using a legal badge. Two possibilities can occur at this point:

o The non-authorized user has installed the SIMON Mobile app on his/her smartphone and knows both the user ID and the password to associate the parking card to the smartphone. This situation is possible, since SIMON has been thought to offer the possibility of being installed in several devices in order to allow relatives or carers to drive a disabled user who is entitled to use the parking card. In this case, a non-authorized user will be validating the parking operation and this will be registered in the system. Only a parking controller or enforcement officer would be able to detect the fraud situation if the identification of the user is required.

o The non-authorized user is not able to validate the parking card when parking be-cause he has not access to the validation functionality of SIMON Mobile app. This us-er would try that the holder of the parking badge does the validation remotely which is not possible because parking card and smartphone have to be located together in the same place to produce the validation. In the alternative case that a copy of the card was available to produce a remote validation, the GPS position allocated to the validation does not correspond to a parking place and it is not valid, in this sense the user validation is not accepted.

Fraud situation 3: the parking badge is stolen to be re-sold to non-authorized users or to be forged to produce additional cards. This situation is quite common in the UK, where parked cars are damaged and windows are broken just to get the parking badge. Pilot cities have de-scribed actual situations of black market around this issue, because given the problems that people have to park in a city center and the associated costs of being on a regulated parking infrastructure, as previously described, many people is willing to pay a lot of money just to have a copy of a parking card.



SIMON solution: the fact of having an IT card which is registered in a system allows the public authority to cancel the parking card in the system as soon as the incidence is notified, what means that the User ID associated to it is not valid any more. This makes the parking card “unusable”.

As a result, we can estimate that situations of misuse of the parking card (TPMR card) that can be avoided with SIMON system suppose more than 50% of the previous fraud cases. This has a conse-quence in the increase of money due to more parking operations in the regulated parking area in-stead of using reserved places illegally. In the context of WP8, a questionnaire was filled by the cities to indicate some figures to estimate fraud. Some interesting inputs are included in D8.2 (5). Madrid situation has been described as fol-lows: The income from parking in the regulated area was expected to be 96 million in 2016. Having 150.000 parking places in this area, income is of 640€/place by year. Estimations are that SIMON con-trols of fraud situations could mean an income between 450.000€ and 1.150.000€ a year due to peo-ple that should have to park in the regulated area instead of making use of reserved places for PMR.



VALUE

Reduce operational costs for parking manag-ers and public authorities

WP7 10% See below

In the framework of WP8 it is described how the deployment of the new model of IT-enhanced park-ing card does not suppose a significant variation of the costs to prepare and register the new models in the platforms of the public authority in charge of managing the schema. During the project dura-tion both Madrid and Lisbon were in a process to develop new platforms to support mobility ser-vices, and this is an ongoing process because both of them are involved in projects addressing mobili-ty. The adaptation of SIMON services supposes the possibility not yet existing of having real time information about the use that parking badge holders do of the parking resources in the city, which is something new in this context. Thus, it is not a question on reducing operational costs –because this is not easy to be quantified- but getting access to additional information after deploying SIMON sys-tem, which can save time of traffic controllers and result in a better allocation of the resources.



VALUE

Number of 1:1 / focus group interviews with app users to record their experiences

WP5, WP7 6 per site See below

Focus groups took place in the framework of WP5 in the second reporting period, when a lot of activ-ities were carried out and reported in D5.2 and WP9 deliverables. The methodology in WP7 to evalu-ate user experience was adapted to the actual possibilities and needs of the users involved in the pi-lot activities. It was established by the SIMON partners that it is not very easy to achieve that disabled users attend meetings and move to participate in face to face meetings several times. Since the first informative meeting that took place in Madrid, most of those attendants that shown an in-terest and a certain willingness to participate in the pilot activities, did not participate again in fur-ther meetings to know more about methodologies or explanation of activities. Furthermore, associa-tions contacted to organize face to face meetings always claimed that most of the people initially confirmed for attending, finally cancelled the participation. Thus, the initial methodology was modi-



fied on the fly and more relevance was done to online questionnaires and direct feedback through e-mail. From the latest, some relevant comments were received from one person working as a techni-cian in ONCE (Spanish association for blind users) and another users who is an expert in technology and accessibility, both of them visually impaired, who made a very interesting assessment about the app, the available functionality and the accessibility point of view. Their feedback is also included in the nest section.



VALUE

Number of questionnaires to users with email addresses on SIMON system

WP7 30 per site

A detailed analysis of this indicator is provided in the next section. The final figures were: Madrid got 70 questionnaires, Lisbon got 7, Parma got 17 and finally Reading got 7.



VALUE

Number of 1:1 interviews with the Parking En-forcement team (on street) to record their ex-perience of using SIMON CONTROLS

WP7 5 per site (except Reading)

Controllers from Madrid, Lisbon and Parma were asked to provide their feedback. The activity of mobility officers and enforcement officers has been more relevant in Madrid and Lisbon in terms of activity on the street, while controllers in Parma did a lot of training and trialing which was used to refine the first versions of the SIMON CONTROLS app. Their reports are included in Section 4.3.



VALUE

Number of 1:1 interviews with the Parking En-forcement manager to record the potential of SIMON SAYS to effectively reduce fraud

WP7 1 per site 3

Public authorities in the Project have provided their feedback about the Authority Operator Tool, fo-cusing on the management of users’ badges and on the set of dashboards proposed. EMEL from Lis-bon and Infomobility from Parma provided their feedback, which is included in the next section.

Number of user associations in the User Group.

The final list is of SIMON Users Group is as follows:

Name User Group

AGE Europe

EPA Europe

POLIS Europe

AISLA Italy

ANFFAS Italy

ANGLAT Italy



ANMIC Italy

ANMIL (National Association of naimed and injured workers ) Italy

Francesca Fogar Italy

La Mano di Scorta Italy

Norberto D'Angelis Italy

Polisportiva Gioco Italy

Struttura Operativa Disabili Italy

Unione Italiana Ciechi Italy

Unione Nazionale Mutilati e Invalidi Serivizio Italy

Universita Degli Study di Parma Italy

ACAPO - Associação dos Cegos e Amblíopes de Portugal Portugal

ADFA – Associação dos Deficientes das Forças Armadas Portugal

ANDAR – Associação Nacional dos Doentes com Artrite Reumatóide Portugal

ANEA – Associação Nacional da Espondilite Anquilosante Portugal

APD – Associação Portuguesa de Deficientes – SEDE Portugal

APELA - Associação Portuguesa de Esclerose Lateral Amiotrófica Portugal

APPACDM Lisboa - Associação Portuguesa De Pais E Amigos Do Cidadão Deficiente Mental

Portugal

ASBIHP – Associação Spina Bifida e Hidrocefalia de Portugal Portugal

Associação Salvador Portugal

Camara Municipal Lisboa Portugal

FAPPC - Federação das Associações Portuguesas de Paralisia Cerebral Portugal

Fundação LIGA Portugal

IPCVI – Instituto Português para a Cidadania e Vida Independente Portugal

SPEM - Sociedade Portuguesa de Esclerose Múltipla Portugal

CERMI Comunidad de Madrid Spain

Union Democrática de Pensionistas y Jubilados de España Spain

FAMMA Cocemfe Madrid Spain

Fundacion ONCE Spain

CEAPAT IMSERSO Spain

ONCE Spain

UK

3.5. USER EXPERIENCE INDICATORS

3.5.1. Online questionnaire

An online questionnaire was prepared to be filled on by users of the SIMON system, as explained in D6.1 (2) and D7.1 (1). Based on the on-line questionnaires received the following results are provided:



Question 1:

Please, indicate all the options that suit your profile:

Answer Options Response Percent

EN IT SP PT TOTAL

Physical disability: upper limbs 12,2% 1 1 6 2 10

Physical disability: lower limbs 61,0% 4 8 33 5 50

Cognitive disability 1,2% 0 0 1 0 1

Hearing disability 7,3% 1 0 5 0 6

Visual disability 4,9% 0 0 4 0 4

Elderly person 3,7% 0 0 2 1 3

Relative 9,8% 1 3 3 1 8

answered question 7 12 54 9 82

skipped question

Table 2 – Title of Table

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

Physicaldisability:

upperlimbs

Physicaldisability:

lowerlimbs

Cognitivedisability

Hearingdisability

Visualdisability

Elderlyperson

Relative

Please, indicate all the options that suit your profile:



Question 2:

Please, indicate all the options that suit your mobility situation:

Answer Options Response Percent

EN IT SP PT TOTAL

Driving own vehicle 46,5% 4 6 32 5 47

Passenger in relative’s vehicle 13,9% 1 3 10 0 14

Public transport 21,8% 2 3 16 1 22

On foot/wheelchair 17,8% 0 5 12 1 18

answered question 7 17 70 7 101

skipped question

0,0%

5,0%

10,0%

15,0%

20,0%

25,0%

30,0%

35,0%

40,0%

45,0%

50,0%

Driving own vehicle Passenger in relative’s vehicle

Public transport On foot/wheelchair

Please, indicate all the options that suit your mobility situation:



Question 3:

Indicate the degree of agreement with the following statements:

Answer Options Rating Average EN

Rating Average IT

Rating Average SP

Rating Average PT

Rating Average

TOTAL

The app perform the tasks successfully 4,00 3,75 3,26 4,00 3,75

This app (SIMON Mobile) is useful. It provides the information expected 4,00 3,71 3,50 4,33 3,89

The app is easy to use. It is simple to carry out tasks 4,20 3,63 3,74 4,33 3,98

Navigation: search for a place on the map is useful 4,20 3,88 3,50 4,00 3,90

Navigation: plan a route is useful 4,20 3,63 3,75 4,00 3,89

Consult the transport timetables is useful 3,00 4,14 3,78 3,00 3,48

The validation process in parking areas is useful 2,80 3,63 3,35 4,33 3,53

The process of accessing restricted areas is useful 2,60 4,00 3,39 3,33

The additional features (schedule, favourites, historical, help) are useful 3,00 3,75 3,53 4,00 3,57

App settings are useful 4,00 3,86 3,67 3,67 3,80

I like the aesthetics of SIMON MOBILE 4,00 4,00 3,84 4,00 3,96

I felt satisfied using SIMON Mobile (improves my mobility) 4,00 3,88 3,42 4,33 3,91

I'd pay to use the services provided by this app 3,00 2,71 2,47 3,67 2,96

TOTAL ANSWERS 5 11 61 7 84



1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00

The app perform the tasks successfully

This app (SIMON Mobile) is useful. It provides the information expected

The app is easy to use. It is simple to carry out tasks

Navigation: search for a place on the map is useful

Navigation: plan a route is useful

Consult the transport timetables is useful

The validation process in parking areas is useful

The process of accessing restricted areas is useful

The additional features (schedule, favourites, historical, help) are useful

App settings are useful

I like the aesthetics of SIMON MOBILE

I felt satisfied using SIMON Mobile (improves my mobility)

I'd pay to use the services provided by this app

Rating Average

1 Totally disagree

2 Disagree

3 Neutral

4 Agree

5 Totally agree



Question 4.

Country OPEN RESPONSES CODED EN

SP It would be necessary for the free and occupied squares to appear different. Sometimes validation does not work. The aesthetics of the maps is not very pretty (we are accustomed to Google Maps) The rest of the items very well.

SP All functions are useful, especially the planning of accessible routes. Accessibility for people with low vision is good, although some instructions are a bit confusing.

SP (NOT RELATED WITH APP) Incorporation of reserved parking areas at some stops. Direct red button with emergency, in case someone is indisposed and needs help. (NOT RELATED WITH APP) More reserved parking spaces in Madrid and that the plates are bigger so that they perceive better.

SP Some messages in green are not visible on the phone with daylight. Most of the time I try to validate the parking space tells me that you cannot connect to the system. I have problems when setting the time of arrival and departure of the parking space.

SP It would be necessary for the free and occupied squares to appear different.

SP If everything worked well it would be great

SP Very interesting, some functions do not work yet quite well, but it is a good idea

SP Some items (maps) are not accessible for people with severe visual impairment

SP Some things do not work well sometimes

SP Maps with better aesthetics. Easier to look for streets. Greater accuracy in validation

SP Services not available outside Madrid

SP I have not been able to check everything, but the design and what I have seen seems useful.

SP Other apps plan better routes with better aesthetics

SP It is OK that there are several features (routes, validation, ...) but the most useful is the validation, since the rest can be done with google maps

SP Frequent failures at the validation process

SP It would be necessary for the free and occupied squares to appear different. Validtaion it's slow

SP The validation through NFC directly has not worked for me at any time, and through the QR code I have done it 3 times out of about 10 at-tempts. It may be that the application has problems or it may be that, because of my involvement of the upper limbs (I am tetraplex and I do not move my arms and hands excessively) I am not able to do it correctly. It would be interesting for people in my same situation, to be able to



Country OPEN RESPONSES CODED EN

validate by entering a code via keyboard. Faster and more effective than manipulating the mobile in uncomfortable positions for people with my level of injury.

SP For people with tetraplegia it is more difficult to handle, it should be easier

SP I prefer to validate using my usual card. It is messy and takes time to use the app.

SP The option of parking on the street outside the reserved parking spaces should be more accurately geolocated

SP It should be possible to receive information about the status of the parking spaces

SP Try to incorporate in real time the state of the mobile barriers, such as escalators, elevators, etc.

SP Database of disabled parking spaces. That works in underground parking

PT (NOT RELATED WITH APP) Do you think everyone has "smartphones" or do you intend to provide them?

PT It's perfect this way.

IT Through the app simon I cannot acquire the qr code instead I acquire with a normal app for reading barcodes

IT Greater accuracy would be required at the route indications Ability to download or transfer the APP also all at once the mobile devices (smartphone and tablet simultaneously for example)

EN Validation, access to restricted areas and public transport information it's not available in Reading. The other features work correctly. Great app when everything it's ready.

EN Validation it's not yet available and it would be very useful



3.5.2. Other feedback from citizens

Besides online questionnaires, some emails were received to inform about incidences and to comment about possible improvements of the system. During the first stage of the pilot activities, form February 2016 to June 2016, some dedicated users were using a not yet totally mature version of the Android app, thus some bugs and inconsistencies were detected and several new releases of the SIMON Mobile app took place.

Madrid

There were 8 users in Madrid reporting in this way, and due to privacy issues we cannot know if they are the same that later on were filling the online questionnaire. As the main topics of interest, the validation of the parking card and the correct location of the reserved place was very commented, as well as the possibility to know occupancy of the parking places as a proposal, which was a functionality to be later on integrated but was not available at this stage.

Sometimes, users misunderstood what they could do with SIMON Mobile app or not. One user was continuously reporting that validation did not work, or took a lot of time to read the QR code, and the app seemed to be blocked. After some calls, we realized she was trying to validate a parking place in a garage (indoor), which means she did not have an internet connection. Thus the app was not working.

Lisbon EMEL received feedback from users by different channels:

Phone: Direct contact or through EMEL complaints’ channel

Email: Direct contact or through [email protected]

SIMON website

Regarding suggestions to improve SIMON app, feedback was received by mail from one member of the user group and one person representing the Portuguese Association of Disabled People (APD). We received positive feedback regarding the new plastic bag in order to avoid fraud. Some users con-tacted with doubts in the registration process of their new plastic bag. In particular, one user men-tioned that do not have a smartphone. One particular user send a negative feedback about the pro-ject through the website and, after we explained why the project SIMON is important to the city, he decided to join the Lisbon pilot. Some relatives sent emails with doubts regarding the use of the plas-tic bag. They do not know if they can use the card in their own cars when transporting their relatives. Other users asked specific questions, for instance if the reserved parking spots were georeferenced in SIMON app, since they want to park in those spots. However, this is not legal and EMEL needed to inform them. Other users reported more complex problems, regarding how bad designed are the parking spots. Those problems were forward to the Municipality (Filipe Albuquerque). In general, the feedback was very positive, and most people are happy to know that EMEL is trying to solve the problem of fraud and illegal parking. Finally, EMEL also received several phone calls of people who do not have a smartphone (mainly old-er people) asking how to do the registration of the card.

Parma

Features about access to restricted urban areas (ZTLs) have not been extensively used by the users because the regulation of Parma to access the gates and therefore to the LTZ, is already wide and flexible by making it possible for the disabled to use two number plates associated with a single permit or going through the electronic gates communicating the use of a third car before the transit. The communication of this data takes place via fax, e-mail or telephone. Now with SIMON APP the access to the ZTL is easier because the users simply can enter the number plate by using the SIMON

mailto:[email protected]



APP, but this is not very common in daily movements. Thus, the user does not need to use SIMON OPENS function to get around town. Even the function that displays a map of the parking lots, SIMON ANSWERS, is not widely used by customers residing in Parma because they already know the location of the stalls, being Parma an average small city (200,000 inhabitants in the city). This kind of services are more recommended for some other users visiting the city.

Reading

Reading started pilot activities in the project some time later. Since they did not participated in a previous phase of small piloting, as the rest of cities did, they have provided now a more exhaustive reporting about their activities and users feedback.

SIMON App Testers Feedback Sept 16 - March 17

Background

The SIMON project was introduced to the Reading local community of the elderly and people with disabilities at a public meeting at the Civic Offices on 15 January 2016. Invitation letters and emails were sent to over 50 organisations representing people with disabilities and over 200 individuals who hold a Blue Badge. Around 20 people representing a variety of disabilities attended and everyone volunteered to help with the testing and giving feedback. The meeting was very positive. Many people said it was very good to have such a hopeful project that should make their journeys and parking in Reading easier.

Further interest from the community came as a result of BBC TV and radio interviews asking for volunteers to test the App in November 2017 and news in Disabled Motoring UK Magazine in December 20162. This resulted in many email and telephone requests for information about how to download and use the App, followed by around 500 downloads and around a further 40 people volunteering to give feedback. This gave a total of 60 volunteer testers.

Collecting feedback

The 20 volunteers from the meeting plus 40 others who had given email addresses were sent a questionnaire on 17 February 2017 to fill in and email back. Many people who were sent the questionnaire preferred to simply email their findings and comments back rather than use the questionnaire. These comments were included in this report. People who contacted the Council and gave comments by letter and phone rather than using the questionnaire were also included in the findings.

Summarising feedback

A total of 39 responses were received by email, phone and letter. A response rate of 65 per cent.

Of those who responded, 76 per cent had used the App for parking information and 25 per cent had used the navigational information. The parking information was regarded the most useful whilst the navigation information was not required by 25 per cent of users because they had local knowledge and a further 10 per cent preferred to use google directions.

The suggestions for improvements covered both the App and the general system. These can be summarised as follows:

Regarding the App, there was initially some confusion about whether a password was needed and as this was only required for Reading’s small team of testers for the enforcement team, this was quickly rectified by informing users that they did not need a password to access the parking and navigation

2 https://www.disabledmotoring.org/news-and-features/news/post/240-smartphone-app-to-detect-blue-badge-spaces

https://www.disabledmotoring.org/news-and-features/news/post/240-smartphone-app-to-detect-blue-badge-spaces

https://www.disabledmotoring.org/news-and-features/news/post/240-smartphone-app-to-detect-blue-badge-spaces



Figure 40 - Briefing session with Team leaders of EMEL’s Enforcement Officers

information. A small number of these users said that they would like a user manual in preference to the videos on the website. A larger 12 per cent were concerned that GPS was draining the battery.

A small number of users commented that prior knowledge of the area was needed so you knew where to expand the map when looking for parking spaces and that that could be a problem for visitors new to the town. Some users were disappointed that they could not get an update on parking availability as they approached the town centre and felt it an inconvenience to have to pull over and stop to use the App again. One tester commented that a voice telling you if your chosen space was taken while you were driving towards it would be the ideal solution.

By far the most frequent comment cited by nearly a third of testers was regarding the system not providing the correct parking information because in Reading there had been an issue on installation that the spaces vacant or occupied were not shown accurately on the App. This was due to cars not parking over a sensor but avoiding the sensors in unmarked bays. These parking bays have now been white-lined to demarcate where cars should park and the information is now being received accurately on the App.

In conclusion, the parking information was regarded as the most useful by Reading users and this was hugely welcomed by almost every respondent. A small number also mentioned that they were glad to be able to reduce the time previously spent cruising around looking for a parking space.

There was widespread enthusiasm and praise for the project (79.5%). 28% of respondents found the App very helpful in the information it conveyed and around a quarter of respondents found the App easy to use. Indeed there were no comments on whether the App could be improved so the handling of the App suited their disability.

Future applications

A further wider benefit to Reading Borough Council is that we now have a bank of volunteers we could approach for further feedback and comments on mobility issues.

Also, given the very good response to the SIMON App, Reading hopes to use the potential to provide information on the availability of all public parking spaces and freight loading bays should funding become available in the future.

3.5.3. Feedback from Parking Controllers

EMEL enforcement officers use OPERA system through a smartphone in their current routines. In or-der to use SIMON Controls, an integration of SIMON controls in EMEL’s system was made. All en-forcement officers needed to install the apk in their smartphones. However, just 75 enforcement of-ficers have installed it. In order to clarify the main objectives of these new procedure, a clarification session with Team Leaders took place on 6th march 2017. In the session, we have explained all fea-tures of SIMON controls as well as the objectives of SIMON Project.

-



The integration in the OPERA app (good? Bad?)

The integration in the Opera system is good. The enforcement officers just need to click in the right icon and it directly opens SIMON Control. The difficult part of the process was to install the SIMON Controls apk in the smartphone of each enforcement officers in order to have the integration with Opera in function. In order to clarify some doubts, the managing team of SIMON decided to organize a clarification session with team leaders in the operational center (see figure 1). Besides the clarifica-tion session, the instruction on how to install apk was sent by email to team leaders.

- Usefulness of the app

The app is very useful since it helps the enforcement officers to recognize a legal card with just a few clicks. However, it does not solve the problem, since we are experiencing problems with the recogni-tion of correct cards. All checks that enforcement officers have done until now are ‘Not valid’.

- Usability (easy? Complicated? Might be improved?)

SIMON Controls app is very simple and easy to use. However, it is difficult to read NFC technology through vehicle and because the card is in a distance which smartphones are not able to read. More-over, the reflection of sun in the front glass of the cars interfere with QR code reader.

- Operation

After the clarification session, few team leaders reported that all checks resulted in ‘NOT valid’ cards. There are two possible reasons for that. The first one is because a lot of users do not have a smartphone. As a consequence, users were not able to register their plastic badge. This may interfere with the correct way of using SIMON controls. Other reason may be problems with the server.

The integration in the Opera system was not as easy as expected. Enforcement officers use a smartphone in their current routines, which is a LG L90. This smartphone model has limited memory and when asked to install the SIMON Controls apk, a lot of enforcement officers reported problems. Since enforcement officers has strict working times and they do not have internet connection on the street, they needed to go to their operational center to solve this problem. This process was time consuming and difficult to manage with 197 enforcement officers and just 8 team leaders.

Other issue with the capacity of smartphones is related with the camera, which is not able to read QR codes if the sunlight reflects in the front glass of vehicles. When enforcement officers have problems with the reader, they consume more time doing their job and besides that they call attention of peo-ple when they are near a car for such long time.

3.5.4. Feedback from Public Authority

Parma

When asked about the Authority Operator Tool, thes are te main comments provided by Parma:

Initially the full list of users was provided, Infomobility suggested to add a filter on status to be able to get this three options: all users, only valid users, only expired users. Also to add a filter on registration (from date – to date). Also, once filtered, to show the number of results.

Dashboards: Parma was missing to add a dashboard about controllers, maybe like the dash-board named “Active users”, to monitor controllers’ utilization; for example it could be based on number of check operations. This was a really good proposal that was implemented in short time, being currently available.

Overall, Parma not only contributed actively with their feedback, but also they use the tool in daily operations to verify the deployment of their badges.



Lisbon

EMEL Feedback about SIMON Authority Operator Tool

Dashboards 1) Active Users

- As a fiscal entity it is important for EMEL to monitor finest data – along the day, per day or

per week. As much detailed data we get better.

- It will be nice to have the average number of validations per user

2) Parking validations per token type

- The type chart is not the best. It would be preferable to have a chart that shows an evolu-

tion over time (trend lines)?

- Here again, it will be nice to have data along the day, per day or per week

3) Parking validations by end mode

- The type chart is not the best. It would be preferable to have a chart that shows an evolu-

tion over time (trend lines)?


- Is it possible to have the average duration of parking?

4) Parking validations per zone

- Here it will be nice to have boroughs zones. Lisbon has provided information about fre-

guesias

- Proposal: to see here also a map with the distribution of validations?




4. Indicators vs Use Cases

The list of impact indicators is used to assess how the different use cases are demonstrated at each of the pilot sites. For each of the combined scenarios proposed in D7.1, there are some collected indicators which are useful to understand to what extent the SIMON use cases have been demonstrated.

Name Choose a name for the demonstration

Scenario Scenario (as described in section 2) to which the demonstration can be associat-ed

Description Description of the demonstration: profile of the user(s) involved, actions per-formed, app used, pilot site

Indicator(s) Indicators from “system performance”

Measurements Indicate parameters, events, actions … measured during the demonstration

Fulfillment Complete / Partially / None

Discussion Comments about the demonstration carried out

Table 3 - Table template to register demonstration results

Name Navigating the city

Scenario “Elena and Agustín”

Description Mobility impaired users want to create routes for moving thought the city , mainly walking but not very fast, also using public transport and finally using car when needed, for example to go the hospital. The possibility of adapt their walking speed to the different transit modes results paramount.

Navigation services are thus demonstrated through the use if SIMON Mobile app. This app is available in the four pilot cities and all of them have integrated basic data for pedestrian routes. The profile of the user can be configured in the app so the speed can be determined as “slow”.

Madrid, Lisbon and Parma integrate information about public transport, incor-porating some elements to inform about accessibility (ie elevators)

Need to access to video tutorials for learning the use of the system

Validation services for parking places using the SIMON Mobile app is imple-mented in the four pilot sites

Information services to show the occupancy of the parking places are available in Madrid, Parma and Lisbon directly related with the validations done by badge holders.

Profile of users: not really used to technologies, but willing to learn

Indicator(s) Indicators obtained from the application-based metrics (see section 3.2) describe how the SIMON Mobile app has been used at this regard

Indicators related to the use of video tutorials to learn the use of the navigation functionalities.

Validations using a parking card

Measurements In SIMON Mobile app the route screen is one of the most selected of the availa-



ble functionalities. After the buttons zoom-in and zoom-out, the most used function in the app is “Start_navigation”, which suggests that SIMON Mobile is widely used for this reason. Video-tutorials about navigation are one of the most visualized in the youtube channel.

Validation of parking badges have taken place mainly in Madrid and in Lisbon.

Fulfillment Complete

Discussion Being that SIMON Mobile app is freely available in the main platforms (Android and Apple) and the fact that the user has not to register or use credentials to use the navigation functionalities, the app may result attractive for users at least to try. Usage duration measured suggests that about the 50% of users who have downloaded the app remain the installation active for a long time. This is a consequence of the degree of maturity of the app, which has been permanently refined and maintained also using the feedback of active users.

Name Student on the move

Scenario “Cristian”

Description Mobility impaired user very active want to create routes for moving thought the city by car.

Navigation services are thus demonstrated through the use if SIMON Mobile app. This app is available in the four pilot cities.


Validation services for parking places using the SIMON Mobile app is imple-mented in the four pilot sites.

Information services to show the occupancy of the parking places are available in Madrid, Parma and Lisbon directly related with the validations done by badge holders. Consultation of historic of the badge is available for registered users.

Notification services to inform about offence: not integrated

Search for private parking spaces: not integrated

Controller’s functionalities are available in Madrid, Lisbon and Parma.

Profile of users: very used to technologies


Validations using a parking card

Controllers operations: they are able to verify the parking validations. No infractions have been notified.

Measurements In SIMON Mobile app the route screen is one of the most selected of the availa-ble functionalities. After the buttons zoom-in and zoom-out, the most used function in the app is “Start_navigation”, which suggests that SIMON Mobile is widely used for this reason.

Validation of parking badges have taken place mainly in Madrid and in Lisbon. Indicators about validations show that the system is performing correctly.

Parking controllers operations are also demonstrated mainly in Madrid and Lis-



bon.

Two of the related use cases have not been demonstrated since implementa-tions were not done:

To send feedback to the system indicating that the parking space was occupied by an unauthorized car (CI_20).

To search for private parking spaces (CI_14) and reserve in one private parking facility.

Fulfillment Partially

Discussion Demonstrations with parking controllers are important in Lisbon, where they have integrated the SIMON app functionality with their own app. Also in Ma-drid, where large scale deployment started in June 2016. For cross-evaluation purposes, we can see that in both cases NFC is not very much used and the oth-er options – QR code and USER ID input- seem to be feasible.

Name Ringo and me

Scenario Ana

Description Blind user very active want to create routes for moving thought the city by pub-lic transport.



Need to access to video tutorials for learning the use of the system

Configuration of accessibility for the app

Profile of users: visual impaired, used to technologies


Indicators related to the use of video tutorials to learn the use of the navigation functionalities.


Video-tutorials about navigation are one of the most visualized in the youtube channel.


Discussion Accessibility is a crucial issue in the SIMON Mobile app. It was studied in deep in the small sacle phase and later on, with the official release of the app for An-droid, it was also evaluated by experts form ONCE (the Spanish Blind users As-sociations). It has been revised in deep to ensure that all texts are correctly tagged to allow the use of voice based apps.



The app for iOS was finally produced and is available at the Apple store.

Name Jogging in Retiro Park

Scenario Fernando

Description Deaf user somehow not very secure with mobility want to create routes for moving thought the city by public transport and by car.



Validation services for parking places using the SIMON Mobile app is imple-mented in the four pilot sites. Validation services for access restricted urban areas has been implemented in Parma.

Information services to show the occupancy of the parking places are available in Madrid, Parma and Lisbon directly related with the validations done by badge holders. Consultation of historic of the badge is available for registered users.

Search and booking private parking spaces: not integrated

Profile of users: nothing specific about technology


Validations using a parking card, for parking in reserved places

Validations using a parking card, for access to restricted areas


Validation of parking badges have taken place mainly in Madrid and in Lisbon. Indicators about validations show that the system is performing correctly.

Validation of parking badges to access limited zonas is successfully integrated in Parma. Indicators about validations show that the system is performing correct-ly.

To search for private parking spaces (CI_14) and reserve in one private parking facility has not been implemented..

Fulfillment Partially

Discussion

Name Going to school with SIMON

Scenario Maria and Luca



Description Relative of mobility impaired user very active want to move by car though the city and find parking places

Validation services for parking places using the SIMON Mobile app is imple-mented in the four pilot sites. The possibility of validating by using the NFC or the QR code is available in the Madrid, Lisbon and Parma, also validation in a park meter would be possible in Madrid

Indicator(s) Validations using a parking card, for parking in reserved places

Validations using a parking card, by type of security token

Measurements Validation of parking badges have taken place mainly in Madrid and in Lisbon. Indicators about validations show that the system is performing correctly. Also both security tokens (NFC, QR Code) have been demonstrated in the pilot cities.


Discussion Comments about the demonstration carried out

5. Coverage analysis

The requirements described in (7) that are affected by each of the use cases listed in (2) were identi-fied in a coverage analysis (Table xx). The degree of fulfillment of this coverage table is now assessed.



ID

CI_

00

CI_

00

B

CI_

01

CI_

02

CI_

03

CI_

04

CI_

05

CI_

06

CI_

09

CI_

11

CI_

12

CI_

12

B

CI_

13

CI_

13

B

CI_

14

CI_

15

CI_

19

CI_

20

CI_

21

CO

_00

CO

_02

CO

_05

OP

_00

OP

_01

OP

_04

OP

_07

ID

F_001 M,

L, P

M, L, P

F_001

F_002 A A A A A A A A A F_002

F_003 A A A A

F_003

F_004 - - - -

F_004

F_005 A A A A A A

F_005

F_006 A A A

F_006

F_007 M,

L, P

M, L, P

M, L, P

M, L, P

M, L, P

F_007

F_008 P P

F_008

F_009 A A A A

F_009

F_010 A A A A A A

F_010

F_011 M,

L, P

M, L, P

F_011

F_012 A A A A

F_012

F_013 A A A A

F_013

F_014 M,

L, P

M, L, P

M, L, P

F_014

F_015 A A A A A A A F_015

F_016 P

F_016

F_017 A

F_017



ID

CI_

00

CI_

00

B

CI_

01

CI_

02

CI_

03

CI_

04

CI_

05

CI_

06

CI_

09

CI_

11

CI_

12

CI_

12

B

CI_

13

CI_

13

B

CI_

14

CI_

15

CI_

19

CI_

20

CI_

21

CO

_00

CO

_02

CO

_05

OP

_00

OP

_01

OP

_04

OP

_07

ID

F_018 A

F_018

F_019 A A A A A F_019

F_020 A

F_020

F_021 A A A A A A A A A A A A A A F_021

F_022 A F_022

F_023 A A A F_023

F_024 A

F_024

F_025 A

F_025

F_026 A

F_026

M.1_001 A A A A A A

M.1_001

M.1_002 M,

L, P

M, L, P

M.1_002

M.1_003 M,

L, P

M.1_003

M.1_004 M,

L, P

M.1_004

M.1_005 A A

M.1_005

M.1_006 -

M.1_006

M.1_008 A A

M.1_008

M.1_009 A A A

M.1_009

M.1_010 A

M.1_010

M.1_011 M M M

M.1_011

M.1_012 A A A A A

M.1_012



ID

CI_

00

CI_

00

B

CI_

01

CI_

02

CI_

03

CI_

04

CI_

05

CI_

06

CI_

09

CI_

11

CI_

12

CI_

12

B

CI_

13

CI_

13

B

CI_

14

CI_

15

CI_

19

CI_

20

CI_

21

CO

_00

CO

_02

CO

_05

OP

_00

OP

_01

OP

_04

OP

_07

ID

M.1_013 P

M.1_013

M.1_014 M,

L, P

M.1_014

M.1_015 - -

M.1_015

M.1_016 A

M.1_016

M.1_017 - - - - - - - - - - - - M.1_017

M.1_018 A A

M.1_018

M.1_019 P

M.1_019

W.1_001 - - - W.1_001

W.1_002 A

W.1_002

W.1_003 A

W.1_003

W.1_004 A W.1_004

T.1_001 -

T.1_001

T.1_002 M M M

T.1_002

T.1_003 -

T.1_003

T.1_004 A

T.1_004

Table 4 – Coverage analysis



6. CONCLUSIONS

The evaluation of the SIMON system has concluded with some interesting evidences that will be useful to decide about the future of the SIMON system and the innovations proposed.

SIMON system comprises some services to enhance mobility of elderly and impaired people. Three tools are implemented to have access to those novel services: SIMON Mobile app, SIMON CONTROLS and SIMON Authority operator Tool. All of them have been demonstrated in the four pilot sites.

From the point of view of system performance, the low recruitment of users remains the main problem since the beginning of the pilot phase. Many efforts have been put to contact relevant associations and interested stakeholders that have no doubt about the need, the current situation and the expectations about disability, mobility and concretely the problem of fraud in the use of the parking card. But when it came to directly engage users to demonstrate the validation functionality, it resulted very complicated to effectively achieve a commitment. Several reflections can be made after three years of project, but probably some lessons learnt can be evident:

- It is difficult to change habits of most of the people. A small percentage will have the curiosity to test or the sufficient initiative to test something new, but overall users want to remain as they are. Some comments were received at this regard, questioning why to change the system.

- Probably citizens can be reached more successfully at a neighbourhood level, and not starting from the “umbrella” associations or federations encompassing a large number of organizations. That was thought to be the correct approach, but recruitment was much more effective when addressing small, social entities working directly with citizens. But due to institutional relations (City Councils and Associations of Disabled) it was also important to keep the contact at that level.

- Probably the only way to achieve a modification in the citizens behaviour is to regulate and start applying incentives (when possible) or sanctions in the use of an IT paring card. SIMON is proposing several options to validate the card and all of them are technologically feasible, do not imply a complicated deployment, and basically the results will be always better than the option of doing nothing.

What can be understood as a critical mass of users for demonstrating the SIMON system has been reached for the SIMON Mobile app, which has more than 1000 installations for Android and 350 for iOS, and is also increasing day by day. The app is mature enough, it is robust and no reports of crashing or malfunctioning are reported. From the point of view of the badge holders, the number of participants has been certainly lower. Nevertheless, the representatives in associations support the parking card and validate the approach decided for its implementation. The same support is provided by the municipalities in charge of the management of the parking card.

Most of the proposed use cases have been demonstrated. Only those related to the booking of parking spaces were not finally implemented, since none of the cities has the infrastructure for it. Moreover, a new and interesting use case was included for a proof of concept of indoor navigation.

Finally, accessibility has been probably what makes the difference in this project, proposing a new approach for inclusive mobility. Experts in accessibility have validated the app itself, and a proof of concept for indoor navigation has been done in Moncloa station (Madrid), being this the other big innovation –together with the ICT parking card- in this project.



7. REFERENCES AND ACRONYMS

7.1. REFERENCES

1. SIMON Consortium. D7.1 Evaluation Plan v2.0. 2016.

2. —. D6.1 Large Scale Demonstration. 2017.

3. —. D5.3 Adapted existing ICT infrastructure. 2015.

4. —. D3.2 Informatiuon Model and Interoperability. 2015.

5. —. D8.2 Guidelines for SIMON deployment and Roadmap for the long-term SIMON deployment. 2017.

6. EC. Annex II General Conditions. Brussels : v5, 2009.

7. SIMON Consortium. SIMON Annex I. Valencia : EC, 2013.

8. —. D4.2 Refinement of services and lessons learnt. 2015.

9. —. D4.1 ICT Services and Applications. 2015.

10. —. D3.1 Reference Architecture and Principles. Brussels : s.n., 2014.

11. —. D2.1 Use Case Specification Document. Brussels : s.n., 2014.

12. —. D1.1 Project handbook. Valencia, Spain : s.n., 2014.

7.2. ACRONYMS

Acronyms List

API Application Programming Interface

EC European Comission

EMT Empresa Municipal de Transportes de Madrid

EU European Union

GPS Global Positioning System

GTFS General Transit Feed Specification

ICT Information and Communication Technology

IT Information Technology

NFC Near Field Communication

OSM OpenStreetMap

PRM People with Reduced Mobility

QR Code Quick Response Code

RFID Radio-Frequency IDentification

SMS Short Message Service

d7.2 evaluation of users, results and...

Documents