a balance of the 2 nd edition stefano chessa evaal 2012 - evaluating aal systems through competitive...

A BALANCE OF THE 2ND EDITIONSTEFANO CHESSA

EvAAL 2012 - Evaluating AAL Systems through Competitive

Benchmarking

1

EvAAL 2012 - final workshop 24/09/2012

Motivations

24/09/2012EvAAL 2012 - final workshop

2

Ambient Assisted Living (AAL) seeks for technical solutions for the improvement of the quality of life of elders and disabled.

EvAAL aims at giving a pulse to research in AAL: by identifying research challenges relevant to AAL

The long term research challenge of EvAAL is the evaluation of complex platforms for AAL

Short term research challenges are the evaluation of components/services of AAL systems

by making available to researchers the datasets obtained during the competition;

by contributing to the construction of a community of people interested in AAL.

EvAAL : a brief history and current status


3

EvAAL is a result of an EU project (universAAL)It was conceived in early 2010In 2011 the first edition

Only one track on Indoor Localization and Tracking for AAL

Relatively easy to formalize as a competition

In 2012 the second edition An additional track on Activity Recognition for AAL Growing community

Already planning the 2013 edition With (hopefully) another track

The EvAAL 2012 team


4

Steering board (9 members) Italy (3), Spain (3) Germany (2) Denmark (1)

Program Committee of the Indoor Localization and Tracking track 19 members from EU, USA and Australia

The local and evaluation team

The EvAAL 2012 team


5

Program Committee of the Activity Recognition track 16 members from EU, USA and Japan

(17 members from Spain and Italy)

Local and evaluation teams

Competitors selection process


6

Indoor Localization: 9 submissions Each submission was reviewed by 3 TPC members 1 rejected as not suitable for AAL 1 admitted as guest 7 admitted as competitors

Activity Recognition: 6 submissions Each submission was reviewed by 2 TPC members 1 rejected as not suitable for AAL 1 withdrew 4 admitted as competitors

Indoor Localization TrackiLocPlus (Germany/Switzerland)

CPS Group @ Utah (USA)

CAR (Spain)

LOCOSmotion (Germany)

Grupo TAIS (Spain)

OwlPS (France)

Smart-Condo (Canada)

Lambda4 (Germany) - GUEST

7


Activity Recognition Track

SenseCam (Ireland)

AMEVA (Spain)

Chiba University (Japan)

Carnegie Mellon University (USA)

8


The Localization Benchmark


9

The benchmark consists of a set of tests, each of which contributes to assessment of the scores for the competing system. Phase 1: Area of interest test Phase 2: Locate the actor Phase 3: The disturber

Each localization system need to produce localization data with a frequency of 1 data every half a second.

The paths followed by the person were the same for each test, and it was not disclosed to competitors before the competition.

We also provide some context information Switch Cyclette

Area of Interest


10

Phase 1: Locate a person inside an Area of Interest (AoI). The AoI in a typically

AAL scenario could be inside a specific room (bathroom, bedroom), in front of a kitchen etc.

Each system was requested to identify: 6 big AoI 4 hierarchical small AoI.

The actor moved along predefined paths staying inside each AoI for at least seconds (227 steps)

Starting Point

End Poin

t

The Actor


11

Phase 2: A person that moves inside the Living Lab had to be located and tracked. Only the person to be

localized was inside the Living Lab.

The first path was 54 steps length, the second path 94.

All the paths were characterized by 3 waiting points, i.e. the actor stayed in the same position for 5 seconds.

The disturber


12

Phase 3: Was organized as the former phase, but the competing artefacts were evaluated in the presence of another person who moves inside the Living Lab. Only the actor had to be localized by

the competing artefacts There was no requirement of

localizing the disturber Only the actor has carried equipment

provided by the competitor The actor started moving at least 5

seconds before the disturber Both the actor and the disturber have

generated contextual events but the first such event was generated by the actor

When the disturber generated an event, the actor was at least 2 meters away.

The first path was 94 steps length (while the disturber path was 54), the second path 80 (while the disturber path was 94).

The reference localization system


13

The reference localization system is essential to measure the accuracy of the competing systems. Was composed by predetermined coordinates of the paths

followed by the actor during the competition The Living Lab’s floor was covered with marks (with

different color to distinguish the right and left foot) that show where the actor had to step on.

In order to facilitate the installation and removal of the paths the marks were put on a piece of wood.

The synchronization between the steps and the evaluation tool was guaranteed by a digital metronome that indicated the right cadence (one beep one step). In this way we guaranteed that the actor repeated the

same paths at the same speed for every competitor.

Evaluation criteria


14

Accuracy [weight 0.25] This is the main purpose of a localization system The score is attributed in a range of errors between 0

and 4 meters An average error above 4 meters would mean that the

system does not really know in which room the user is…

Availability [weight 0.2] …nobody wants a system that does not responds…

Evaluation criteria


15

Installation complexity [weight 0.15] Measure the invasiveness of the system installation in

the home of the final user Think whether if you would be happy to install these

systems in your home…

Interoperability with AAL systems [weight 0.15] How easy is to use your system from the point of view

of AAL applications programmers

Evaluation criteria


16

User acceptance [weight 0.25] Would you use your system in your daily life? More advanced and better engineered systems where

advantaged against research prototypes… … but we considered also your system in the

perspective of an engineering

Financing of EvAAL


17

SPONSORS

ASUS and Texas Instruments contributed with prizes for winners

GIRAFF Plus and universAAL contributed with free registrations to AAL Forum for competitors

18


a balance of the 2 nd edition stefano chessa evaal 2012 - evaluating aal systems through competitive...

Documents

final workshopevaal

final workshopthe evaal

final workshop24092012evaal

localization data

aal systemsby

evaluation teamsevaal

evaluation teamevaal

research challenges