Semantic and Fuzzy Modelling for Human Behaviour Recognition in Smart

Spaces. A case study on Ambient Assisted Living

1Doctoral Defense 24th April 2015, Turku, Finland

Natalia A. Díaz Rodríguez

Supervisors: Prof. Johan Lilius and Prof. Miguel Delgado Calvo-Flores. Advisor: Prof. Manuel Pegalajar Cuéllar

Turku Centre for Computer Science (TUCS), Dept. of Information Technologies, Åbo Akademi University (Finland)

Dept. of Computer Science and Artificial Intelligence, University of Granada (Spain)

2

SPAIN: 15m of elders in 2049 (1/3 of the population) (INE)

FINLAND population 65+ years: 18.14% [1]

• [1] http://www.finnbay.com/media/news/government-prepares-to-set-out-new-requirements-for-senior-caretakers/

3Ros et. Al. 2011

OBJECTIVES

Understand Smart Spaces

–Human Activity Modelling

and Recognition

Program Smart Spaces

4

Ambient Assisted Living (AAL): usage of

technology to provide assistance to people who

needs it in their daily activities, in the less

obstrusive way

Aim: Independent living, safety, support

older/disadvantaged people

Includes: methods, systems, products and

services

5

Background

Activity Recognition in Smart Spaces6

[Image: http://www.businesskorea.co.kr/sites/default/files/field/image/smart%20home.jpg + The noun project]

Human Activity Recognition

7

Handling uncertainty, vagueness

and imprecision

Broken/ missing sensors

Incomplete data, vagueness

etc.

Different ways of perfor-

ming activities

– Different object usage

Behaviour change

8

9

Tools

Methods

WHY Semantic Technologies & Ontologies?

Semantic Web: well-defined meaning

Ontology:

– In Philosophy: study of entities and their

relations

– In Artificial Intelligence: “Explicit specification of

a conceptualization” [Gruber, 93]

– Web Ontology Language (OWL)

10

11[CONON Context Ontology]

Methods: Ontologies

12

Methods: Ontologies

JULIOANA MARIA

NATALIA

Has Brother

Has Mother

13

JULIOANA MARIA

NATALIA

Has Brother

Has UncleHas Mother

Methods: Ontologies

Methods: Fuzzy Logic

WHY fuzzy (description) logics and fuzzy

ontologies?

Real life is not black & white

– Classical (Crisp) Logic: True/False

– Fuzzy Logic: [0, 1]

• e.g. blond, tall

For automatic reasoning about uncertain,

vague or imprecise knowledge

For natural language expressions

14

15

Case study on Ambient Assisted Living:

A fuzzy ontology for activity modelling and recognition

[Image: http://www.harmonizedsystems.co.uk/]

Example: Take Medication

16

Case study on Ambient Assisted

Living: A fuzzy ontology for activity modelling and

recognition

Classes, Individuals, Data Properties and Object Properties

SUBJECT PREDICATE OBJECT

User performs activity Taking medicine =

(0.3 User performs sub-activity reach Cup or Medicine Box)

(0.3 User performs sub-activity move Cup or Medicine Box)

(0.1 User performs sub-activity place Cup or Medicine Box)

(0.1 User performs sub-activity open Medicine Box)

(0.1 User performs sub-activity eat Medicine Box)

(0.1 User performs sub-activity drink Cup)

17

Hybrid activity modelling and

recognition with fuzzy ontologies

2 phased algorithm:

1. Sub-activities (data-driven phase)

2. High-level activities (knowledge-based phase)

Validation: CAD-120 dataset:

10 sub-activities, 10 activities, 10 objects, 4 users

Cornell Activity Dataset[Koppula et al. 2013]

18

Hybrid data-

driven and

knowledge-

based

activity

recognition

19

ACTIVITY prediction:

accuracy results

20

Activity recognition

- comparison with state-of-the-art

21

22Ros et. Al. 2011

Programming Smart SpacesImplementation

A visual language to

configure the Smart Space behaviour

TARGET USER: non-technical

background

AIM:

– Rapid & easy programming of applications/

services

– Improve interoperability and usability

23

PROPOSAL: Smart Space visual programming

24

Main contributions

1. A set of ontologies to model human behaviour and tackle

uncertainty and vagueness inherent to real life

2. An architecture that integrates Semantic Web and

Fuzzy Logic for interpretable activity recognition

3. A hybrid knowledge-based and data-driven algorithm for

real-time, effective and robust activity recognition (84.1%

precision)

4. Design and development of a toolbox for non-expert

users and rapid and easy programming of Smart Spaces

[4 Journals -3 on 3rd Q1-, 9 conference papers, Google Anita Borg,

Nokia and HLF scholar. Entrepreneurship award]

25

Technology transfer:

26

Future Challenges

Multiple humans sensing

Parallel/interleaved activities

Automatic ontology learning and

evolution

Lifestyle modelling (Philips Research)

Start-up

27

Thank you for your attention

Natalia A. Díaz Rodríguez

https://research.it.abo.fi/personnel/ndiaz

ndiaz@abo.fiEmbedded Systems Lab. Dept. of Information Technologies

Åbo Akademi University, Turku, Finland

TUCS (Turku Centre for Computer Science)

Dept. of Computer Science

and Artificial Intelligence

University of Granada, Spain

28

Appendix

DTW: O(mn) (m,n length of the time

series)

PAA compression size = 2

Take out food is a subsequence of

microwaving (Subsumption heuristic/filter)

29

SUB-ACTIVITY

prediction: accuracy results

30

31

Modelling activities with

fuzzy ontologies

Classes, Individuals, Data Properties and Object Properties

SUBJECT PREDICATE OBJECT

Filomena is a Person

Filomena has heart rate 60

Filomena performs sub-activity Reach glass

Filomena performs sub-activity Move medicine

Filomena performs sub-activity Pour water in glass

Filomena performs sub-activity Eat medicine

Filomena performs sub-activity Drink from glass

A crucial but challenging task in Ambient

Intelligence and AAL. Requires:

Context-awareness and heterogeneous data

sources

Training data: examples

Common-sense knowledge

Adaptation of behaviours

Alzheimer, Parkinson

32

Human Activity Recognition

33

Cornell Activity Dataset[Koppula et al. 2013]

34

Cornell Activity Dataset[Koppula et al. 2013]