의미 모델링

Elaborating Sensor Data using Temporal and Spatial Commonsense Reasoning

+Mining Models of Human Activities from the Web

지능 기반 시스템 응용2006. 11. 민준기

2

Agenda

B. Morgan and P. Singh, “Elaborating Sensor Data using Temporal and Spatial Commonsense Reasoning,” BSN 2006.

The Problem Space

LifeNet : A First-Person Model

The Plug Sensor Network

M. Perkowitz, et al., “Mining Models of Human Activities from the Web,” WWW 2004.

Introduction

Proposed Technique

Evaluation

Summary and Future Work

3

The Problem Space

Two distinct directions for researchHuman-out (This paper)

Telephone

Technology-in (Much sensor network research)Text messaging on cell phones

Three topicsLifeNet probabilistic human modelThe Plug sensor networkAn experimental design for evaluation of the LifeNet learning method

4

LifeNet : A First-Person Model

First-person common-sense inference modelOpenMind Common Sense, ConceptNet, The PlaceLab data, Honda’s indoor common sense data

Attempts to anticipate and predict what humans do in the world

All of the reasoning in LifeNet is based on probabilistic propositional logic

“I am washing my hair” before “my hair is clean”

5

The Plug Sensor Network

Using for both learning common sense and for recognizing and predicting human behavior

Using this sensor network to monitor how individuals interact with their physical environment

Nine sensor modalities: sound, vibration, brightness, current, wall voltage, acceleration

6

Agenda

B. Morgan and P. Singh, “Elaborating Sensor Data using Temporal and Spatial Commonsense Reasoning,” BSN 2006.

The Problem Space

LifeNet : A First-Person Model

The Plug Sensor Network

M. Perkowitz, et al., “Mining Models of Human Activities from the Web,” WWW 2004.

Introduction

Proposed Technique

Evaluation

Summary and Future Work

7

Introduction : Recognize Humans Activities

Applications include activity-based actuationDimming lights when a video is being watchedProviding directions for someone using unfamiliar facilitiesetc.

Ubiquitous, proactive, disappearing computingComputers have to understand people’s needs by observing their physical activities (and to act autonomously)The cost of developing recognition infrastructure is too high

Even small classes of activities is hard to recognize

A broadly applicable system should be general-purpose and easy to use

8

Motivation

Vision based systemsNone have reported detecting more than tens of activities in practiceThe features robustly detectable from vision are coarse

Represent the relationships between “blobs” in the image rather than specific objectsEach activity is expensive to model

Learning of the modelsThe developers define the structure of the possible models

System tunes the parameters of the model based on examples from the user

The user is expected to label the patternsThe variety of activities is quite restricted

Introduction

9

Proposed Technique

RFID (Radio Frequency Identification)Cheap: Postage-stamp sized, forty-centWireless and battery free

Activity modelingDefine an activity in terms of the probability and sequence of the objectsGenerate the models by translating textual definitions

Structured like recipes

Produced automatically by mining appropriate web sitesMining models is part of a larger activity recognition system, PROACT (Proactive Activity Toolkit)

10

Usage Model

Assumes that interesting objects in the environment contain RFID tags (tens ~ hundreds)

Making a database entry mapping the tag ID to a name

Within a few years, many household objects may be RFID-tagged before purchase, thus eliminating the overhead of tagging

Medium-range readers (Tag-detecting Gloves) andLong-range readers (Run robots, Carts, …)

PROACT uses the sequence and timing of object to deduce what activity is happening

Likelihood of various activities, details of those activities, degree of certainty, etc…

Proposed Technique

11

System Overview

PROACT provides an activity viewer for debuggingReal-time view of activities in progressThe sensor data seenChanging of belief in each activity with the data

Inference Engine converts the activity models produced by the mining engine into Dynamic Bayesian Networks

D. Patterson, L. Liao, D. Fox, H. Kautz, “Inferring High-Level Behavior from Low-Level Sensors,” Ubicomp 2003.

Proposed Technique

12

Sensors and Models

SensorsUse two different kinds of RFID readers

Long-range reader (mobile robot): map the location of objectsShort-range reader (glove): determine the objects that are

touched

ModelsEach model (activity) is composed of a sequence (step) s1 ~snEach step si has optional duration ti and object oij involved along with the probability pij

Proposed Technique

13

The Model Extractor

Builds formal models of activities using directions

Directions are written in natural language by humanHow-to (ehow.com), recipes (epicurious.com), training manuals, protocols, etc.

Syntactic structure of directions1. A title t for the activity2. A textual list r1~rm, Each step ri has:

Possibly a special keyword delimiting duration diWhat to do during the step: subset of the objects and duration

Proposed Technique

14

Converting Directions to Activity Models

Key steps1. Labeling

Set label of the mined model to title of the directions

2. Parsing stepsDuration: Gaussian with mean = d, stdev = S(d, i, l ) Object Oi and Probability P

3. Tagged object filtering

FunctionsObject

Object extraction: WordNet ontologyNoun-phrase extraction: QTag tagger

ProbabilityFixed probabilitiesGoogle conditional probabilities (GCP)

Proposed Technique

For example,[“making tea”] has 24,200 matches, and[“making tea” cup] has 7,340 matches, thenconditional probability of a cup being involved inmaking tea is 7340/2400 = 0.3

15

ExampleProposed Technique

16

Evaluation

Mined modelsehow.com: 2300 directionsffts.com: 400 recipes epicurious.com: 18,600 recipes

Three strategies to approximate comprehensive evaluationHuman activity-trace recognition

Activities of Daily Living (ADLs)

Inter-corpus consistencyMaking cookies recipes

Intra-corpus distinguish-abilityDistinguish-ability within activity domains

17

Distinguish-ability

18

Human and inter-corpus trace recognition

ADLs domainMany objects were not tagged, missed, and interleavedModels were not perfect

Cookie domainThe identical recipe can have quite different structureFor some of the recipes, there is no counterpart in the other corpus

Evaluation

19

Impact of techniques on accuracy

ADLsDomain is fairly sparse, with many activities involving only few object

Cookie domainEach activity model involves many more objects

Evaluation

20

Impact of techniques on compactnessEvaluation

21

Summary and Future Work

An introduction to the idea of mining activity detection from the web

Future workPerform a more comprehensive evaluationImproving the effectiveness of mined models

Include location

Synonymous wordsSynsets (collections of synonymous words) can be extractedfrom WordNet