Slice&Dice: recognizing food preparation activities using embedded accelerometers

Cuong Pham & Patrick OlivierCulture Lab

School of Computing ScienceNewcastle University

Overview

Introduction Instrumented utensils Activity Recognition framework Experiment

Data collection & annotation Evaluation

Reflections

Introduction: Ambient Kitchen project

Goal: help people with dementia live more independent by providing situated services and prompting based on context recognition

Kitchen context: what people are doing objects people are interacting (i.e. food

ingredients) user locations etc.

Introduction: Ambient Kitchen project

Ambient kitchen: a lab-based ambient intelligence environment, designed using high fidelity prototype.

Introduction: prior work

Sensors worn on different parts of users body [Bao2004, Tapia2007, Ravi2005].

Detected outdoor activities such as running, walking, climbing, cycling etc. or high level activities[Wu2007]

Data collected under laboratory [Ravi2005] or semi-realistic conditions [Bao2004]

People with dementia needed fine-grained prompts to complete low-level activities [Wherton2008]

Introduction: system requirements

Detect low-level activities Sensors hidden from users No wires The cost & ease of deployment Comfortable-to-use Reasonable accuracy

Instrumented utensils: Wii ADXL330

A thin, low power, 3-axis accelerometer

Signal conditioned voltage outputs Dynamic acceleration can be

measured motion, shock and vibration

Acceleration can be measured in a range of ±3g

Instrumented utensils

Modified Wii Remotes were embedded in the kitchen utensils

Activity Recognition Framework

Data Communication & Processing acceleration data X, Y, Z sent to the computer

through a bluetooth device pitch and roll were computed for each triple

X,Y,Z Data Segmentation

data stream were segmented into 32, 64, 128, 256, and 512 sample windows

50% overlap between two consecutive windows.

Activity Recognition Framework

Feature Computation Mean Standard deviation Energy Entropy

Classification algorithms (from Weka Lib) Decision Tree C4.5 Bayesian Networks Naïve Bayes

Experiment: data collection

20 subjects 5 IP cameras 4 utensils: 3 knives and one serving spoon Given ingredients: potatoes, tomatoes,

lettuce, carrots, onions, kiwi fruit, grapefruit, peppers, bread, and butter

No instruction and no time-constrained to the subjects

Task: prepare a mixed salad and sandwich

Experiment: data annotation

Collected videos were annotated using Anvil Multimodal Tool [Kipp2001]

Experiment: example

Experiment: example

Experiment: data annotation

Dataset B annotated by one coder

Dataset A independently annotated by three coders only regions where all there coders agreed

were extracted Dataset B is larger than dataset A, but

dataset A is more consistent than dataset B

Experiment: subject independent evaluation

Trained 19 subjects Tested the remaining one Repeated the process for 20 subjects Finally, aggregated the results Subject to test was not included in the

training dataset

Experiment: evaluation results

Algorithm Dataset A Dataset B

Decision Tree 82.9 77.2

Bayesian nets 78.9 71.3

Naïve Bayes 52.4 73.5

Best accuracies were achieved on window size of 256-sample

Experiment: evaluation analysis

Peeling and stirring were highly distinctive (more than 90%)

Chopping, slicing, coring, scooping performed really good (around 80-90%)

Eating, spreading, shaving, scraping and dicing were below 80%: eating sometimes misclassified as scooping spreading sometimes misclassified as shaving

and coring dicing often misclassified as chopping

Reflection

Low-level food preparation activities can be reliably recognized using sensors embedded in kitchen utensils

Our work will continue with finding features most impact on algorithm

performance detecting objects developing Models

Thank you for your attention

Q&A