LoCaF:Detecting Real-World

States with Lousy Wireless Cameras

Benjamin Meyer, Richard Mietz, Kay Römer

1

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

2

Introduction– Motivation– Challenges

System Architecture Evaluation

Structure

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

3

Towards the Internet of Things– High-level state of things

on the internet– Scalar/specialized sensors

are often limited to one scenario

– Cameras are more flexible

Motivation

SFpark project:

http://sfpark.org/

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

4

Sensor nodes– Constrained

resources Low-cost cameras

– Low resolution– Poor image quality– Low frame rate

Processing is shifted to the gateway

Low-cost hardware

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

5

ScenariosOccupancy of a

roomFree seats in a

roomIndividual

occupancy of parking spots

States Free/occupied Number of persons

Free/occupied for each parking spot

Challenges Possibly lots of movement

Possibly lots of movement

Outdoor Changing lighting

conditions

Picture

Objects to detect People People Cars

Flexible Framework to infer and publish states for divers scenarios

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

6

System Architecture: Overview

0

HTML

RDF

SQL

Tweet

Image capture Compression Wireless transmission

State publication Text templates Different media

Image processing

Regions of interest

Enhancing filters

Object detection

Face detection Mobile object

detection

State inference Rule-based

language

Customizable workflow

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

7

0

HTML

RDF

SQL

Tweet

State publication Text templates Different media

Image processing

Regions of interest

Enhancing filters

Object detection

Face detection Mobile object

detection

State inference Rule-based

language

System Architecture: Sensor Node

Image capture Compression Wireless transmission

Customizable workflow

Camera equipped sensor node

Two capture modes– Time-triggered– Event-triggered (by PIR)

JPEG-compression in hardware

Fragmented transmission to gateway

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

8

0

HTML

RDF

SQL

Tweet

State publication Text templates Different media

Image processing

Regions of interest

Enhancing filters

Object detection

Face detection Mobile object

detection

State inference Rule-based

language

Image capture Compression Wireless transmission

Customizable workflow

Image processing

Regions of interest

Enhancing filters

System Architecture: Processing

INSTITUTE OF COMPUTER ENGINEERING

Parking spot a

Parking spot b

Region selection Lighting compensation Texture enhancement Contrast enhancement Orchestration and

parameterization of enhancements

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

9

Image processing

Regions of interest

Enhancing filters

Object detection

Face detection Mobile object

detection

State inference Rule-based

language

Object detection

Face detection Mobile object

detection

Face detection Adaptive background

subtraction– Classification into fore- and

background– Can adapt to small changes

Blob detection– Each blob is an object

Number of & area covered by objects

System Architecture: Processing

INSTITUTE OF COMPUTER ENGINEERING

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

10

0

HTML

RDF

SQL

Tweet

State publication Text templates Different media

Image processing

Regions of interest

Enhancing filters

Object detection

Face detection Mobile object

detection

State inference Rule-based

language

Image capture Compression Wireless transmission

State inference Rule-based

language

Customizable workflow

Rule-based state inferencecount:map:0:1:freecount:map:1:-1:occupied

State-based

Event-based

area:switch:free:80:occupiedarea:switch:occupied:80:free

System Architecture: Processing

INSTITUTE OF COMPUTER ENGINEERING

count:map:0:1:All seats freecount:map:10:45:Enough seatscount:map:45:70:Almost fullcount:map:70:-1:No seats left

count:map:0:1:freecount:map:1:-1:occupied

area:map:0:80:freearea:map:80:100:occupied

free

occupied

80% covera

ge

80% covera

ge

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

11

0

HTML

RDF

SQL

Tweet

State publication Text templates Different media

Image processing

Regions of interest

Enhancing filters

Object detection

Face detection Mobile object

detection

State inference Rule-based

language

System Architecture: Publishing

Image capture Compression Wireless transmission

Customizable workflow

Every text format (HTML, RDF, TXT, …) Template-based Publishing via

– FTP– Twitter– SQL-Database

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

12

Camera in front of lecture hall during lecture

Estimate number of students Also looking at binary state

(free/occupied) One region, background subtraction &

no filter Three phases:

– Beginning: Entering persons in dribs and drabs

– During: Not many movements– End: Abrupt leaving of students

Evaluation Setup

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

13

Evaluation: Under- and Overestimation Underestimation

– Several persons identified as one

– Persons not recognized because of no movement

Overestimation– Legs recognized as

individual

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

14

Evaluation: Entry phase

OE: 130%

UE: 70%

Avg: 48%Binary state always correct

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

15

Evaluation: Lecture phase

UE: 105%

Avg: 54%Binary state always correct

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

16

Evaluation: Exit phase

OE: ∞

UE: 222%

Avg: 95%Binary state not correct for picture 11-13

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

17

Evaluation: Entry phase revisited

Image filters can significantly change the estimation

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

18

Evaluation: Entry phase revisited

Parameters can significantly change the estimation

Improved avg error: 12%

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

19

Conclusion Flexible framework Use of cameras to be applicable in

divers scenarios Fully customizable by the user in each

step Accuracy quite high

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

20

Questions?

Thank you for your attention.

Time for questions.

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

21

SetupCamera

node

Gateway Netbook with software

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

The Framework: Connection Configuration

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

The Framework: Data Exchange

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

The Framework: Image Processing

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

The Framework: Region Selection / State Inference

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

The Framework: Publishing

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

Filter

INSTITUTE OF COMPUTER ENGINEERING

Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras

28

Evaluation: Parking Spot Scenario

area:switch:free:80:occupiedarea:switch:occupied:80:free Select single spot State switches from free to occupied when car enters (b) and c))

State will switch back when car leaves