adaptive cleaning for rfid data streams vldb 2006 9/12/06 shawn jeffery minos garofalakis michael...

Adaptive Cleaning for Adaptive Cleaning for RFID Data StreamsRFID Data Streams

VLDB 20069/12/06

Shawn Jeffery Minos Garofalakis Michael Franklin UC Berkeley Intel Research Berkeley UC Berkeley

Shawn JefferyHiFi Project

UC Berkeley EECS

RFID: Radio Frequency RFID: Radio Frequency IDentificationIDentification


UC Berkeley EECS

RFID data is dirtyRFID data is dirtyShelf 0 Shelf 1

RFIDReaders

StaticTags

Mobile Tags

15ft

1.5ft

3ft9ft

3ft

3ft

3ft

A simple experiment:

•2 RFID-enabled shelves

•10 static tags

•5 mobile tags


UC Berkeley EECS

RFID Data CleaningRFID Data Cleaning

Time

Raw readings

Smoothed output

• RFID data has many dropped readings• Typically, use a smoothing filter to

interpolateSELECT distinct tag_idFROM RFID_stream [RANGE ‘5 sec’]GROUP BY tag_id

SELECT distinct tag_idFROM RFID_stream [RANGE ‘5 sec’]GROUP BY tag_idBut, how to set the size

of the window?

But, how to set the size of the window?

Smoothing Filter


UC Berkeley EECS

Window Size for RFID Window Size for RFID SmoothingSmoothing

Fido moving Fido resting

Small windowSmall windowRealityReality

Raw readingsRaw readings

Large windowLarge window

Need to balance completeness vs. capturing tag movement

Need to balance completeness vs. capturing tag movement


UC Berkeley EECS

Truly Declarative Truly Declarative SmoothingSmoothing

• Problem: window size non-declarative• Application wants a clean stream

of data• Window size is how to get it

• Solution: adapt the window size in response to data


UC Berkeley EECS

ItineraryItinerary

• Introduction: RFID data cleaning• A statistical sampling perspective• SMURF

• Per-tag cleaning• Multi-tag cleaning

• Ongoing work• Conclusions


UC Berkeley EECS

A Statistical Sampling A Statistical Sampling PerspectivePerspective

• Key Insight: RFID data random sample of present tags

• Map RFID smoothing to a sampling experiment


UC Berkeley EECS

RFID’s Gory DetailsRFID’s Gory Details

Epoch TagID ReadRate

0 1 .9

0 2 .6

0 3 .3

Tag 1

Tag 2

Tag 3

Tag 4

Antenna & readerTags

E1 E2 E3 E4 E5 E6 E7 E8 E9E0

Read Cycle (Epoch)

Read Cycle (Epoch)

(For Alien readers)

Tag List


UC Berkeley EECS

RFID Smoothing to SamplingRFID Smoothing to Sampling

RFID Sampling

Read cycle (epoch) Sample trial

Reading Single sample

Smoothing window Repeated trials

Read rate Probability of inclusion (pi)

Now use sampling theory to drive adaptation!


UC Berkeley EECS

SMURFSMURF

• Statistical Smoothing for Unreliable RFID Data

• Adapts window based on statistical properties

• Mechanisms for:• Per-tag and multi-tag cleaning

Multi-tagCleaning

SMURF

Per-tagCleaning

raw RFID streams

cleanedcount readings

cleanedper-tag readings

Application(s) Application(s)


UC Berkeley EECS

Per-Tag Smoothing: Per-Tag Smoothing: Model and BackgroundModel and Background

• Use a binomial sampling model

Time (epochs)

pi

1

0

Smoothing Window

wi Bernoulli trials

piavg

Si

(Read rate of tag i)

E1 E2 E3 E4 E5 E6 E7 E8 E9E0


UC Berkeley EECS

Per-Tag Smoothing: Per-Tag Smoothing: CompletenessCompleteness

• If the tag is there, read it with high probability

Want a large window

pi

1

0

Reading with a low pi

Expand the window

Time (epochs)E1 E2 E3 E4 E5 E6 E7 E8 E9E0


UC Berkeley EECS

Per-Tag Smoothing: Per-Tag Smoothing: CompletenessCompleteness

Expected epochs needed to read

With probability 1-

Desired window size for tag i

1

ln*1avgi

ip

w


UC Berkeley EECS

Per-Tag Smoothing: Per-Tag Smoothing: TransitionsTransitions• Detect transitions as statistically

significant changes in the data

pi

1

0

Statistically significant difference Flag a transition and

shrink the window

The tag has likely left by this point



UC Berkeley EECS

Per-Tag Smoothing: Per-Tag Smoothing: TransitionsTransitions

# expected readings

Is the difference “statistically significant”?

# observed readings

)1(**2|*||| avgi

avgii

avgiii ppwpwS


UC Berkeley EECS

SMURF in ActionSMURF in Action

Fido moving Fido resting

SMURFSMURF

Experiments with real and simulated data show similar results


UC Berkeley EECS

Multi-tag CleaningMulti-tag Cleaning

• Some applications only need aggregates• E.g., count of items on each shelf Don’t need to track each tag!

• Use statistical mechanisms for both:• Aggregate computation • Window adaptation


UC Berkeley EECS

Aggregate Aggregate ComputationComputation

• –estimators (Horvitz-Thompson) • Count:

• P[tag i seen in a window of size w]:

Use small windows to capture movementUse the estimator to compensate for lost

readings

wSiwN

1

wavgii p )1(1


UC Berkeley EECS

Window AdaptationWindow Adaptation

• Upper bound window similar to per-tag

• “Transition” based on variance within subwindows

1

ln*1avgp

w

Count

Nw

Nw’


'VarVar2ww NN


UC Berkeley EECS

Multi-tag ScenarioMulti-tag Scenario


UC Berkeley EECS

Ongoing Work: Spatial Ongoing Work: Spatial SmoothingSmoothing

• With multiple readers, more complicated

Reinforcement

A? B? A U B? A B?

Arbitration

A? C? All are addressed by statistical framework!

U

A

B

C

D

Two rooms, two readers per room


UC Berkeley EECS

Beyond RFIDBeyond RFID

• -estimator for other aggregates Use SMURF for sensor networks

• Use SMURF in general streaming systems (e.g., TelegraphCQ)

Remove RANGE clause from CQL

Other sensor dataOther sensor data

Other streaming dataOther streaming data


UC Berkeley EECS

Related WorkRelated Work

• Commercial RFID middleware• Smoothing filters: need to set smoothing

window

• RFID-related work• Rao et al., StreamClean: complementary• Intel Seattle, HiFi, ESP: static window size

• BBQ, MauveDB• Heavyweight, model-based• SMURF is non-parametric, sampling-based

• Statistical filters (digital signal processing)• Non-linear digital filters inspired SMURF design


UC Berkeley EECS

ConclusionsConclusions

• Current smoothing filters not adequate

• Not declarative!

• SMURF: Declarative smoothing filter

• Uses statistical sampling to adapt window size


UC Berkeley EECS

Thanks!Thanks!

[email protected]

adaptive cleaning for rfid data streams vldb 2006 9/12/06 shawn jeffery minos garofalakis michael...

Documents

tag smoothing

data slide

tag movement slide

distinct tag

rate of tag

multitag cleaning slide

smoothing filter slide

unreliable rfid data