1 fjording the stream an architecture for queries over streaming sensor data samuel madden, michael...
TRANSCRIPT
1
Fjording The StreamFjording The StreamAn Architecture for Queries over An Architecture for Queries over
Streaming Sensor DataStreaming Sensor Data
Samuel Madden, Michael Samuel Madden, Michael FranklinFranklin
UC BerkeleyUC Berkeley
Feb 28, 2002 Fjording ICDE 2
IntroductionIntroduction
Telegraph Sensor Query Processing Telegraph Sensor Query Processing ArchitectureArchitecture FjordsFjords
Enable push and pull in query plansEnable push and pull in query plans Operators for streaming dataOperators for streaming data Sensor proxySensor proxy
Sensor - Query MediatorSensor - Query MediatorQuery processor
Web
Sensors
User
Proxy
Feb 28, 2002 Fjording ICDE 3
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 4
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 5
Sensor ChallengesSensor Challenges
Battery Powered Battery Powered 2AA Cells (2800 mAh; 1.01x102AA Cells (2800 mAh; 1.01x104 4 J), Coin Cell J), Coin Cell
(100mAh)(100mAh) Communication Dominates Power CostCommunication Dominates Power Cost Can be exhausted - tens to hundreds of MBs of data Can be exhausted - tens to hundreds of MBs of data
/ sensor/ sensor WirelessWireless
High loss rates (20% @ 5meters , typical)High loss rates (20% @ 5meters , typical) Low bandwith (10kbps)Low bandwith (10kbps)
Near Real TimeNear Real Time Streaming DataStreaming Data
Pushed at (user defined) regular intervalsPushed at (user defined) regular intervals
TinyOS “Mote”TinyOS “Mote”
Feb 28, 2002 Fjording ICDE 6
Requirements of Sensor Requirements of Sensor Query ProcessingQuery Processing
Power SensitivityPower Sensitivity Tolerance to unbounded streams of Tolerance to unbounded streams of
datadata Tolerance to pushTolerance to push Tolerance to intermittent, lossy Tolerance to intermittent, lossy
connectionsconnections Tolerance to failed sensorsTolerance to failed sensors
Feb 28, 2002 Fjording ICDE 7
Traffic ScenarioTraffic Scenario
CA Department of Transportation (CalTrans) CA Department of Transportation (CalTrans) has sensors all over bay area freewayshas sensors all over bay area freeways Inductive loop sensors give speed, flow, vehicle sizeInductive loop sensors give speed, flow, vehicle size Motes could collect this data cheaply Motes could collect this data cheaply
Many possible queriesMany possible queries Commuters want to find congestion Commuters want to find congestion California Highway Patrol (CHP) wants to find California Highway Patrol (CHP) wants to find
accidentsaccidents
Feb 28, 2002 Fjording ICDE 8
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 9
FjordsFjords
Query plan implementationQuery plan implementation Useful for streams and distributed environmentsUseful for streams and distributed environments
Combine push (streaming) data and pull Combine push (streaming) data and pull (static) data(static) data E.g. traffic sensors with CHP accident reportsE.g. traffic sensors with CHP accident reports
Feb 28, 2002 Fjording ICDE 10
Push vs. PullPush vs. Pull
Problem: Need an API to combine push & Problem: Need an API to combine push & pullpull
Operators (e.g. join) data-direction agnosticOperators (e.g. join) data-direction agnostic Push vs. pull implemented in queues Push vs. pull implemented in queues
((connectorsconnectors)) Contrast withContrast with
Iterator modelIterator model Exchange operatorExchange operator
Allows arbitrary combinations of push and Allows arbitrary combinations of push and pullpull
Feb 28, 2002 Fjording ICDE 11
Pull ExamplePull Example
OperatorOperatorQueue q;…Queue q;…
Tuple Tuple process()process() { {
Tuple t = q.Tuple t = q.get(),get(), outt = null; outt = null;
If (t != null) { If (t != null) {
<process t> <process t>
} else { … do something else … }} else { … do something else … }
return outt;return outt;
}}
Pull QueuePull QueueOperator parent, child;Operator parent, child;Tuple Tuple get()get() { { Tuple t = null;Tuple t = null; while (t == null) {while (t == null) { t = child.t = child.process()process();; }} return t;return t;}}
get()Pull
Connection
Scan
process()
Notice:
•Iterator semantics by making get() blocking
•Get() can return null
•Process() can return null
Feb 28, 2002 Fjording ICDE 12
Push ExamplePush ExampleOperatorOperatorQueue q;…Queue q;…
Tuple Tuple process()process() { {
Tuple t = q.Tuple t = q.get()get(), outt = null;, outt = null;
If (t != null) { If (t != null) {
<process t> <process t>
} else { … do something else … }} else { … do something else … }
return outt;return outt;
}}
ThreadThreadwhile(true) {while(true) {
Tuple t = op.Tuple t = op.process()process();;
if (t != null) op.outq.if (t != null) op.outq.enqueue(t)enqueue(t);;
}}
Push QueuePush QueueOperator parent, child;Operator parent, child;Vector v = new Vector();Vector v = new Vector();Tuple Tuple get()get() { {
if (v.size() > 0) return v.removeFirst();if (v.size() > 0) return v.removeFirst();else return null;else return null;
}}Tuple Tuple enqueueenqueue(Tuple t) {(Tuple t) {
v.put(t);v.put(t);}}
Push Connection
Scan
get()
Feb 28, 2002 Fjording ICDE 13
Fjord ExampleFjord Example
Push
Push
Pull
Query processor
Feb 28, 2002 Fjording ICDE 14
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 15
Continuous QueriesContinuous Queries
Given user queries over current sensor dataGiven user queries over current sensor data Expect that many queries will be over the same data Expect that many queries will be over the same data
sources (e.g. traffic sensors)sources (e.g. traffic sensors) Queries over current data always looking at same Queries over current data always looking at same
tuplestuples Those queries can share Those queries can share
Current tuples Current tuples Work (e.g. selections)Work (e.g. selections)
Sharing reduces messages, thereby power!Sharing reduces messages, thereby power! A new query can be “folded” into an existing A new query can be “folded” into an existing
queryquery Use old instances of scans and selectionsUse old instances of scans and selections
Feb 28, 2002 Fjording ICDE 16
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 17
Relational Operators And Relational Operators And StreamsStreams
Selection and Projection Apply NaturallySelection and Projection Apply Naturally No Blocking Operators No Blocking Operators
Sorts and aggregates over the entire streamSorts and aggregates over the entire stream Nested loops and sort-merge joinNested loops and sort-merge join
Windowed OperatorsWindowed Operators Sorts, aggregates, etc.Sorts, aggregates, etc.
Online, Interactive QP TechniquesOnline, Interactive QP Techniques In memory symmetric hash joinIn memory symmetric hash join Alternatives: ripple-join, Xjoin, etc.Alternatives: ripple-join, Xjoin, etc. Partial ResultsPartial Results
Feb 28, 2002 Fjording ICDE 18
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 19
Query Processor
Sensor ProxiesSensor Proxies Mediate between Sensors and FjordsMediate between Sensors and Fjords
Push operators out to sensorsPush operators out to sensors Hide query processing, knowledge of multiple queries Hide query processing, knowledge of multiple queries
from sensorsfrom sensors Hide details of sensors from query processorHide details of sensors from query processor Enable power-sensitivityEnable power-sensitivity
Sensor Proxy
Operators
Query
Query
Parsed Queries[sources, ops][fields, filters,
aggregates, rates]
[tuples]
Registration
Feb 28, 2002 Fjording ICDE 20
What runs where?What runs where?
(Multi-query) Distributed Optimization (Multi-query) Distributed Optimization Challenge:Challenge: Given set of operators, proxy must choose:Given set of operators, proxy must choose:
Run on sensor or,Run on sensor or, Run on local query processorRun on local query processor
Running on sensors saves powerRunning on sensors saves power Simple computations cheaper than messagesSimple computations cheaper than messages Selection, aggregation reduce communication costSelection, aggregation reduce communication cost
Sensors have limited resourcesSensors have limited resources All queries can’t run in all sensors simultaneouslyAll queries can’t run in all sensors simultaneously Limited state precludes big joins or lots of groupsLimited state precludes big joins or lots of groups
Queries share operatorsQueries share operators Operators vary in selectivityOperators vary in selectivity
Feb 28, 2002 Fjording ICDE 21
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 22
Building a QueryBuilding a Query How to translate a declarative query into a Fjord?How to translate a declarative query into a Fjord? Just like traditional query processing, except:Just like traditional query processing, except:
Branches originating in sensors connected by push connectorsBranches originating in sensors connected by push connectors Sensor proxy handles scans, selections over sensorsSensor proxy handles scans, selections over sensors
Proxy delivers tuples from sensorsProxy delivers tuples from sensors Proxy pushes-down selections transparently Proxy pushes-down selections transparently
Output of join is push if one or both inputs is pushOutput of join is push if one or both inputs is push Join carefully chosenJoin carefully chosen
Sensor Proxyaccident = true
Pull: Data Request
Data Request
Data
Index NL Join
Push PushSpeed < 30
Feb 28, 2002 Fjording ICDE 23
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 24
Query & FjordQuery & Fjord
Simple Test Query:Simple Test Query:
SELECT AVG(s.speed, SELECT AVG(s.speed, ww))
FROM sensors AS sFROM sensors AS s
WHERE s.loc in WHERE s.loc in {{userLocsuserLocs}}
Loc {userLocs}
BHL Server
Telegraph Server
Average
Feb 28, 2002 Fjording ICDE 25
Multiple Queries in a FjordMultiple Queries in a Fjord
Loc {userLocs}
BHL Server
Telegraph Server
Loc {userLocs}Loc {userLocs}
Feb 28, 2002 Fjording ICDE 26
Fjord Performance Fjord Performance (In Telegraph)(In Telegraph)
Running Time Per Query
0
500
1000
1500
2000
2500
3000
3500
4000
1 2 5 10
Number of Queries
Runnin
g T
ime (
seconds)
Single Fjord Multiple Fjords
Feb 28, 2002 Fjording ICDE 27
Sensor Proxy for TrafficSensor Proxy for Traffic Measure benefit of pushing Measure benefit of pushing
computation into sensors; in this computation into sensors; in this case, vehicle identification.case, vehicle identification. Simple “aggregation” dramatically Simple “aggregation” dramatically
reduces power costsreduces power costsw
w w
t0: Vehicle Covers Su t1: Vehicle Covers Su and Sd
t2: Vehicle Covers Sd t3: Vehicle Covers Neither Sensor
Feb 28, 2002 Fjording ICDE 28
Pushing Aggregates Saves Pushing Aggregates Saves PowerPower
Atmel (TinyOS CPU) Simulator
100 samples / sec5 vehicles / sec
7x power savings
Power Drain (W) vs. Sample Method
00.0010.0020.0030.0040.0050.0060.0070.008
Every Sample Every Vehicle
Sampling Method
Pow
er (W
)
Feb 28, 2002 Fjording ICDE 29
RoadmapRoadmap BackgroundBackground
SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario
FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults
GraphsGraphs Related WorkRelated Work
Feb 28, 2002 Fjording ICDE 30
Related WorkRelated Work
CougarCougar Interactive & Adaptive Query Interactive & Adaptive Query
ProcessorsProcessors Tukwila, Xjoin, Eddy, CONTROLTukwila, Xjoin, Eddy, CONTROL
Continuous Query ProcessorsContinuous Query Processors NiagaraCQ, Xfilter, CACQNiagaraCQ, Xfilter, CACQ
Directed DiffusionDirected Diffusion Volcano / Exchange OperatorVolcano / Exchange Operator Temporal DatabasesTemporal Databases
Feb 28, 2002 Fjording ICDE 31
Conclusions & Future WorkConclusions & Future Work Required for sensor query processing:Required for sensor query processing:
Fjords : API for combining push & pullFjords : API for combining push & pull Sensor Proxy: Mediator between sensors and QPSensor Proxy: Mediator between sensors and QP Streaming OperatorsStreaming Operators
Big benefit from pushing selections, aggregates Big benefit from pushing selections, aggregates into networkinto network
Combining multiple queries is a winCombining multiple queries is a win Extensions & Future Work:Extensions & Future Work:
Multi-query optimization & adaptivity (CACQ, SIGMOD Multi-query optimization & adaptivity (CACQ, SIGMOD 2002)2002)
Push down selections & aggregates (Submitted to VLDB Push down selections & aggregates (Submitted to VLDB 2002)2002)
Sensor proxy policiesSensor proxy policies