the stanford data streams research project

The Stanford Data Streams The Stanford Data Streams Research ProjectResearch Project

Profs. Rajeev Motwani & Jennifer Widom

And a cast of full- and part-time students:Arvind Arasu, Brian Babcock, Shivnath Babu,

Mayur Datar, Gurmeet Manku, Liadan O’Callaghan, Justin Rosentein, Qi Sun, Rohit Varma

stanfordstreamdatamanager

stanfordstreamdatamanager 2

Data StreamsData Streams• Traditional DBMS -- data stored in finite,

persistent data setsdata sets

• New applications -- data as multiple, continuous, rapid, time-varying data streamsdata streams– Network monitoring and traffic engineering– Security applications– Telecom call records– Financial applications– Web logs and click-streams– Sensor networks– Manufacturing processes


ChallengesChallenges• Multiple, continuous, rapid, time-varyingMultiple, continuous, rapid, time-varying

streams of data

• Queries may be continuous continuous (not just one-time)– Evaluated continuously as stream data arrives– Answer updated over time

• Queries may be complexcomplex– Beyond element-at-a-time processing– Beyond stream-at-a-time processing


Using Traditional DatabaseUsing Traditional Database

User/ApplicationUser/Application

LoaderLoader

QueryQuery ResultResultResultResult

……QueryQuery

……


New Approach for Data StreamsNew Approach for Data Streams


Register QueryRegister Query

Stream QueryStream QueryProcessorProcessor

ResultResult


New Approach for Data StreamsNew Approach for Data Streams


Register QueryRegister Query

Stream QueryStream QueryProcessorProcessor

ResultResult

Scratch SpaceScratch Space(Memory and/or Disk)(Memory and/or Disk)

DataStream

ManagementSystem(DSMS)


DBMS versus DSMSDBMS versus DSMS


DBMS versus DSMSDBMS versus DSMS• Persistent relations • Transient streams (and

persistent relations)


DBMS versus DSMSDBMS versus DSMS• Persistent relations

• One-time queries

• Transient streams (and persistent relations)

• Continuous queries




• Random access



• Sequential access




• Random access

• Access plan determined by query processor and physical DB design




• Unpredictable data arrival and characteristics




• Random access

• Access plan determined by query processor and physical DB design

• “Unbounded” disk store




• Unpredictable data arrival and characteristics

• Bounded main memory


Sample ApplicationsSample Applications• Network management and traffic engineering

(e.g., Sprint)– Streams of measurements and packet traces– Queries: detect anomalies, adjust routing

• Telecom call data (e.g., AT&T)– Streams of call records– Queries: fraud detection, customer call patterns,

billing


Sample Applications (cont’d) Sample Applications (cont’d) • Network security

(e.g., iPolicy, NetForensics/Cisco, Netscreen)– Network packet streams, user session information– Queries: URL filtering, detecting intrusions & DOS

attacks & viruses

• Financial applications (e.g., Traderbot)– Streams of trading data, stock tickers, news feeds– Queries: arbitrage opportunities, analytics, patterns


Sample Applications (cont’d) Sample Applications (cont’d) • Web tracking and personalization

(e.g., Yahoo, Google, Akamai)– Clickstreams, user query streams, log records– Queries: monitoring, analysis, personalization

• Truly massive databases (e.g., Astronomy Archives)– Stream the data by once (or over and over)– Queries do the best they can


Making Things ConcreteMaking Things Concrete• Database = two streams of mobile call records

– Outgoing(connectionID, caller, start, end)– Incoming(connectionID, callee, start, end)

• Query language = SQLFROM clauses can refer to streams and/or relations


Query Example 1Query Example 1• Find all outgoing calls longer than 2 minutes

(relational selection)SELECT O.connectionID, O.callerFROM Outgoing OWHERE O.end – O.start > 2

• Result requires unbounded storage

• Can provide result as data stream


Query Example 2Query Example 2• Pair up callers and callees (relational join)

SELECT O.caller, I.calleeFROM Outgoing O, Incoming IWHERE O.connectionID = I.connectionID

• Can still provide result as data stream

• Requires unbounded temporary storage (without additional assumptions)


Query Example 3Query Example 3• Find total connection time for each caller

(relational grouping and aggregation)SELECT O.caller, sum(O.end – O.start)FROM Outgoing OGROUP BY O.caller

• Cannot provide result in (append-only) stream


Project GoalProject Goal Reconsider all aspects of data management

and processing in presence of data streams


Remainder of TalkRemainder of Talk• Data stream model

• Queries over data streams– Language, semantics, evaluation & optimization

• DSMS query processing architecture and system internals

• Results to date

• Ongoing work

• Related work


Data ModelData Model• Database: relations + data streamsrelations + data streams

• Stream characteristics– Type of data (schema)– Data distribution– Flow rate– Stability of distribution and flow– Ordering and other constraints– Synchronization of multiple streams– Distributed streams


Data Stream Queries -- Basic IssuesData Stream Queries -- Basic Issues• Answer availability

– One-time– Multiple-time– Continuous (“standing”), stored or streamed

• Registration time– Predefined– Ad-hoc

• Stream access– Arbitrary– Sliding window (special case: size = 1)


Data Stream Queries -- Basic IssuesData Stream Queries -- Basic Issues• Answer availability

– One-time– Multiple-time– Continuous (“standing”), stored or streamed

• Registration time– Predefined– Ad hoc

• Stream access– Arbitrary– Sliding window (special case: size = 1)


Query Language & SemanticsQuery Language & Semantics• Specifying queries over streams

– SQL-like versus dataflow network of operators– Sliding windows as first-class query construct

• Semantic issues– Blocking operators, e.g., aggregation, order-by– Streams as sets versus lists– Timestamping


Query Evaluation -- ApproximationQuery Evaluation -- Approximation• Why approximate?

– Streams are coming too fast– Exact answer requires unbounded storage or

significant computational resources– Ad hoc queries reference history

• Issues in approximation– Sliding windows, sampling, synopses, …– How is approximation controlled?– How is it understood by user?

• Accuracy-efficiency-storage tradeoffAccuracy-efficiency-storage tradeoff


Query Evaluation -- AdaptivityQuery Evaluation -- Adaptivity• Why adaptivity?

– Queries are long-running– Fluctuating stream arrival & data characteristics– Evolving query loads

• Issues in adaptivity– Adaptive resource allocation (memory,

computation)– Adaptive query execution plans


Query Evaluation -- Multiple QueriesQuery Evaluation -- Multiple Queries• Possibly large number of continuous queries

• Long-running

• Shared resources

• Multi-query optimization


Query Evaluation -- Distributed StreamsQuery Evaluation -- Distributed Streams1 Many physical streams but one logical stream

– E.g., maintain top 100 visited pages at Yahoo

2 Correlate streams at distributed servers– E.g., network monitoring

3 Many streams controlled by a few servers– E.g., sensor networks

• Issues– Move processing to streams, not streams to

processor– Approximation-bandwidth tradeoffApproximation-bandwidth tradeoff


Query Processing ArchitectureQuery Processing Architecture

Input Data Streams

Usersissue

continuous and ad-hoc queries

Administrator can monitor query

executionand adjust run-time

parameters

Applicationsregister

continuous queries

OutputStream

X

X

Waiting Op

Ready Op

Running Op

Synopses Query Plans


DSMS InternalsDSMS Internals• Query plans: operators, synopses, queuesoperators, synopses, queues

• Memory management– Dynamic allocation to buffers, queues, synopses– Accuracy vs. memory tradeoff– Operators adapt gracefully to memory reallocation

• Scheduler– Handles variable-rate input streams– Handles varying operator and query requirements


Some Results to DateSome Results to Date• Algorithms on data streams

– Online clustering [FOCS 2000, ICDE 2002]

– Online quantiles [SIGMOD 98, SIGMOD 99]

– Statistics over sliding windows [SODA 2002]

– Online frequency counting

• Theory of stream query processing– Memory requirements of stream queries [PODS02]

• System design– STREAMSTREAM: stanfordstreamdatamanager


STREAM System ImplementationSTREAM System Implementation• Comprehensive DSMS query processor

• Broad suite of operators and synopses

• Sophisticated “developer’s workbench” interface– Submit queries in extended SQL or algebra– Submit or edit query plans in XML or GUI– Query plan execution visualizer– On-the-fly modification of memory allocation,

scheduling policies, etc.


Ongoing WorkOngoing Work• Algebra for streams

• Synopses and algorithmic issues

• Memory management issues

• Exploiting constraints on streams

• Approximation in query processing

• Distributed stream processing

• System development


Ongoing Work -- ConstraintsOngoing Work -- Constraints• Exploiting constraints on streams in query

processing– Foreign-key joins, referential integrity, clustering,

ordering– Need not be exact (e.g., k-clustered)– Reduce memory requirements– Unblock blocking operators


Ongoing Work -- Approximation in Ongoing Work -- Approximation in Query ProcessingQuery Processing

• Understanding behavior of approximate operators when composed

• Memory allocation to operators in a plan, given per-operator memory-accuracy curve

• Best query plan, assuming best memory allocation

• Multiple (weighted) queries sharing resources


Related WorkRelated Work• Triggers, alerters, materialized views,

continuous queries on conventional DBs, pub/sub, sequence & temporal databases, …

• TelegraphTelegraph project at UC Berkeley

• NiagaraNiagara project at Wisconsin/OGI

• AmazonAmazon project at Cornell

• AuroraAurora project at Brown/MIT

• And others

For Papers and General Info.For Papers and General Info.

http://www-db.stanford.edu/stream

stanfordstreamdatamanager

the stanford data streams research project

Documents

persistent data

streams of mobile

user query streams

stream data arrivesanswer

time processingbeyond

persistent relationsdbms

query processor

new applications