Wishbone: Profile-based Partitioning for Sensornet Applications

Ryan Newton, Sivan Toledo, Lewis Girod, Hari Balakrishnan, and Samuel MaddenMIT CSAIL

AbstractThe ability to partition sensor network application code acrosssensor nodes and backend servers is important for running com-plex, data-intensive applications on sensor platforms that haveCPU, energy, and bandwidth limitations. This paper presentsWishbone, a system that takes a dataflow graph of operatorsand produces an optimal partitioning. With Wishbone, userscan run the same program on a range of sensor platforms, in-cluding TinyOS motes, smartphones running JavaME, and theiPhone. The resulting program partitioning will in general bedifferent in each case, reflecting the different node capabili-ties. Wishbone uses profiling to determine how each opera-tor in the dataflow graph will actually perform on sample data,without requiring cumbersome user annotations. Its partition-ing algorithm models the problem as an integer linear programthat minimizes a linear combination of network bandwidth andCPU load and uses program structure to solve the problem ef-ficiently in practice. Our results on a speech detection applica-tion show that the system can quickly identify good trade-offsgiven limitations in CPU and network capacity.

1 IntroductionAn important class of sensor computing applicationsare data-intensive, involving multiple embedded sensorseach sampling data at tens or hundreds of kilohertz andgenerating many megabytes per second in aggregate. Ex-amples include acoustic localization of animals, gun-shots, or speakers; structural monitoring and vibrationanalysis of bridges, buildings, and pipes; object trackingin video streams, etc. Over the past few years, impres-sive advances in sensor networking hardware and soft-ware have made it possible to prototype these applica-tions. However, two challenges confront the developerwho wants to deploy and sustain these applications:

• Heterogeneity: Thanks to hardware advances, onecan run these applications on a variety of embed-ded devices, including “motes”, smartphones (whichthemselves are varied), embedded Linux devices(e.g., Gumstix, WiFi access points), etc. This rich-ness of hardware and software is good because it al-lows the developer to pick the right platforms for atask and evolve the infrastructure with time. On theother hand, it poses a software nightmare becauseit requires code to be developed multiple times, orported to different platforms.

• Decomposition: A simple way of designing suchsystems would deliver all the gathered data to a cen-tral server, with all the computation running there.

This approach may consume an excessive amount ofbandwidth and energy. A different approach is torun all of the computation “in the sensor network”,but often the computational capabilities of the sen-sor nodes are insufficient. The question is: how bestto partition an application between the server(s) andthe embedded nodes? Improper partitioning can loseimportant data, waste energy, and may cause appli-cations to simply not work as desired.

No current solution addresses both of these challenges.To support heterogeneity, one might be able to write pro-grams in a language like Java. Unfortunately, some plat-forms do not support Java, or may not support it in its fullgenerality; in addition, Java virtual machines for embed-ded devices are of uneven quality. More importantly, itis difficult to partition such a program in a way that willperform well on any given platform without a significantamount of tuning and manual optimization. That, in turn,limits the ability to swap out the underlying hardwareplatform, or even to move computation between the em-bedded nodes and servers.

We have developed Wishbone, a system that allowsdevelopers to achieve both goals for applications that sat-isfy two conditions:

• Streaming dataflow model: The application shouldbe written as a stream-oriented collection of opera-tors configured as a dataflow graph.

• Predictable input rates and patterns: The input datarates at the sensors gathering data don’t change inunpredictable ways.

To use Wishbone, the developer writes a program in ahigh-level stream-processing language, WaveScript [16],which has a common runtime for both embedded nodesand servers. We have extended our open-source Wave-Script compiler to produce efficient code for several em-bedded platforms: TinyOS 2.0, smartphones runningJava J2ME, the iPhone, Nokia tablets, various WiFi ac-cess points, and any POSIX compliant platform support-ing GCC. These platforms are sufficiently diverse thatgenerating high-performance native code from a sharedhigh-level language is itself a challenge. Fortunately, wehave an advantage in WaveScript’s domain-specificity:the compiler has additional information that it can use tooptimize programs for specific streaming workloads.

We have used WaveScript in several applications, in-cluding: locating wild animals with microphone arrays,locating leaks in water pipelines, and detecting potholesin sensor-equipped taxis. For the purposes of this paper,

we chose to focus on two applications that highlight theprogram partitioning features of Wishbone: a speech de-tector that identifies when a person is speaking in a roomand a 22-channel EEG application. Each is based on anapplication currently in use by our group (EEG) or byother groups (speaker detection). Both were ported1 toWaveScript for the evaluation in this paper.

The key function of Wishbone is, given a WaveScript-produced dataflow graph of stream operators, to parti-tion it into in-network and server-side components. Ituses a profile-driven approach, where the compiler exe-cutes each operator against programmer-supplied sampledata, using real embedded hardware or a cycle-accuratesimulation. After profiling, we are able to estimate theCPU and communication requirements of every opera-tor on every platform. Wishbone depends on this sampledata being representative of the actual input the sensorwill see during deployment; we believe this is a valid as-sumption and justify it in our experiments.

Determining a good partitioning is difficult even af-ter one uses a profiler to determine the computationaland network load imposed by each operator. Wishbonemodels the partitioning problem as an integer linear pro-gram (ILP), seeking to minimize a combination of net-work bandwidth and CPU consumption subject to hardupper bounds on those resources. With these criteria,our ILP formulation will find optimal solutions—and al-though ILP is an NP-hard problem, in practice our imple-mentation can partition dataflow graphs containing overa thousand operators in a few seconds.

Our results show that the system can quickly identifythe optimal partition given constraints on CPU and net-work capacity. And picking the right partition matters. Inour evaluation, our weakest platform got 0% of speakerdetection results through the network successfully whendoing all work on the server, and 0.5% when doing allwork at the node. We can do 20× better by picking theright intermediate partition. Because the optimal parti-tioning changes depending on the hardware platform andthe number of nodes in the network, manual partitioningis likely to be tedious at best. For larger graphs (suchas our 1412 node electroencephalography (EEG) appli-cation), doing the partitioning by hand with any degreeof confidence becomes extremely difficult.

Finally, we note that we do not intend that Wishbonebe used only as a completely automated partitioning tool,but also as a part of an interactive design process with theprogrammer in the loop. In addition to recommendingpartitions, Wishbone can find situations in which thereis no feasible partitioning of a program; e.g., because

1WaveScript is an imperative language with a C-like syntax. Aninitial port of an application from C/C++ is very quick: cut, paste, andclean it up. Refactoring to expose the parallel/streaming structure ofthe application may be more involved.

fun F I R F i l t e r ( c o e f f s , s t rm ) {N = Array : l e n g t h ( c o e f f s ) ;f i f o = FIFO : make (N ) ;f o r i = 1 to N−1 { FIFO : enqueue ( f i f o , 0 ) } ;i t e r a t e x in s t rm {

FIFO : enqueue ( f i f o , x ) ;sum = 0 ;f o r i = 0 to N−1 {

sum += c o e f f s [ i ] ∗ FIFO : peek ( f i f o , i ) ;} ;FIFO : dequeue ( f i f o ) ;emit sum ;

}}

fun L o w F r e q F i l t e r ( s t rm ) {e v e n S i g n a l = GetEven ( s t rm ) ;o d d S i g n a l = GetOdd ( s t rm ) ;/ / even samples go t o one f i l t e r , odds t h e o t h e r :lowFreqEven = F I R F i l t e r ( hLow Even , e v e n S i g n a l ) ;lowFreqOdd = F I R F i l t e r ( hLow Odd , o d d S i g n a l ) ;/ / now recombine themAddOddAndEven ( lowFreqEven , lowFreqOdd )

}fun G e t C h a n n e l F e a t u r e s ( s t rm ) {

low1 = L o w F r e q F i l t e r ( s t rm ) ;low2 = L o w F r e q F i l t e r ( low1 ) ;low3 = L o w F r e q F i l t e r ( low2 ) ;

h igh4 = H i g h F r e q F i l t e r ( low3 ) ; / / we need t h i slow4 = L o w F r e q F i l t e r ( low3 ) ;l e v e l 4 = MagWithScale ( f i l t e r G a i n s [ 3 ] , h igh4 ) ;

h igh5 = H i g h F r e q F i l t e r ( low4 ) ; / / and t h i s onelow5 = L o w F r e q F i l t e r ( low4 ) ;l e v e l 5 = MagWithScale ( f i l t e r G a i n s [ 4 ] , h igh5 ) ;

h igh6 = H i g h F r e q F i l t e r ( low5 ) ; / / and t h i s onel e v e l 6 = MagWithScale ( f i l t e r G a i n s [ 5 ] , h igh6 ) ;zipN ( [ l e v e l 4 , l e v e l 5 , l e v e l 6 ] ) ;

}

Figure 1: Excerpts from running code in EEG-application. The“low level” FIRFilter function constructs new dataflow opera-tors using iterate. FIRFilter is stateful because it maintainsand modifies fifo. Higher level functions such as LowFreq-Filter and GetChannelFeatures wire together a larger graph.

the bandwidth requirements will always exceed avail-able network bandwidth, or because there are insufficientCPU resources to place bandwidth-reducing portions ofthe program inside the sensor network. In these cases,the programmer will have to either switch to a more pow-erful node platform, reduce the sampling rates or thenumber of sensors, or be willing to run the network inan overload situation where some samples are lost. Inthe overload case, Wishbone can compute how much thedata rates need to be reduced to achieve a viable partition.

2 Language and front-end compilerThe developer writes a program in WaveScript that con-structs a dataflow graph of stream operators. Each op-erator consists of a work function and optional privatestate. The job of the WaveScript front-end compiler isto partially evaluate the program to create the dataflowgraph, whereas the WaveScript backend performs graphoptimizations and reduces work functions to an interme-diate language that can be fed to a number of backendcode generators. Each work function contains an im-

perative routine that processes a single stream element,updates the private state for that dataflow operator, andproduces elements on output streams. (Later, we willsingle out stateless operators that maintain no mutablestate between invocations.)

A WaveScript source program can manipulate streamsas values and thereby wire together operator graphs, asseen in Figure 1. The example in Figure 1 contains psue-docode that wires together the cascading filters found inone of the 22-channels of our EEG application. The eval-uation of the iterate form creates a new dataflow opera-tor and provides its work function. The return value of aniterate is its output stream. For example, the functionFIRFilter in Figure 1 takes a stream as one of its inputsand returns a stream. Within the body of the iterate theemit keyword produces elements on the output stream.The equal (=) operator introduces new variables and thelast expression in a {...} block is its return value. Typeannotations are unnecessary.

2.1 Program Distribution

Thus far, our description applies to WaveScript programsthat run on a single node. To support distributed execu-tion, we extended the language to allow developers tospecify which part of the dataflow graph should be repli-cated on all embedded nodes. This specification is log-ical rather than physical; the physical locations of oper-ators are computed by Wishbone’s partitioner using theprogrammer’s annotations and profiler data.

To create the logical specification in Wishbone, theuser places a subset of the program’s top-level streambindings in a Node{} namespace. All operators in theNode{} namespace are replicated once per embeddednode. This separation is particularly important for state-ful operators, because stateful operators in the Node par-tition have an instance of their state for every node in thenetwork. Stateful operators on the server side are instan-tiated only once.

As an example, consider the code snippet in Fig-ure 2, which shows a node/server program that samplesdata from the microphone and filters it. The operatorreadMic, producing the stream s1, must reside on eachnode, as it samples data from hardware only available onthe embedded node. Because the filtAudio call produc-ing s2 is in the Node partition, its operators will be repli-cated once per node, but can be physically placed eitheron the embedded node or the server, depending on whatthe partitioner determines would be best. If filtAudiocreates stateful operators, their state will need to repli-cated once per node, regardless of where they are placed.This example illustrates the basic repartitioning model,and shows that, while the system is free to move someoperators, there are certain relocation constraints the par-titioner must respect, discussed in the next section.

s2s2 s2

fs3

main

implicit merge point

embedded node partition

server partition

radio msgs

s1s1 s1Unpinned nodesMoveable by partitioner

namespace Node {s1 = readMic(...)s2 = filtAudio(s1)

}s3 = f(s2)main = s3

Figure 2: A program skeleton specifying a replicated streamcomputation across all embedded nodes.

2.1.1 Relocation Constraints

Operators are classified as movable or pinned as fol-lows. First, operators with side-effects—for example,OS-specific foreign calls to sample sensors and blinkLEDs—are pinned to their partition. Likewise, operatorson the server that print output to the user or to a file arepinned. Stateless operators without side-effects are notpinned and are always moveable, allowing them to bemoved into the other partition if the system determinesthat to be advantageous. Finally, stateful operators aretreated differently for the node and server partitions. It isnot generally possible to move stateful server operatorsinto the network—they have a serial execution seman-tics and a single state instance. However, it is possibleto move stateful operators from the node partition to theserver. The state of the operator is duplicated in a tableindexed by node ID. Thus, a single server operator canemulate many instances running within the network.

Relocating stateful operators in this way raises a dif-ferent issue—message loss on wireless links. Operatorsin the node partition may safely assume that all edges be-tween the raw sensors and themselves are lossless. Re-locating an operator to the server means putting poten-tial data loss upstream of it that was not there previously.Stateless operators are insensitive to this kind of loss be-cause they process each element without any memory ofpreceding elements, but stateful operators may performerratically in the face of unexpected missing data, unlessthey have been intentionally engineered to tolerate it.

Because tolerance to data loss in stateful operators isan application-specific issue, Wishbone supports two op-erational modes that can be specified by the programmerat compile time. In conservative mode it will not relocatestateful operators onto the server, refusing to add lossi-ness to a previously lossless edge. In permissive mode,the system will automatically perform these relocations.In the future, it would be possible to extend the system tomake many finer distinctions, such as labeling individualedges as loss-tolerant, or grouping operators together inblocks that cannot be divided by a lossy edge.

2.1.2 Restrictions

The system we present in this paper targets a restricteddomain: first, because we focus on a specific dataflowmodel and, second, because of limitations of our currentimplementation. (Section 9 will discuss generalizing andextending the model.) Presently, our implementation re-quires that any path through the operator graph connect-ing a data source on the node to a data sink on the servermay only cross the network once. The graph partitioningalgorithm in Section 4 does, however, support back-and-forth communication. The reason for the restriction isthat we haven’t yet implemented arbitrary communica-tion for all of our platforms. Note that this does not ruleout all communication from the server to the nodes, it isstill possible, for example, to have configuration param-eters sent from a server to in-network operators.

We make the best of this restriction by leveraging itin a number of ways. As we will see, it enables a sim-plified version of the partitioning algorithm. It can alsofurther filter the set of moveable operators as describedin Section 2.1.1, because pinning an operator pins all up-or down-stream operators (can’t cross back).

3 Profile & PartitionThe WaveScript compiler, implemented in the Schemelanguage, can profile stream graphs by executing themdirectly within Scheme during compilation (using sam-ple input traces). This produces platform-independentdata rates, but cannot determine execution time on em-bedded platforms. For this purpose, we employ a sep-arate profiling phase on the device itself, or on a cycle-accurate simulator for its microprocessor.

First, the partitioner determines what operators mightpossibly run on the embedded platform, discountingthose that are pinned to the server, but including movableoperators together with those that are pinned to the node.The code generator emits code for this partition, insert-ing timing statements at the beginning and end of eachoperator’s work function, and at emit statements, whichrepresent yield points or control transfers downstream.

The partition is then executed on simulated or realhardware. The inserted timing statements print output toa debug channel read by the compiler. For example, weexecute instrumented TinyOS programs either on TMoteSky motes or by using the MSPsim simulator2. In eithercase, timestamps are sent through a real or virtual USBserial port, where they are collected by the compiler.

For most platforms, the above timestamping method issufficient. That is, the only relevant information for parti-tioning is how long each operator takes to execute on that

2We also tried Simics and msp430-gdb for simulation, but MSP-sim was the easiest to use. Note that TOSSIM is not appropriate forperformance modeling.

platform (and therefore, given an input data rate, the per-cent CPU consumed by the operator). For TinyOS, someadditional profiling is necessary. To support subdividingtasks into smaller pieces, we must be able to perform areverse mapping between points in time (during an oper-ator’s execution) and points in the operator’s code. Ide-ally, for operator splitting purposes, we would recover afull execution trace, annotating each atomic instructionwith a clock cycle. Such information, however, wouldbe prohibitively expensive to collect. We have found it issufficient to instead simply time stamp the beginning andend of each for or while loop, and count loop iterations.As most time is spent within loops, and loops generallyperform identical computations repeatedly, this enablesus to roughly subdivide execution of an operator into aspecified number of slices.

After profiling, control transfers to the partitioner. Themovable subgraph of operators has already been deter-mined. Next, the partitioner formulates the partitioningproblem in terms of this subgraph, and invokes an exter-nal solver (described in Section 4) to identify the optimalpartition. The program graph is repartitioned along thenew boundary, and code generation proceeds, includinggenerating communication code for cut edges (e.g., codeto marshal and unmarshal data structures). Also, afterprofiling and partitioning, the compiler generates a visu-alization summarizing the results for the user. The visu-alization, produced using the well-known GraphViz toolfrom AT&T Research, uses colorization to represent pro-filing results (cool to hot) and shapes to indicate whichoperators were assigned to the node partition.

4 Partitioning AlgorithmsIn this section, we describe Wishbone’s algorithms topartition the dataflow graph. We consider a directedacyclic graph (DAG) whose vertices are stream operatorsand whose edges are streams, with edge weights repre-senting bandwidth and vertex weights representing CPUutilization or memory footprint. We only include verticesthat can move across the node-server partition; i.e., themovable subset. The server is assumed to have infinitecomputational power compared to the embedded nodes,which is a close approximation of reality.

The partitioning problem is to find a cut of the graphsuch that vertices on one side of the cut reside on thenodes and vertices on the other side reside on the server.The bandwidth of a given cut is measured as the sumof the bandwidths of the edges in the cut. An exampleproblem is shown in Figure 3.

Unfortunately, existing tools for graph partitioning arenot a good fit for this problem. Tools like METIS [12]or Zoltan [7] are designed for partitioning large scien-tific codes for parallel simulation. These are heuristicsolutions that generally seek to create a fixed number of

1

2

5

4

1

1

2

5

4

1

1

2

5

4

1

1

2

5

4

1

1

2

5

4

1

1

2

5

4

1

budget = 2 budget = 3 budget = 4

bandwidth = 8 bandwidth = 6 bandwidth = 5

Figure 3: Simple motivating example. Vertices are labeled withCPU consumed, edges with bandwidth. The optimal mote par-tition is selected in red. This partitioning can change unpre-dictably, for example between a horizontal and vertical parti-tioning, with only a small change in the CPU budget.

balanced graph partitions while minimizing cut edges.Newer tools like Zoltan support unbalanced partitions,but with a specified ratios, not allowing unlimited andunspecified capacity to the server partition. Further, theyexpect a single weight on each edge and each vertex.They cannot support a situation where the cost of a ver-tex changes depending on the partition is it placed in.This is the situation we’re faced with: diverse hardwareplatforms that not only have varying capacities, but forwhich the relative cost of operators varies (for example,due to a missing floating point unit).

We may also consider traditional task scheduling algo-rithms as a candidate solution to our partitioning prob-lem. These algorithms assign a directed graph of tasksto processors, attempting to minimize the total executiontime. The most popular heuristics for this class of prob-lem are variants of list scheduling, where tasks are prior-itized according to some metric and then added one at atime to the working schedule. But there are three majordifferences between this classic problem and our own.First, task-scheduling does not directly fit the nondeter-ministic dataflow model, as no conditional control flow isallowed at the task level—all tasks execute exactly once.Second, task-scheduling is not designed for vastly un-equal node capabilities. Finally, schedule length is notthe appropriate metric for streaming systems. Schedulelength would optimize for latency: how fast can the sys-tem process one data element. Rather, we wish to op-timize for throughput, which is akin to scheduling for atask-graph repeated ad infinitum.

Thus we have developed a different approach. Ourtechnique first preprocesses the graph to reduce the parti-tion search space. Then it constructs a problem formula-tion based on the desired objective function and calls anexternal ILP solver. By default, Wishbone currently usesthe minimum-cost cut subject to not exceeding the CPUresources of the embedded node or the network capacity

of the channel. Cost here is defined as a linear combina-tion of CPU and network usage, α·CPU+β·Net (whichcan be a proxy for energy usage). Therefore we set fournumbers for each platform: the CPU/Network resourcelimits, and coefficients α, β. The user may override thesequantities to direct the optimization process.

4.1 Preprocessing

The graph preprocessing step precedes the actual parti-tioning step. The goal of the preprocessing step is toeliminate edges that could never be viable cut-points.Consider an operator u that feeds another operator v suchthat the bandwidth from v is the same or higher than thebandwidth on the output stream from u. A partition witha cut-point on the v’s output stream can always be im-proved by moving the cut-point to the stream u → v; thebandwidth does not increase, but the load on the embed-ded node decreases (v moves to the server). Thus, anyoperator that is data-expanding or data-neutral may bemerged with its downstream operator(s) for the purposesof the partitioning algorithm, reducing the search spacewithout eliminating optimal solutions.

4.2 Optimal Partitionings

It is well-known that optimal graph partitioning is NP-complete [8]. Despite the intrinsic difficulty of the prob-lem, the problem proves tractable for the graphs seen inrealistic applications. Our pre-processing heuristic re-duces the problem size enough to allow an ILP solver tosolve it exactly within a few seconds to minutes.

4.2.1 Integer Linear Programming (ILP)

Let G = (V,E) be the directed acyclic graph (DAG) ofstream operators. For all v ∈ V , the compute cost on thenode is given by cv > 0 and the communication (radio)cost is given by ruv for all edges (u, v) ∈ E. One mightthink of the compute cost in units of MHz (megahertzof CPU required to process a sample and keep up withthe sampling rate), and the bandwidth cost in kilobits/sconsumed by the data going over the radio. Adding ad-ditional constraints for RAM usage (assuming static allo-cation) or code storage is straightforward in this formu-lation, but we do not do it here. For each of these costswe can use either mean or peak load (profiling computesboth). Because our applications have predictable rates,we use mean load here. Peak loads might be more appro-priate in applications characterized by “bursty” rates.

The DAG G contains a set of terminal source ver-tices S, and sink vertices T , that have no inward andoutward edges, respectively, and where S, T ⊂ V . Asnoted above, we construct G from the original operatorgraph such that these boundary vertices are pinned—allthe sources must remain on the embedded node; all sinkson the server. Recall that the partitioning problem is tofind a single cut of G that assigns vertices to the nodes

and server. We can think of the graph G as correspondingto the server and a single node, but vertices assigned tothe node partition are instantiated on all physical nodesin the system.

We encode a partitioning using a set of indicator vari-ables fv ∈ {0, 1} for all v in V . If fv = 1, then operatorv resides on the node; otherwise, it resides on the server.The pinning constraints are:

(∀u ∈ S) fu = 1(∀v ∈ T ) fv = 0(∀v) fv ∈ {0, 1} .

(1)

Next, we constrain the sum of node CPU costs to beless than some total budget C.

cpu ≤ C where cpu =∑v∈V

fvcv (2)

A simple expression for the total cut bandwidth is∑(u,v)∈E(fu − fv)2ruv . (Because fv ∈ {0, 1}, the

square evaluates to 1 when the edge (u, v) is cut and to 0if it is not; |fu−fv| gives the same values.) However, weprefer to formulate the integer programming problem asone with a linear rather than quadratic objective function,so that standard ILP techniques can be used.

We can convert the quadratic objective function to alinear one by introducing two variables per edge, euv ande′uv , which are subject to the following constraints:

(∀(u, v) ∈ E) euv ≥ 0(∀(u, v) ∈ E) e′uv ≥ 0(∀(u, v) ∈ E) fu − fv + euv ≥ 0(∀(u, v) ∈ E) fv − fu + e′uv ≥ 0 .

(3)

The intuition here is that when the edge (u, v) is notcut (i.e., u and v are in the same partition), we wouldlike euv and e′uv to both be zero. When u and v are indifferent partitions, we would like a non-zero cost to beassociated with that edge; the constraints above ensurethat the cost is at least 1 unit, because fu− fv is -1 whenu is on the server and v on the embedded node. Theseobservations allow us to formulate the bandwidth of thecut, cap that bandwidth, and define the objective functionin terms of both CPU and network load.

net < N where net =

∑(u,v)∈E

(euv + e′uv)ruv

(4)

objective : min (α cpu + β net) (5)

Any optimal solution of (5) subject to (1), (2), (3), and(4) will have euv +e′uv equal to 1 if the edge is cut and to0 otherwise. Thus, we have shown how to express ourpartitioning problem as an integer programming prob-lem with a linear objective function, 2|E|+ |V | variables

(only |V | of which are explicitly constrained to be inte-gers), and at most 4|E| + |V | + 1 equality or inequalityconstraints.

We could use a standard ILP solver on the formulationdescribed above, but a further improvement is possibleif we restrict the data flow to not cross back and forthbetween node and server, as described in Section 2.1.2.On the positive side, the restriction reduces the size ofthe partitioning problem, which speeds up its solution.

With the above restriction, we can then flip all edgesgoing from server to node for the purpose of partitioning(the communication cost would be the same under ourmodel). With all edges pointed towards the server, andonly one crossing of the network allowed, another set ofconstraints now apply:

(∀(u, v) ∈ E) fu − fv ≥ 0 (6)

With (6) the network load quantity simplifies:

net =

∑(u,v)∈E

(fu − fv)ruv

. (7)

This formulation eliminates the euv and e′uv variables,simplifying the optimization problem. We now haveonly |V | variables and at most |E| + |V | + 1 con-straints. We have chosen this restricted formulationfor our current, prototype implementation, primarily be-cause the per-platform code generators don’t yet supportarbitrary back-and-forth communication between nodeand server. We use an off-the-shelf integer programmingsolver, lp solve3, to minimize (7) subject to (1) and (2).

We note that the restriction of unidirectional data flowdoes preclude cases when sinks are pinned to embed-ded nodes (e.g., actuators or feedback in the signal pro-cessing). It also prevents a good partition when a high-bandwidth stream is merged with a heavily-processedstream. In the latter case, the merging must be doneon the node due to the high-bandwidth stream, but theexpensive processing of the other stream should be per-formed on the server. In our applications so far, we havefound our restriction to be a good compromise betweenprovable optimality and speed of finding a partition.

4.3 Data Rate as a Free Variable

It is possible that the partitioning algorithm will not beable to find a cut that satisfies all of the constraints (i.e.,there may be no way to “fit” the program on the embed-ded nodes.) In this situation we wish to find the maxi-mum data rates for input sources that will support a vi-able partitioning. The algorithm given above cannot di-rectly treat data rate as a free variable. Even if CPU and

3lp solve was developed by Michel Berkelaar, Kjell Eikland,and Peter Notebaert. It uses branch-and-bound to solve integer-constrained problems, like ours, and the Simplex algorithm to solvelinear programming problems.

network load varied linearly with data rate, the resultingoptimization problem would be non-linear. However, itturns out to be inexpensive to perform the search overdata-rates as an outer loop that on each iteration calls thepartitioning algorithm.

This is because in most applications, CPU and net-work load increase monotonically with input data rate. Ifthere is a viable partition when scaling input data rates bya factor X , then any factor Y < X will also have a viablepartitioning. Thus Wishbone simply does a binary searchover data rates to find the maximum rate at which the par-titioning algorithm returns a valid partition. As long aswe are not over-saturating the network such that sendingfewer packets actually result in more data being success-fully received, this maximum sustainable rate will be thebest rate to pick to maximize outputs (throughput) of thedata flow graph. We will re-examine this assumption inSection 7.

5 Wishbone Platform BackendsIn this section, we describe three new WaveScriptcode generators we built for Wishbone, which are de-scribed here for the first time. These support ANSI C,NesC/TinyOS and JavaME.

5.1 Code Generation: ANSI C and JavaME

In contrast with the original WaveScript C++ back-end (and XStream runtime engine), our current Ccode-generator produces simple, single threaded codein which each operator becomes a function definition.Passing data via emit becomes a function call, and thesystem does a depth-first traversal of the stream graph.The generated code requires virtually no runtime and iseasily portable. This C backend is used to execute theserver-side portion of a partitioned program, as well asthe node-side portion on Unix-like embedded platformsthat run C, such as the iPhone (jailbroken), Gumstix, orMeraki.

Generating code for JavaME also straightforward, asJava provides a high level programming environment thatabstracts hardware management. The basic mapping be-tween the languages is the same as in the C backend. Op-erators become functions, and an entire graph traversal isa chain of function calls. Some minor problems arise dueto Java’s limited set of numeric types.

5.2 Code Generation: TinyOS 2.0

Supporting TinyOS 2.0 is much more challenging. Thedifficulties are both due to the extreme resource con-straints of TinyOS motes (typically less than 10 KB ofRAM and 100 KB of ROM), and to the restricted con-currency model of TinyOS (tasks must be be relativelyshort-lived and non blocking; all IO must be performedwith split-phase asynchronous calls). Also, program ob-jects be serialized and split into small network packets.

Wishbone’s support for TinyOS demonstrates its abilityto use platforms with severe resource restrictions and un-usual concurrency models.

Our prototype does not currently support WaveScript’sdynamic memory management in code running onmotes. We may support it in the future, but it remains tobe seen whether this style of programming can be madeeffective for extremely resource constrained devices. In-stead, we enforce that all operators assigned to motes useonly statically allocated storage in our applications.

The most difficult issue in mapping a high-level lan-guage onto TinyOS is handling the TinyOS concurrencymodel. All code executes in either task or interrupt con-text, with only a single, non-preemptive task running at atime. Wishbone simply maps each operator onto a task.Each data element that arrives on a source operator, forexample a sensor sample or an array of samples, will re-sult in a depth-first traversal of the operator graph (exe-cuted as a series of posted tasks). This graph traversalis not re-entrant. Instead, the runtime buffers data at thesource operators until the current graph traversal finishes.

This simple design raises several issues. First, gen-erated TinyOS tasks must be neither too short nor toolong. Tasks with very short durations incur unneces-sary overhead, and tasks that run too long degrade sys-tem performance by starving important system tasks (forexample, sending network messages). Second, the bestmethod for transferring data items between operators isno longer obvious. In the basic C backend, we simplyissue a function call to the downstream operator, wait forit to complete, and then continue computation. We can-not use this method under TinyOS, where it would forceus to perform an entire traversal of the graph in a singlevery long task execution. But the obvious alternative alsopresents problems: executing an operator in its entiretybefore any downstream operators would require a queueto buffer all output elements of the current operator.

The full details of TinyOS code generation are beyondthe scope of this paper. In short, the WaveScript com-piler can convert programs programs into a cooperativemulti-tasking form (via a CPS conversion). This servestwo purposes: every call to emit can serve as a yieldpoint, causing the task to yield to its downstream oper-ator in a depth-first fashion (with no queues), which inturn will re-post the upstream operator upon completingthe traversal. Second, based on profiling data, additionalyield points can be inserted to “split” tasks to adjust gran-ularity for system health.

6 ApplicationsWe evaluate Wishbone in terms of two experimental ap-plications: acoustic speech detection and EEG-basedseizure onset detection. Both of these applications ex-ercise Wishbone’s capability to automatically partition a

Figure 4: Custom audio board attached to a TMote Sky.

single high-level program into components that run overa network containing sensor nodes and a server or “basestation”. Neither of these applications is in itself novel.In both cases we ported existing implementations fromMatlab and C to Wishbone and verified that the resultsmatched the original implementations.

6.1 Application: Seizure Onset Detection

We used Wishbone to implement a patient-specificseizure onset detection algorithm [20]. The applicationwas previously implemented in C++, but by porting itto Wishbone/WaveScript we enabled its embedded/dis-tributed operation, while reducing the amount of code bya factor of four without loss of performance.

The algorithm is designed to be used in a system fordetecting seizures outside a clinical environment. In thisapplication, a user would wear a monitoring cap that typ-ically consists of 16 to 22 channels. Data from the cap isprocessed by a low-power portable device.

The algorithm we employ [21] samples data from 22channels at 256 samples per second. Each sample is 16-bits wide. For each channel, we divide the stream into2 second windows. When a seizure occurs, oscillatorywaves below 20 Hz appear in the EEG signal. To extractthese patterns, the algorithm looks for energy in certainfrequency bands.

To extract the energy information, we first filter eachchannel by using a polyphase wavelet decomposition.We use a repeated filtering structure to perform the de-composition. The filtering structure first extracts theodd and even portions of the signal, passes each signalthrough a 4-tap FIR filter, then adds the two signals to-gether. Depending on the values of the coefficients in thefilter, we either perform a low-pass or high-pass filteringoperation. This structure is cascaded through 7-levels,with the high frequency signals from the last three levelsused to compute the energy in those signals. Note that ateach level, the amount of data is halved.

As a final step, all features from all channels, 66 intotal, are combined into a single vector which is inputinto a patient-specific support vector machine (SVM).The SVM detects whether or not each window containsepileptiform activity. After three consecutive positivewindows have been detected, a seizure is declared.

There are multiple places where Wishbone can par-tition this algorithm. If the entire application fits on theembedded node, then the data stream is reduced to only afeature vector—an enormous data reduction. But data isalso reduced by each stage of processing on each chan-nel, offering many intermediate points which are prof-itable to consider.

6.2 Acoustic Speech Detection

We used Wishbone to build a speech detection applica-tion that uses sampled audio to detect the presence of aperson who is speaking near a sensor. The ultimate goalof such an application would be to perform speaker iden-tification using a distributed network of microphones.For example, such a system could potentially be used tolocate missing children in a museum by their voice, or toimplement various security applications.

However, in our current work we are only concernedwith speech detection, a precursor to the problem ofspeaker identification. In particular, our goal is to reducethe volume of data required to achieve speaker identifi-cation, by eliminating segments of data that probably donot contain speech and by summarizing the speech datathrough feature extraction.

Our implementation of speech detection and data re-duction is based on Mel Frequency Cepstral Coefficients(MFCC), following the approach of prior work in thearea. Recent work by Martin, et al. has shown that clus-tering analysis of MFCCs can be used to implement ro-bust speech detection [14]. Another article by Saasta-moinen, et al. describes an implementation of speakeridentification on smartphones, based on applying learn-ing algorithms to MFCC feature sets [19]. Based on thisprior work, we chose to exercise our system using an im-plementation of MFCC feature extraction.

6.2.1 Mel Frequency Cepstral Coefficients

Mel Frequency Cepstral Coefficients (MFCC) are themost commonly used features in speech recognition al-gorithms. The MFCC feature stream represents a signif-icant data reduction relative to the raw data stream.

To compute MFCCs, we first compute the spectrum ofthe signal, and then summarize it using a bank of over-lapping filters that approximates the resolution of hu-man aural perception. By discarding some of the datathat is less relevant to human perception, the output ofthe filter bank represents a 4X data reduction relativeto the original raw data. We then convert this reduced-

resolution spectrum from a linear to a log spectrum. Us-ing a log spectrum makes it easier to separate convolu-tional components such as the excitation applied to thevocal tract and the impulse response of a reverberant en-vironment, because transforms that are multiplicative ina linear spectrum are additive in a log spectrum.

Finally, we compute the MFCCs as the first 13 coef-ficients of the Discrete Cosine Transform (DCT) of thisreduced log-spectrum. By analyzing the spectrum of aspectrum, the distribution of frequencies can be charac-terized at a variety of scales [6, 5].

6.2.2 Trade-offs in MFCC Extraction

The high level goal of Wishbone is to explore how acomplex application written in a single high level lan-guage can be efficiently and easily distributed acrossa network of devices and support many different plat-forms. As such, the MFCC application presents an in-teresting challenge because for sensors with very limitedresources there appears to be no perfect solution; rather,using Wishbone the application designer can explore dif-ferent trade-offs in application performance.

These trade-offs arise because this algorithm squeezesa resource-limited device between two insoluble prob-lems: not only is the network capacity insufficient to for-ward all the raw data back to a central point, but the CPUresources are also insufficient to extract the MFCCs inreal time. If the application has any partitioning thatfits the resource constraints, then the goal of Wishboneis to select the best partition, for example, lowest cost interms of energy. If the application does not fit at its idealdata rate, ultimately, some data will be dropped on sometarget platforms. The objective in this case is to find apartitioning that minimizes this loss and therefore maxi-mizes the throughput: the amount of input data success-fully processed rather than dropped at the input sourcesor in the network.

6.2.3 Implementing Audio Capture

Some platforms, such as the iPhone and embedded-Linux platforms (such as the Gumstix), provide a com-plete and reliable hardware and software audio capturemechanism. On other platforms, including both TMotesand J2ME phones, capturing audio is more challenging.

On TMotes, we used a custom-built audio board toacquire audio. The board uses an electret microphone,four opamp stages, a programmable-gain amplifier , anda 2.5 V voltage reference. We have found that whenthe microphone was powered directly by the analog sup-ply of the TMote, the audio board performed well whenthe mote was only acquiring audio, but was very noisywhen the mote was communicating. The communi-cation causes a slight modulation of the supply volt-age, which gets amplified into significant noise. Us-

ing a separately regulated supply for the microphone re-moved this noise. The anti-aliasing filter is a simpleRC filter; to better reject aliasing, the TMote samplesat a high rate and applies a digital low-pass filter (fil-tering and decimating a 32 Ks/s stream down to 8 Ks/sworks well). The amplified and filtered audio signalis presented to an ADC pin of the TMote’s microcon-troller, which has 12 bits of resolution. We use TinyOS2.0 ReadStream<uint16 t> interface to the ADC,which uses double buffering to deliver arrays of samplesto the application.

Phones naturally have built-in microphones and mi-crophone amplifiers, but we have nonetheless encoun-tered a number of problems using them as audio sen-sors. Many J2ME phones support the Mobile MediaAPI (JSR-135), which may allow a program to recordaudio, video, and take photographs. Support for JSR-135 does not automatically imply support for audio orvideo recording or for taking snapshots. Even when au-dio recording is supported, the API permits only batchrecording to an array or file (rather than a continuousstream) resulting in gaps.

We ran into a bug on the Nokia N80: after recordingaudio segments for about 20 minutes, the JVM wouldcrash. Other Nokia phones with the same operating sys-tem (Symbian S60 3rd Edition) exhibited the same bug.We worked around this bug using a simple Python scriptthat runs on the phone and accepts requests to record au-dio or take a photograph through a TCP connection, re-turning the captured data also via TCP. The J2ME pro-gram acquires audio by sending a request to this Pythonscript, which can record indefinitely without crashing.

The J2ME partition of the Wishbone program usesTCP to stream partially processed results to the server.When the J2ME connects, the phone asks the user tochoose an IP access point; we normally use a WiFi con-nection, but the user can also choose a cellular IP con-nection. With any of these communication methods, de-pendence on user interaction presents a practical barrierto using phones in an autonomous sensor network. Yetthese software limitations are incidental rather than fun-damental, and should not pose a long-term problem.

7 EvaluationIn this section we evaluate the Wishbone system on theEEG and speech detection applications we discussed inSection 6. We focus on two key questions:

1. Can Wishbone efficiently select the best partitioningfor a real application, across a range of hardwaredevices and data rates?

2. In an overload situation, can Wishbone effectivelypredict the effects of load-shedding and recommenda “good” partitioning?

0

10

20

30

40

50

60

70

80

0 2 4 6 8 10 12 14 16 18 20

Nu

mb

er

of

op

era

tors

in

op

tim

al n

od

e p

art

itio

n

Input data rate as a multiple of 8 kHz

TmoteSky/TinyOSNokiaN80/Java

(a)

0.001

0.01

0.1

1

10

100

1000

10000

source/1 filtbank/7 logs/8 cepstral/9

Ha

nd

led

in

pu

t ra

te a

s m

ultip

le o

f 8

kH

z

Cutpoint / number of operators in node partition

TinyOSJavaMEiPhoneVoxNet

Scheme

(b)

Figure 5: Relationship between partitioning and compute-bound sustainable data rates. On the left (a), a subset of the EEGapplication (one channel). The X axis shows a required data rate, the Y axis the number of operators in computed optimal nodepartition. On the right (b), the speaker detection application; we flip the axes due to the small number of viable cut-points. For eachviable cut-point, we show the maximum data-rate supported on each hardware platform.

7.1 EEG Application

Our EEG application provides an opportunity to explorethe scaling capability of our partitioning method. In par-ticular, we look at our worst case scenario—partitioningall 22-channels (1412 operators). As the CPU budget in-creases, the optimal strategy for bandwidth reduction isto move more channels to the nodes. On our lower-power platforms, not all the channels can be processedon one node. The graph in Figure 5(a) shows partition-ing results only for the first of 22 channels, where wevary the input data rate on the X axis and measure thenumber of operators that “fit” on different platforms. Weran lp solve to derive a partitioning 2100 times, linearlyvarying the data rate to cover everything from “every-thing fits easily” to “nothing fits”. To remove confound-ing factors, the objective function was configured to min-imize network bandwidth subject to not exceeding CPUcapacity (α = 0, β = 1): that is, allow the CPU to befully utilized (but not over-utilized). As we increased thedata rate (moving right), fewer operators can fit withinthe CPU bounds on the node (moving down). The slop-ing lines show that every stage of processing yields datareductions.

The distribution of resulting execution times are de-picted as two CDFs in Figure 6, where the x axis showsexecution time in seconds, on a log scale. The top curvein Figure 6 shows that even for this large graph, lp solve

always found the optimal solution in under 90 seconds.The typical case was much better: 95 percent of the ex-ecutions reached optimality in under 10 seconds. Whilethis shows that an optimal solution is typically discov-ered in a reasonable length of time, that solution is notnecessarily known to be optimal. If the solver is usedto prove optimality, both worst and typical case runtimesbecome much longer, as shown by the lower CDF curve

0

10

20

30

40

50

60

70

80

90

100

0.1 1 10 100 1000

Pe

rce

ntile

Run time in seconds

CDF of runtime required to find optimal solution

Time to discover optimalTime to prove optimal

Figure 6: CDF of the time required for lp solve to reach anoptimal partitioning for the full EEG application (1412 oper-ators), invoked 2100 times with data rates. The higher curveshows the execution time at which an optimal solution wasfound, while the lower curve shows the execution time requiredto prove that the solution is optimal. Execution times are froma 3.2 GHz Intel Xeon.

(yet still under 12 minutes). To address this, we can usean approximate lower bound to establish a terminationcondition based on estimating how close we are to theoptimal solution.

7.2 Speech Detection Application

The speech detection application is a linear pipeline ofonly a dozen operators. Thus the optimization processfor picking a cut point should be trivial—a brute forcetesting of all cut points will suffice. Nevertheless, thisapplication’s simplicity makes it easy to visualize andstudy, and the fact that the data rate it needs to process alldata is unsustainable for TinyOS devices provides an op-

10

100

1000

10000

100000

1e+06

source

preemph

hamm

ing

prefilt

FFTfiltBank

logscepstrals

0

10

20

30

40

50E

xe

cu

tio

n t

ime

of

op

era

tor

(mic

rose

co

nd

s)

Ba

nd

wid

th o

f cu

t (K

Byte

s/S

ec)

Marginal CPU Cost of OperatorCumulative CPU Cost

Bandwidth (Right-hand scale)

Figure 7: Data is reduced by processing, lowering bandwidthrequirements, but increasing CPU requirements.

portunity to examine the other side of Wishbone’s usage:what to do when the application doesn’t fit.

In applying Wishbone to the development process forour speech detection application, we were able to quicklyassess the performance on several different platforms.Figure 7 is a detailed visualization of the performancetrade-offs, showing only the profiling results for TMoteSky (a TinyOS platform). In this figure, the X axis repre-sents the linear pipeline of operators, and the Y axis rep-resent profiling results. Each vertical impulse representsthe number of microseconds of CPU time consumed bythat operator per frame (left scale), while the line repre-sents the number of bytes per second output by that op-erator. It is easy to visualize the trade-off between CPUcost and data rate. Each point on the X-axis represents apotential graph cut, where the sum of the red bars to theleft provides the processing time per frame.

Thus, we see that the MFCC dataflow has multipledata-reducing steps. The algorithm must natively process40 frames per second in real time, or one frame every25 ms. The initial frame is 400 bytes; after applying thefilter bank the frame data is reduced to 128 bytes, using250 ms of processing time; after applying the DCT, theframe data is further reduced to 52 bytes, but using a totalof 2 s of processing time. This structure means that al-though no split point can fit the application on the TMoteat the full rate, we can achieve different CPU/bandwidthtrade-offs by selecting different split points. Selectinga bad partitioning can result in retrieving no data, andthe best “working” partition provides 20 times more datathan the worst. Figure 5(b) shows an axes-flipped ver-sion of Figure 5(a): predicted data-rate as a function ofthe partition point. Only viable (data reducing) cutpointsare shown. Bars falling under the horizontal line indicatethat the platform cannot be expected to keep up with thefull (8 kHz) data rate.

As expected, the TMote is the worst performing plat-form, with the Nokia N80 performing only about twiceas fast—surprisingly poor performance given that the

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

source

preemph

hamm

ing

prefilt

FFTfiltBank

logscepstrals

Fra

ctio

n o

f to

tal C

PU

co

st

MoteN80PC

Figure 8: Normalized cumulative CPU usage for different plat-forms. Relative execution costs of operators vary greatly on thetested systems.

N80 has a 32-bit processor running at 55X the clock rateof the TMote. This is due to the poor performance ofthe JVM implementation. The 412 MHz iPhone plat-form using GCC performed 3X worse than the 400 MHzGumstix-based Linux platform; we believe that this isdue to the frequency scaling of the processing kicking into conserve power.

We can also visualize the relative performance of dif-ferent operators across different platforms. For each plat-form processing the complete operator graph, Figure 8shows the fraction of time consumed by each operator. Ifthe time required for each operator scaled linearly withthe overall speed of the platform, all three lines would beidentical. However, the plot clearly shows that the dif-ferent capabilities of the platforms result in very differ-ent relative operator costs. For example, on the TMote,floating point operations, which are used heavily in thecepstrals operator, are particularly slow. Thisshows that a model that assumes the relative costs of op-erators are the same on all platforms would mis-estimatecosts by over an order of magnitude.

7.3 Wishbone Deployment

To validate the quality of the partitions selected by Wish-bone, we deployed the speech detection application ona testbed of 20 TMote Sky nodes. We also used thisdeployment to validate the specific performance predic-tions that Wishbone makes using profiling data (e.g., ifa combination of operators were predicted to use 15%CPU, did they?).

7.3.1 Network Profiling

The first step in deploying Wishbone is to profile thenetwork topology in the deployment environment. It isimportant to note that simply changing the network sizechanges the available per-node bandwidth and thus re-quires re-profiling of the network and re-partitioning ofthe application. We run a portable WaveScript programthat measures the goodput from each node in the net-work. This tool sends packets from all nodes at an iden-

0

20

40

60

80

100

1 2 3 4 5 6

Perc

ent

Cutpoint index

1 TMote + Basestation: Percent input received1 TMote + Basestation: Percent network msgs received

1 TMote + Basestation: Percent goodput (product)

Figure 9: Loss rate measurements for a single TMote plusbasestation across different partitionings. Lines show the per-centage of input data processed, the percentage of networkmessages received, and the product of these: the goodput.

tical rate, which gradually increases. For our 20 nodetestbed the resulting network profile is typical for TMoteSky devices: each node has a baseline packet drop ratethat stays steady over a range of sending rates, and thenat some drops off dramatically as the network becomesexcessively congested. Our profiling tool takes as input atarget reception rate (e.g. 90%), and returns a maximumsend rate (in msgs/sec and bytes/sec) that the networkcan maintain.For the range of sending rates within thisupper bound the assumption mentioned in 4.3 holds—attempting to send more data does not result in actualbytes of data received. Thus we are free to maximizethe data rate within the upper bound provided by the net-work profiling tool, and thereby maximize total applica-tion throughput. This enables us to use binary search tofind the the maximum sustainable data rate when we arein an overload situation.

To empirically verify that our computed partitions areoptimal, we established a ground truth by exhaustivelyrunning the speech detection application at every cutpoint on our testbed. Figures 9 and 10 show the resultsfor six relevant cutpoints, both for a single node network(testing an individual radio channel) and for the full 20node TMote network. Wishbone counts missed inputevents and dropped network messages on a per-node ba-sis. The relevant performance metric is the percentageof sample data that was fully processed to produce out-put. This is roughly the product of the fraction of dataprocessed at sensor inputs, and the fraction of networkmessages that were successfully received.

Figure 9 shows the input event loss and network lossfor the single TMote case, as well as the resulting good-put. On a single mote, the data rate is so high at earlycutpoints that it drives the network reception rate to zero.At later cutpoints too much computation is done at thenode and the CPU is busy for long periods, missing in-

0

2

4

6

8

10

1 2 3 4 5 6

Perc

ent G

oodput

Cutpoint index

1 TMote + Basestation: Percent goodput20 TMote Network: Percent goodput

Figure 10: Goodput rates for a single TMote and for a networkof 20 TMotes, over different partitionings when running on ourTMote testbed.

put events. In the middle, even a underpowered TMotecan process 10% of sample windows. This is equivalentto polling for human speech four times a second—a rea-sonably useful configuration.

Figure 10 compares the goodput achieved with a sin-gle TMote and basestation to the case of a network of 20TMotes. For the case of a single TMote, peak through-put rate occurs at the 4th cut point (filterbank), while forthe whole TMote network in aggregate, peak throughputoccurs at the 6th and final cut point (cepstral). As ex-pected, the throughput line for the single mote tracks thewhole line closely until cut point six. For a high-datarate application with no in-network aggregation, a manynode network is limited by the same bottleneck as a net-work of only one node: the single link at the root of therouting tree. At the final cut point, the problem becomescompute bound and the aggregate power of the 20 TMotenetwork makes it more potent than the single node.

We also ran the same test on an a Meraki Mini basedon a low-end MIPS processor. While the Meraki has rel-atively little CPU power—only around 15 times that ofthe TMote—it has a WiFi radio interface with at least10x higher bandwidth. Thus for the Meraki the optimalpartitioning falls at cut point 1: send the raw data directlyback to the server.

Having determined the optimal partitioning in ourreal deployment, we can now compare it to the recom-mendation of our partitioning algorithm. Doing this isslightly complex as the algorithm does not model mes-sage loss; instead, it keeps bandwidth usage under theuser-supplied upper bound (using binary search to findthe highest rate at which partitioning is possible), andminimizes the objective function. In the real network,lost packets may cause the actual delivered bandwidthto be somewhat less than expected by the profiler. For-tunately, the cut-point that maximizes throughput shouldbe the same irrespective of loss as CPU and network loadscale linearly with data rate.

In this case, binary search found that the highest datarate for which a partition was possible (respecting net-work and CPU limits) was at 3 input events per second(with each event corresponding to a window of 200 au-dio samples). The optimal partitioning at that data rate4

was in fact cut point 4, right after filterbank, as in theempirical data. Likewise, the computed partitions forthe 20 node TMote network and single node Meraki testmatched their empirical peaks, which gives us some con-fidence in the validity of the model.

In the future, we would like to further refine the preci-sion of our CPU and network cost predictions. To useour ILP formulation we necessarily assume that bothcosts are additive—two operators using 10% CPU willtogether use 20%, and don’t account for operating sys-tem overheads or processor involvement in network com-munication. For example, on the Gumstix ARM-linuxplatform the entire speaker detection application was pre-dicted to use 11.5% CPU based on profiling data. Whenmeasured, the application used 15% CPU. Ideally wewould like to take an automated approach to determin-ing these scaling factors.

8 Related WorkFirst we overview other systems that, like Wishbone,automatically partition programs—either dynamically orstatically—to run on multiple devices. Generally speak-ing, Wishbone differs from these existing systems by us-ing a profile-driven approach to automatically derive apartitioning, as well as its support for diverse platforms.

The Pleiades/Kairos systems [13] statically partitiona centralized C-like program into a collection of node-level nesC programs that run on motes. Pleiades is pri-marily concerned with the correct synchronization ofshared state between nodes, including consistency, seri-alizability, and deadlocks. Wishbone, in contrast, is con-cerned with high-rate shared-nothing data processing ap-plications, where all nodes run the same code. BecauseWishbone programs are composed of a series of dis-crete dataflow operators that repeatedly process stream-ing data, they are amenable to our profile-based approachfor cost estimation. Finally, by constraining ourselvesto a single cut point, we can generate optimal partition-ings quickly, whereas Pleiades uses a heuristic partition-ing approach to generate a number of cut points.

Triage [3] is a related system for “microservers” thatact as gateways in sensor network applications. Triage’sfocus is on power conservation on such servers by using alower-power device to wake a higher-power device basedon a profile of expected power consumption and utilityof data coming in over the sensor network. However,

4In this case with α = 0, β = 1, although the linear combinationin the objective function is not particularly when we are maximizingdata rate we are saturating either CPU or bandwidth

it does not attempt to automatically partition programsacross the two device classes as Wishbone does.

In stream processing there has been substantial worklooking at the problem of migrating operators at run-time [2, 18]. Dynamic partitioning is valuable in environ-ments with variable network bandwidth, unpredictableload, but also comes with serious downsides in termsof runtime overheads. Also, by focusing on static par-titioning, Wishbone is able to provide feedback to usersat compile time about whether their program will “fit”their sensor platform and hardware configuration.

There has been related work in the context of tradi-tional, non-sensor related distributed systems. For ex-ample, the Coign [11] system automatically partitionsbinary applications written using the Microsoft COMframework across several machines, with the goal ofminimizing communication bandwidth. Like Wishbone,it uses a profile-driven approach. Unlike Wishbone,Coign does not formulate partitioning as an optimiza-tion problem, and only targets Windows PCs. Neubauerand Thiemann [15] present a similar framework for parti-tioning client-server programs. Automatic partitioning isalso widely-used in high-performance computing, whereit is usually applied to some underlying mesh, and in au-tomatic layout of circuits. Finally, several systems, in-cluding JESSICA2 [25], MagnetOS [4], and cJVM [1],implement distributed Java virtual machines that appearas a single system. These systems must use runtimemethods to load-balance threads between machines. Theoverheads on communication and synchronization aretypically high, and only applications with a high ratioof computation to communication will scale effectively.

Tenet [9] proposes a two-tiered architecture with pro-grams decomposed across sensors and a centralizedserver, much as in Wishbone. The VanGo system [10],which is related to Tenet, proposes a framework forbuilding high data rate signal processing applications insensor networks, similar to the applications that inspiredour work on Wishbone. But VanGo is constrained to alinear chain of filters, does not support automatic parti-tioning, and runs only TinyOS code.

Marionette [24] and SpatialViews [17] use static par-titioning of programs between sensor nodes and a serverthat is explicitly under the control of the programmer.These systems work by allowing users to invoke pre-defined handlers (written in, for example, nesC) from ahigh-level centralized program that runs on a server, butneither offers automatic partitioning.

Abstract Regions [22] and Hood [23] enable opera-tions over clusters of nodes (or “regions”) rather than sin-gle sensors. They allow data from multiple nodes to becombined and processed, but are targeted at coordinatingsensors rather than stream processing.

9 Future Work/ConclusionsThe model presented in this paper enables communica-tion between embedded endpoints and a central server.But it would be straightforward to extend our modelwith a basic form of in-network aggregation: namely,tree-based aggregation that happens at every node in thenetwork, useful, for example, for taking average sensorreadings. This communication pattern would be exposedas a “reduce” operator that would reside in the logicalnode partition, but would implicitly take its input not justfrom streams within the local node, but from child nodesrouting through it in an aggregation tree. The partition-ing algorithm remains the same. If the reduce operatoris assigned to the embedded node, aggregation happensin-network, otherwise all data is sent to the server.

Also, while our prototype implementation only sup-ports networks of one type of node, the model can alsohandle certain kinds of mixed networks. A single log-ical node partition can take on different physical parti-tions at different nodes. This is accomplished simply byrunning the partitioning algorithm once for each type ofnode. The server would need to be engineered to dealwith receiving results from the network at various stagesof partial processing. In the future, mixed partitions maybe desirable even for homogeneous networks. Varyingwireless link quality can create a situation where eachnode should partitioned differently.

A more radical change would extend the model withmultiple logical partitions corresponding to categories ofdevices. This opens up several design choices; for exam-ple, what communication relationship should the logicalpartitions should have? We have verified that we can usean ILP approach for a restricted three tier network ar-chitecture. (Motes communicate only to microservers,and microservers to the central server.) But going furtherwould require revisiting the partitioning algorithm.

Acknowledgements: This work was supported by theNSF under grants 0520032, 0720079, and 0754662.

References[1] Y. Aridor, M. Factor, and A. Teperman. cjvm: A single

system image of a jvm on a cluster. In Proc. ICPP, 1999.[2] M. Balazinska, H. Balakrishnan, and M. Stonebraker.

Contract-Based Load Management in Federated Dis-tributed Systems. In Proc. USENIX NSDI, San Francisco,CA, Mar. 2004.

[3] N. Banerjee, J. Sorber, M. D. Corner, S. Rollins, andD. Ganesan. Triage: balancing energy and quality of ser-vice in a microserver. In MobiSys, 2007.

[4] R. Barr, J. C. Bicket, D. S. Dantas, B. Du, T. W. D. Kim,B. Zhou, and E. G. Sirer. On the need for system-levelsupport for ad hoc and sensor networks. SIGOPS Oper-ating Systems Review, 36(2), 2002.

[5] R. A. Cole, J. Mariani, H. Uszkoreit, A. Zaenen, andV. Zue. Survey of the state of the art in human languagetechnology, 1995.

[6] S. Davis and P. Mermelstein. Comparison of paramet-ric representations for monosyllabic word recognition incontinuously spoken sentences. IEEE Trans. on ASSP,28:357–366, 1980.

[7] E. B. et al. Zoltan 3.0: Data Management Services forParallel Applications; User’s Guide. Sandia NationalLaboratories, 2007. Tech. Report SAND2007-4749W.

[8] M. Garey, D. Johnson, , and L. Stockmeyer. Some simpli-fied NP complete graph problems. Theoretical ComputerScience, 1:237–267, 1976.

[9] O. Gnawali, K.-Y. Jang, J. Paek, M. Vieira, R. Govindan,B. Greenstein, A. Joki, D. Estrin, and E. Kohler. The tenetarchitecture for tiered sensor networks. In SenSys, 2006.

[10] B. Greenstein, C. Mar, A. Pesterev, S. Farshchi, E. Kohler,J. Judy, and D. Estrin. Capturing high-frequency phenom-ena using a bandwidth-limited sensor network. In SenSys,pages 279–292, 2006.

[11] G. C. Hunt and M. L. Scott. The Coign automatic dis-tributed partitioning system. In Proc. OSDI, 1999.

[12] G. Karypis and V. Kumar. A fast and high quality multi-level scheme for partitioning irregular graphs. SIAM Jour-nal on Scientific Computing, 20:359–392, 1998.

[13] N. Kothari, R. Gummadi, T. Millstein, and R. Govindan.Reliable and efficient programming abstractions for wire-less sensor networks. In Proc. PLDI, 2007.

[14] A. Martin, D. Charlet, and L. Mauuary. Robustspeech/non-speech detection using LDA applied toMFCC. In IEEE Intl. Conference on Acoustics, Speech,and Signal Processing, pages 237–240, 2001.

[15] M. Neubauer and P. Thiemann. From sequential programsto multi-tier applications by program transformation. InProc. PLDI, 2005.

[16] R. R. Newton, L. D. Girod, M. B. Craig, S. R. Madden,and J. G. Morrisett. Design and evaluation of a compilerfor embedded stream programs. In Proc. LCTES, 2008.

[17] Y. Ni, U. Kremer, A. Stere, and L. Iftode. Programmingad-hoc networks of mobile and resource-constrained de-vices. In Proc. PLDI, 2005.

[18] P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos,M. Welsh, and M. Seltzer. Network-aware operator place-ment for stream-processing systems. In ICDE, 2006.

[19] J. Saastamoinen, E. Karpov, V. Hautamki, and P. Frnti.Accuracy of MFCC-Based speaker recognition in series60 device. EURASIP Journal on Applied Signal Process-ing, (17):2816–2827, 2005.

[20] A. Shoeb et al. Patient-Specific Seizure Onset. Epilepsyand Behavior, 5(4):483–498, 2004.

[21] A. Shoeb et al. Detecting Seizure Onset in the Ambula-tory Setting: Demonstrating Feasibility. In IEEE EMBS2005, September 2005.

[22] M. Welsh and G. Mainland. Programming sensor net-works using abstract regions. In NSDI, 2004.

[23] K. Whitehouse, C. Sharp, E. Brewer, and D. Culler.Hood: a neighborhood abstraction for sensor networks.In Proc. Mobisys, 2004.

[24] K. Whitehouse, G. Tolle, J. Taneja, C. Sharp, S. Kim,J. Jeong, J. Hui, P. Dutta, and D. Culler. Marionette: us-ing RPC for interactive development and debugging ofwireless embedded networks. In Proc. IPSN, 2006.

[25] W. Zhu, C.-L. Wang, and F. C. M. Lau. Jessica2: A dis-tributed java virtual machine with transparent thread mi-gration support. In Proc. CLUSTER, page 381, 2002.