InferLine: ML Inference Pipeline Composition FrameworkDaniel Crankshaw, Gur-Eyal Sela, Corey Zumar, Xiangxi Mo

Joseph E. Gonzalez, Ion Stoica, Alexey Tumanov

UC Berkeley RISELab

AbstractThe dominant cost in production machine learningworkloads is not training individual models but serv-ing predictions from increasingly complex predictionpipelines spanning multiple models, machine learningframeworks, and parallel hardware accelerators. Due tothe complex interaction between model configurationsand parallel hardware, prediction pipelines are challeng-ing to provision and costly to execute when serving in-teractive latency-sensitive applications. This challengeis exacerbated by the unpredictable dynamics of burstyworkloads.

In this paper we introduce InferLine, a systemwhich efficiently provisions and executes ML inferencepipelines subject to end-to-end latency constraints byproactively optimizing and reactively controlling per-model configuration in a fine-grained fashion. Unpre-dictable changes in the serving workload are dynami-cally and cost-optimally accommodated with minimalservice level degradation. InferLine introduces (1) auto-mated model profiling and pipeline lineage extraction,(2) a fine-grain, cost-minimizing pipeline configurationplanner, and (3) a fine-grain reactive controller. Infer-Line is able to configure and deploy prediction pipelinesacross a wide range of workload patterns and latencygoals. It outperforms coarse-grained configuration alter-natives by up 7.6x in cost while achieving up to 32xlower SLO miss rate on real workloads and generalizesacross state-of-the-art model serving frameworks.

1 IntroductionThe design of systems for machine learning has largelyfocused on model training. While training is central tomachine learning research, it is only a small part of pro-duction machine learning. The dominant cost of pro-duction machine learning is inference, the process of

InferLineProfiler

Load Generator

Planner

Optimizer

Estimator

Driver.py

SampleQueries

LatencySLO

Offline

Online

LiveQueries Preds.

Model Profiles

Pipeline

ConfigSampleTrace

Physical ExecutionEngine

Latency-Aware Batched Queuing System

GPU

GPUCPU

CPU

Con

tain

er

BiggerGPU

CPU

CPU

Con

tain

er

Reactive Controller

Figure 1: InferLine System Architecture

rendering predictions from trained models. While train-ing systems run periodically on large static datasets, in-ference systems must run continuously and render pre-dictions within tight end-to-end latency budgets and inresponse to stochastic and often bursty query arrivalprocesses. Individual predictions may require giga-flopsof computation and expensive specialized hardware tomeet latency objectives. Indeed, several of the largestindustrial users of machine learning were driven by thecost of inference to develop custom hardware [9, 17].

As machine learning matures into an engineeringdiscipline, individual models are being replaced bycomplex prediction pipelines spanning data transforma-tions, multiple off-the-shelf and custom models, condi-tional logic, and sophisticated post-processing. Predic-tion pipelines offer the opportunity to improve accu-racy [3, 6], increase throughput [16, 33], and simplifymodel development [4, 28]. For example, by decompos-ing complex prediction tasks (e.g., speech translation)into a sequence of simpler prediction tasks (e.g., speechrecognition and text translation), established modelscan be reused to develop new functionality. Moreover,model composition yields the familiar software engi-

arX

iv:1

812.

0177

6v1

[cs

.DC

] 5

Dec

201

8

neering benefits of modularity by amortizing the devel-opment and maintenance cost of models across a widerange of prediction tasks.

While prediction pipelines simplify model develop-ment, they substantially complicate inference. Oftenprediction pipelines will span multiple machine learningframeworks and heterogeneous parallel devices (e.g.,GPUs and CPUs) to render a single prediction. Thelarge and growing number of parallel hardware accel-erators including multi-core processors, multiple GPUgenerations (e.g., K80, V100), and specialized accelera-tors (e.g., TPUs [17]) offer the opportunity to improvethroughput and reduce cost. However, to fully lever-age the parallel hardware, queries must be processedin batches which can adversely affect latency. Alterna-tively, the throughput of each component in a pipelinecan be increased by replicating the component acrossacross multiple parallel devices and distributing queriesamong them. However, introducing additional devicesincreases prediction costs. Systems for serving predic-tion pipelines must navigate a complex trade-off spacespanned by latency, throughput, and monetary costs thatdepend on per-model choices of (a) parallel hardware,(b) per-model batching and (c) number of replicas.

The complexity of navigating this configuration spacegrows exponentially with each model (stage) in thepipeline and combinatorially in the hardware types,per-model batching options, and replication factor. Theneed to meet end-to-end latency and throughput re-quirements fundamentally couples each configurationdecision. For example, increasing the batch size of anupstream model to maximize parallelism may tightenthe latency requirements of a downstream model andchange its optimal hardware configuration. Thus, globalreasoning is required to optimally configure a predictionpipeline. Optimal pipeline configuration can have sig-nificant consequences on end-to-end latency and cost.For example, optimally configuring a two stage com-puter vision pipeline resulted in a 7.6X reduction in cost(Sec. §7.1.1).

The complexity of configuring and serving predic-tions pipelines is exacerbated by the inherently stochas-tic and bursty nature of real-world prediction querystreams. A system that has been aggressively tunedto minimize cost under the assumption of a uniformarrival process can quickly destabilize under realisticbursty workloads resulting in frequent missed latencydeadlines. As a consequence, the conventional wis-dom in production deployments is to over-provision thepipeline. However, over-provisioning specialized hard-ware accelerators can be especially costly. In our eval-

uation, we demonstrate that it is possible to dynami-cally react to observed arrival process burstiness in afine-grain, per-model fashion and maintain high levelsof latency SLO attainment while achieving up to 4.2Xreduction in cost.

To address these challenges we propose InferLine—a high-performance, general purpose system for pro-visioning and serving prediction pipelines. InferLineefficiently provisions ML inference pipelines subjectto end-to-end latency constraints by proactively opti-mizing and reactively controlling per-model configu-ration in a fine-grain fashion. InferLine combines aPlanner to optimize the initial pipeline configurationand a Reactive Controller to continuously moni-tor and tune the pipeline configuration at runtime. ThePlanner relies on the end-to-end latency Estimator

model profiles extracted by the Profiler to find a costminimizing configuration that meets the latency SLO.The prediction pipeline is then served by the physicalexecution engine with a dynamic, SLO-aware, batchedqueuing system. The reactive controller monitors the dy-namic behavior of the arrival process and extends thefine-grain per-model configuration control at runtime torespond to transient variability in demand.

There are two key enabling observations in the designof InferLine. First, the complex performance character-istics of individual models can be accurately profiled us-ing offline training data and combined to estimate end-to-end pipeline performance across hardware and batch-ing configurations. This end-to-end performance esti-mator can then be used to both proactively and reactivelyconfigure the system to minimize cost and meet latencySLOs. Second, that functional nature of ML inferencenaturally supports online reactive scaling through repli-cation. The resulting contributions of InferLine are: (1)fine-grain proactive pipeline optimizer that minimizespipeline cost while satisfying given query process andend-to-end latency constraints cost-efficiently, (2) fine-grain reactive controller algorithm and system artifactthat monitors and adjusts per-model configuration tomaintain high SLO attainment at minimum cost. (3) theSLO- and heterogeneity-aware batched queuing layer inthe physical execution engine.

We evaluate InferLine across a range of real pipelinesusing real models subjected to real query traces span-ning multiple arrival distributions. We compare Infer-Line to the state-of-the-art model serving baselinesthat (a) use coarse-grain proactive configuration of thewhole pipeline as a unit (e.g., TensorFlow Serving [32],an open-source model serving system developed atGoogle), and (b) state-of-the-art coarse-grain reactive

2

mechanisms [10]. Even with the benefit of InferLine’sdeadline-aware, dynamic batching queuing layer, wefind that InferLine significantly outperforms the base-lines by a factor of up to 7.6X on cost, while maintainingthe highest level of latency SLO attainment.

2 Prediction PipelinesPrediction pipelines combine multiple machine learn-ing models and data transformations to support complexprediction tasks [29]. For instance, state-of-the-art vi-sual question answering services [1, 21] combine natu-ral language models for question parsing with computervision models to answer to the question.

Prediction pipelines simplify model development byallowing model developers to reuse models [34] thathave been pre-trained on large well studied bench-mark datasets or developed internally and used acrossthe organization. In many cases, a single model (e.g.,ResNet152 [13]) may be re-used as a feature functionfor a wide range of prediction tasks [15]. The output ofthese feature functions are then used as inputs to morerobust and easier to train models (e.g., linear models).By avoiding the need to go through the costly model de-sign and training process, this form of model reuse cansubstantially accelerate application development.

A prediction pipeline can be formally encoded as adirected acyclic graph (DAG) where each vertex cor-responds to a model (e.g., a mapping from images toa list of objects in the image) or some other more ba-sic data transformation (e.g., extracting key frames froma video) and each edge represents a data flow. In thispaper we study several (Figure 2) common predictionpipelines. The photo analysis pipeline, consists of ba-sic image pre-processing (e.g., cropping and resizing)followed by image classification using a deep neuralnetwork. The video monitoring pipeline was inspiredby [35] and uses an object detection model to identifyvehicles and people and then performs subsequent anal-ysis including vehicle and person identification and li-cense plate extraction on the relevant any images. Thesocial media prediction pipeline translates and catego-rizes posts based on both text and linked images bycombining computer vision models with multiple stagesof language model to identify the source language andtranslate the post if necessary. Finally, the TF Cascadepipeline, combines fast and slow models implementedfully in TensorFlow and invokes the slow model onlywhen the fast model is uncertain about its prediction.

In both the social media and TF cascade pipelines,a subset of models are invoked based on the output ofearlier models in the pipeline. This is common patternappears in bandit algorithms [3, 20] used for model per-

sonalization as well as more general cascaded predictionpipelines [2, 12, 22, 30]. Capturing this conditional exe-cution (corresponding to a sub-graph of the predictionpipeline DAG) is critical when provisioning and execut-ing these pipelines.

2.1 Systems ChallengesPrediction pipelines present categorically new chal-lenges for the design and provisioning of predictionserving systems. In this section we discuss the key chal-lenges around parallel hardware accelerators, pipelinelevel parallelism, the combinatorial configuration space,messaging and queuing delays, and the complexities ofmeeting tight latency service level objectives (SLOs) un-der bursty stochastic query loads.

Parallel Hardware Accelerators. Many machinelearning models can be computationally intensive withsubstantial opportunities for parallelism. A single deepneural network may have many stages that can bene-fit from parallel hardware. In some cases, this paral-lelism can result in orders of magnitude improvementsin throughput and latency. For example, in our experi-ments we found that TensorFlow can render predictionsfor the relatively large ResNet152 neural network at 0.6queries per second (QPS) on a CPU and at 191 QPSon an NVIDIA Tesla V100 GPU (a 300x difference inthroughput). However, not all models benefit equallyfrom hardware accelerators. For example, many widelyused classical models (e.g., decision trees [7]) can bedifficult to parallelize on GPUs.

Determining the Per-Model Batchsize. In manycases, to fully utilize the available parallel hardware,queries must be processed in batches (e.g., ResNet152required a batch size of 32 to maximize throughput onthe V100). However, processing queries in a batch canalso increase latency. Because most hardware accelera-tors operate at vector level parallelism, the first query ina batch is not returned until the last query is completed.As a consequence, it is often necessary to set a maxi-mum batch size to bound query latency. However, thechoice of the maximum batch size depends on the hard-ware and model and can affect the end-to-end latency ofthe pipeline.

Pipeline Operator Level Parallelism. In heavy queryload settings it is often necessary to leverage parallelismat the level of the pipeline to scale-out individual modelsand high-fanout stages. Identifying and scaling the bot-tleneck models is critical to achieving high-throughputat low cost. Pipelines with high-fanout stages (e.g., en-sembles of models) can often be accelerated by di-

3

Preprocess Alexnet

(a) Image Processing

YOLO V3

Vehicle Classifier

(Resnet 50)

OpenALPR

Human Classifier

(Resnet 50)

Car/T

ruck

Human

(b) Video Monitoring

German NMT

Spanish NMT

Image Classification (Resnet 50)

Language Identification

NSFW Detector

(Resnet 152)

(c) Social Media

Resnet 50

Inception V3

(d) TF Cascade

Figure 2: Example pipelines for evaluation. We evaluate InferLine on four prediction pipelines that span a wide range of models,control flow, and input characteristics.

viding compute resources across parallel paths in thepipeline. However, the optimal placement of resourcesdepends heavily on the performance characteristics ofeach model and the optimal choice of batch size. Fur-thermore, as we scale the compositions of models, wequickly discover that individual components scale dif-ferently, an effect that can be amplified by the use ofconditional control flow within a pipeline causing somecomponents to be queried more frequently than others.

Combinatorial Configuration Space. Allocatingparallel hardware resources to a single model presents acomplex model dependent trade-off space between cost,throughput, and latency. This trade-off space growsexponentially with each model in a prediction pipeline.Decisions made about the choice of hardware, batchingparameters, and replication factor at one stage of thepipeline can affect choices at subsequent stages. Forexample, trading latency for increased throughput onone model reduces the latency budget of subsequentmodels in the pipeline and as a consequence the feasiblehardware configurations.

Distributed Messaging and queuing. Because pre-diction pipelines span multiple hardware devices thatrun at different speeds and batch sizes, buffering in theform of queues is needed between stages. However,queuing adds to the latency of the system. Thereforequeuing delays, which depend on the arrival process andsystem configuration, must be considered when provi-sioning the end-to-end pipeline.

Bursty, Stochastic Query Arrival Process. Finally,prediction serving systems must respond to bursty,stochastic query streams while meeting tight latency ob-jectives. At a high-level these stochastic processes canbe characterized by their average arrival rate λ and theircoefficient of variation, a dimensionless measure of dis-persion defined by CV = σ2

µ2 , where σ and µ = 1/λ

are the standard-deviation and mean of the query inter-arrival time. Processes with higher CV have higher vari-ability and often require additional over-provisioningto meet query latency requirements. However, over-provisioning an entire pipeline on specialized hardware

can be prohibitively expensive. Therefore, it is criticalto be able to identify and provision the hotspots in apipeline to accommodate the bursty arrival process. Fi-nally, as the workload changes, we need mechanisms toquickly detect and scale up (or down) replicas of indi-vidual stages in the pipeline.

3 System Design and ArchitectureIn this section, we provide a high-level overview ofthe main system components in InferLine (Fig. 1): theProfiler, the Planner, the Physical Execution

Engine, and the Reactive Controller. TheProfiler and Planner are run offline and used toestimate model performance characteristics and opti-mally provision and configure the system for a givensample workload and latency SLO. The Physical

Execution Engine then provisions model containerresources according to the configuration producedby the Planner and hosts pipeline for serving. TheReactive Controller monitors and detects unex-pected changes in the arrival process and reactivelyadjusts the configuration to meet the latency SLO.

3.1 Pipeline SpecificationInferLine serves prediction pipelines by processing astream of queries submitted by clients while dynami-cally adjusting its configuration to meet the specifiedend-to-end latency constraints while minimizing cost.

Prediction pipelines in InferLine are implemented asa stateless driver program that takes as input a queryand returns a prediction. Within this driver function, de-velopers interleave application-specific code executedin the driver with asynchronous RPC calls to modelshosted in InferLine. This flexible programming modeladdresses the fundamental need for an expressive mech-anism to compose models, control flow, and data trans-formations within a prediction pipeline.

To support user defined logic and flexible model com-position, InferLine allows users to specify their pipelinein a fully featured Python function. However, in orderto support query planning and scale-out model execu-tion on different hardware accelerators, InferLine needs

4

1 2 4 8 16 32 64Batch Size

8163264

128256512

1024Th

roug

hput

(qps

) preprocess res152 tf-nmt

Figure 3: Example Model Profiles. The preprocess modelhas no internal parallelism and cannot utilize a GPU. Thus,it sees no benefit from batching. Res152 (image classification)& TF-NMT(text translation model) benefit from batching on aGPU.a way to capture statistics about the pipeline call graphand move model invocation out of the driver process.

InferLine exposes a Python client API backed by aC++ RPC client that issues queries to the InferLine serv-ing system. To query an individual model within thepipeline, the driver program makes an RPC request toInferLine specifying the model name and providing theinput. The RPC request is then dispatched to one ofthe containers (see §3.6) which serves that model. ThisRPC request is asynchronous and returns a future so thatpipelines can evaluate multiple models in parallel. Fi-nally, the futures are used to extract statistics about therelative frequency of model invocations used for plan-ning.

3.2 ProfilerThe Profiler takes a pipeline driver and and samplequery trace and extracts the logical pipeline structureas a directed acyclic graph (DAG) of data and controlflow dependencies between models. The Profiler alsocreates a performance profile of each of the componentmodels in the pipeline as a function of model batch sizeand hardware.

InferLine uses runtime tracing to infer the pipelinestructure directly from the user-provided driver programby executing a sample set of queries. The profiler eval-uates the driver on each of the sample queries, tracingthe lineage of each query as it traverses the pipelinethrough the RPC system. The union of all the tracedquery lineage graphs forms a single pipeline graph. Wetrack the frequency of a query visiting each model. Thisfrequency represents the conditional probability that amodel will be evaluated given a query entering thepipeline independent of the behavior of any other mod-els, which we refer to as the scale factor s of the model.The scale factor is used by the Estimator to estimatethe effects of conditional control flow on latency (§3.4).

The profiler captures model throughput as a function

of hardware type and maximum batch size to create per-model performance profiles. An individual model con-figuration corresponds to a specific value for each ofthese parameters. Both the batch size and the hardwaretype provide mechanisms for exploiting a model’s par-allelism and offer diminishing returns. The profiler ex-ploits this property in two ways. First, profiling a singlereplica is sufficient, as the models are side-effect freeand scale horizontally. Second, batch sizes are consid-ered in powers of 2 to detect maximum performancewithout the need for linear search.

While we ensure that models have exclusive accessto the CPUs and GPUs allocated to them in their re-source bundles there are still sources of kernel level con-tention (e.g., memory allocation or GPU calls) that canreduce model throughput on a heavily loaded multi-coremachine. To ensure that InferLine’s configurations arefeasible on fully utilized machines, we simulate a loadon the machine by running additional replicas of themodel on the available compute resources during pro-filing, rather than profiling on an idle machine.

3.3 Planner

The Planner is responsible for the initial pipelineconfiguration subject to the end-to-end latency SLOand the specified arrival process. It sets the three con-trol parameters for each model in the pipeline usinga globally-aware, cost-minimizing optimization algo-rithm. The Planner uses the model profiles extractedby the Profiler to select cost-minimizing steps ineach iteration while relying on the Estimator to checkfor latency constraint violations. We cover the algorithmin §4.

3.4 Estimator

The Estimator is responsible for rapidly estimatingthe end-to-end latency of a given pipeline configura-tion for the sample query trace. It takes as input apipeline configuration, the individual model profiles,and a sample trace of the query workload, and returnsaccurate estimates of the latency for each query in thetrace. The Estimator is implemented as a continuous-time, discrete-event simulator [5], simulating the entirepipeline, including queuing delays. The simulator main-tains a global logical clock that is advanced from onediscrete event to the next with each event triggering fu-ture events that are processed in temporal order. Becausethe simulation only models discrete events, we were ableto faithfully simulate hours worth of real-world trace inhundreds of milliseconds.

5

3.5 Reactive ControllerThe Reactive Controller monitors the dynamicbehavior of the arrival process as well as the instan-taneous end-to-end system performance to adjust per-model control parameters and maintain high SLO at-tainment at low cost. More specifically, the Reactive

Controller continuously monitors the current arrivalcurve [19] to detect deviations from the planned arrivalprocess at different timescales simultaneously. By an-alyzing the timescale at which the deviation occurred,the Reactive Controller is able to take appropriatemitigating action to ensure that SLOs are met withoutunnecessarily increasing cost. We discuss the ReactiveController in more detail in §5 and provide an exten-sive analysis of its behavior under different arrival pro-cesses in §7.2.

3.6 Physical Execution EngineSimilar to [8, 14], the Physical Execution Engine

adopts a distributed microservice architecture with anRPC interface, allowing models to dictate their environ-ment dependencies and enabling the serving system tobe distributed across a cluster of heterogeneous hard-ware. Each model replica is hosted in a separate modelcontainer and queried via RPC.

The Physical Execution Engine interposes alatency-aware batched queuing system to tightly controlhow queries are distributed among model replicas. Allrequests to a model are placed in a unique centralizedqueue for all replicas of that model. To ensure that In-ferLine is always processing the queries that will expirefirst, InferLine uses earliest deadline first (EDF) priorityqueues.

When a model replica is ready to process a new batchof inputs, it requests a new batch from the centralizedqueue for that model. The size of the batch is boundedabove by the maximum batch size for the model as con-figured by the Planner. By employing a pull-basedqueuing strategy and imposing a maximum batch size,InferLine places an upper bound on the time that a querywill spend in the model container itself after leaving thequeue. Furthermore, unlike randomized load-balancingthe deterministic centralized queuing policy allows theEstimator to accurately estimate queuing latencies.

4 The Proactive Planner AlgorithmAt a high-level, the Planner is a cost-minimizing al-gorithm that iteratively optimizes the pipeline configu-ration subject to the latency constraints. The Planner

algorithm can be divided into two phases. In the firstphase, it finds a feasible but expensive configuration of

the pipeline that meets the latency constraint to serve asan initial configuration. In the second phase, the plannergreedily modifies the configuration to reduce the costwhile using the Estimator to ensure that each newconfiguration does not violate the latency SLO on thesample query trace. The algorithm has converged whenit can no longer make any cost reducing modificationsto the configuration without violating the latency SLO.

Initialization: To generate the initial configuration,the Planner first greedily minimizes pipeline latencyby configuring each model to use the most expensiveavailable hardware and setting the batch size to one. Ifthe service time of the pipeline under this configurationis greater than the SLO then the latency constraint isinfeasible given the available hardware and the plannerterminates. It then iteratively determines the through-put bottleneck in the pipeline and increases that model’sreplication factor until the model is no longer a bottle-neck. Once the Estimator indicates that the latencyconstraint is met, this stage terminates.

Cost-Minimization: In each iteration of the cost-minimizing process, the Planner considers three can-didate modifications for each model: increase the batchsize, decrease the replication factor, or downgrade thehardware. It evaluates each of these actions on everymodel in the pipeline, eliminating candidate actions thatviolate the latency SLO according to the Estimator.

The batch size is increased by factors of two to matchthe profiled batch sizes. Increasing the batch size onlyaffects throughput and does not affect cost, and willtherefore only be chosen if no other actions are feasible.In contrast, decreasing the replication factor directly re-duces costs by removing container replicas.

The action of downgrading hardware is substantiallymore complex than the other two actions, as the batchsize and replication factor for the model must be re-evaluated to account for the differing batching behaviorof the new hardware. For example, downgrading from aGPU to a CPU eliminates a significant amounts of par-allel computation and so retaining the batch size fromthe GPU can lead to a large increase in latency. It istherefore often necessary to reduce the batch size andincrease in replication factor to find a feasible pipelineconfiguration. However, the reduction in hardware pricecan occasionally compensate for the increased replica-tion factor. For example, in Fig. 9, the steep decreasein cost when moving from an SLO of 0.1 to 0.15 canbe largely attributed to downgrading the resource allo-cation of a language identification model from a GPU toa CPU.

To evaluate a resource allocation downgrade on a

6

dt

queries

dt 2dt 4dt 8dt(a) arrival curve

dt

quer

ies

2dt 4dt 8dt

max Tq

(b) arrival+service

Figure 4: InferLine Reactive Controller tracks thearrival curve (blue) at runtime. X-axis represents ∆t with ex-ponentially increasing look-back time window. Y-axis countsqueries arrived in the past ∆t time window. A well-behavedarrival process doesn’t exceed its provisioned arrival curveat any point. The service curve (red) represents the numberof queries that the system is provisioned to service in ∆t.The maximum horizontal distance between the arrival andservice curves is the maximum expected queuing delay Tq.Reactive Controller heuristic algorithm fits a servicecurve to the observed arrival curve to cap Tq.

model, we first fix the configurations of the other modelsin the pipeline and re-perform the initial configurationgeneration phase. The planner then performs a localizedversion of the overall cost-minimizing algorithm to findthe batch size and replication factor for the model on thenewly downgraded resource allocation needed to reducethe cost of the previous configuration. If there is no costreducing feasible configuration than the hardware down-grade action is rejected.

The Planner terminates when no actions can betaken without violating one of the invariants, and returnsthe latest configuration as the final, cost-minimizingone.

5 Reactive Controller AlgorithmThe reactive controller is responsible for runtime mon-itoring and adjustment of the inference pipeline config-uration to ensure robustness to the unforeseen dynamicsof the arrival process. Recall that the arrival process mayexperience bursts, causing transient overloads, as well asunexpected changes in the mean arrival rate. The reac-tive controller algorithm detects when to take reactiveaction, which model an action should be applied to, andwhat reactive action to take. InferLine’s reactive algo-rithm leverages techniques from network calculus andqueuing theory.

Detecting change. Detecting when to take an actionis essential to responding quickly while ensuring hys-teresis. Recall that the pipeline is provisioned to absorbthe burstiness present in the sample query trace (§4).Naive approaches that react to instantaneous spikes inthe observed arrival rate are too sensitive and wouldrespond to expected variability in the inter-arrival pro-

cess. On the opposite extreme, tracking changes in themoving average may help detect changes in the aver-age arrival rate (λ ), but will miss unexpected increasesin burstiness, which we show are subtle, yet debilitat-ing §7.2.3).

InferLine’s Reactive Controller maintains adiscretized view of the observed arrival curve (Fig. 4).The x-axis marks different time intervals ∆t. The y-axisplots the number of queries arriving in the last ∆t. Wewill refer to this as A(δ ti) As the system starts, it ini-tializes a static reference arrival curve constructed fromthe sample arrival process used by the proactive plan-ner. We will refer to it as Amax(∆t). Intuitively, as longas the observed A(∆ti) does not exceed Amax(∆ti) forany ∆ti, the observed arrival process is within the provi-sioned envelop. Conversely, if A(∆ti) exceed Amax(∆ti)then a reactive action should be taken. The arrival curveis tracked by maintaining the query count arriving in thepast ∆t for an exponentially increasing ∆t. This mech-anism allows us to track changes in the average arrivalrate as A(∆tmax) exceeds Amax(∆tmax) while also detect-ing changes in burstiness even when the arrival ratestays constant. This manifests itself as A(∆ti) exceedsAmax(∆ti) for the first few bucket sizes i.

The choice of the smallest bucket size ∆tmin affectsthe sensitivity of the reactive controller. We draw onintuition from network calculus (Fig. 4(b)) and set thesmallest sized bucket to ∑ T̂ m

s , which is the sum of es-timated service times Ts for all models m on the criticalpath of a query. Ts can be easily monitored at runtime,but ∆tmin must be set statically by the Planner to cal-culate Amax(∆ti). Recall from §4 that the planner has ac-cess to per-model throughput and configures per-modelbatch sizes. Given this information, we derive T m

s = bsmµm

,where bsm is model m’s batch size, and µm is its profiledthroughput. Thus we set ∆tmin =

bsmµm

.

Adding model replicas. Second, the activated reac-tive algorithm decides what reactive action to take. Ittakes corrective action limited to changes in the replica-tion factor per-model. Given A(∆ti), we compute λmax =

maxi

(A(∆ti)

∆ti

), and the new replication factor k′m = λmax

µm.

In addition, at static planning stage, we calculate ρmax =λ

km∗µm, where km is the replication factor for model m

and µm is profiled throughput for model m. This pro-vides maximum model load ρmax at proactive planningstage. We make sure that k′m >= λmax

ρmaxµm.

Removing model replicas. The reactive controllertracks the maximum λ = A(30)

30 over the last 30s,smoothed over a 5 second window. If the reactive con-troller has not detected any change in A(∆ti)∀i in the last

7

10s, it will recalculate k′m.

6 Experimental SetupTo evaluate InferLine we constructed four representa-tive prediction pipelines (Fig. 2) that span the funda-mental model composition patterns. We configure eachpipeline with varying input arrival processes and latencybudgets, and then evaluate the latency SLO attainmentand pipeline cost-effectiveness under a range of bothsynthetic and real world workloads. We characterize thesynthetic workloads by their mean request rate (λ ) andcoefficient of variation (CV), evolving these propertiesover time within the same workload to add additionaldynamicity to the process. We use the same real worldtraces as those used in [10], and compare the InferLineReactive Controller to the reactive control algo-rithm described in that work. We evaluate InferLine onthe four representative pipelines in Fig. 2, using modelstrained in a variety of machine learning frameworks in-cluding PyTorch [25], TensorFlow [31], DarkNet [26],and OpenALPR [24].

6.1 Baseline ComparisonCurrent prediction serving systems do not provide first-class support for prediction pipelines with end-to-end la-tency constraints. Instead, the individual pipeline com-ponents are each deployed as a separate microserviceto a prediction serving system such as [8, 14, 32] anda pipeline is manually constructed by individual calls toeach service. Any performance tuning for end-to-end la-tency or cost treats the entire pipeline as a single black-box service and tunes it as a whole. Throughout the ex-perimental evaluation we refer to this as the Coarse-Grained baseline. We deploy both the InferLine andcoarse-grained pipelines on the Physical Execution

Engine to eliminate any performance variability causedby different underlying execution environments.

We adopt the techniques proposed in [10] for bothprovisioning and scaling the coarse-grained pipelines.We profile the entire pipeline as a single black box toidentify the single maximum batch size capable of meet-ing the SLO. The pipeline is then replicated as a sin-gle unit to achieve the required throughput as measuredon the same sample arrival trace using by the InferLinePlanner. We evaluate two strategies for determiningrequired throughput. CG-Mean uses the mean through-put computed over the arrival trace while CG-Peak de-termines the peak throughput in the trace computed us-ing a sliding window of size equal to the SLO. Thecoarse-grained reactive controller scales the number ofpipeline replicas using the scaling algorithm describedin [10].

6.2 Physical Execution Environment

We ran all experiments in a distributed cluster on Ama-zon EC2. In all experiments, the pipeline driver clientwas deployed on an m4.16xlarge instance which has64 vCPUs, 256 GiB of memory, and 25Gbps networkingacross two NUMA zones. We used large client instancetypes to ensure that network bandwidth from the clientis not a bottleneck in the experiments. The models weredeployed to a cluster of up to 16 p2.8xlarge GPUinstances. This instance type has 8 NVIDIA K80 GPUs,32 vCPUs, 488.0 GiB of memory and 10Gbps network-ing all within a single NUMA node. All instances wererunning Ubuntu 16.04 with Linux Kernel version 4.4.0.

When deploying a pipeline configured by thePlanner, the queuing system was deployed on the sameinstance as the client while the model replicas were ran-domly distributed across the GPU instances. When de-ploying the coarse-grained baselines, an entire pipelinereplica (queues and all model containers) were deployedto the same instance to minimize network overhead.

6.3 Workload Setup

We generated the traces by sampling inter-arrival timesfrom a Gamma distribution with differing mean β tovary the ingest rate, and coefficient of variation CV tovary the workload burstiness. When reporting perfor-mance on a specific workload as characterized by β andCV, a trace for that workload was generated once andreused across all comparison points to provide a moredirect comparison of performance. We generated sepa-rate traces with the same performance characteristics forprofiling and evaluation to avoid overfitting to the sam-ple trace.

To generate synthetic time-varying workloads, weevolve the workload generating function between differ-ent Gamma distributions over a specified period of time,the transition time. This allows us to generate workloadsthat vary in mean throughput, CV, or both, and thus eval-uate the performance of the Reactive Controller

under a wide range of conditions.In Fig. 7 we evaluate InferLine on traces derived from

real workloads studied in [10]. To derive traces fromthese workloads, we re-scaled the max throughput to300 QPS, the maximum throughput supported by thecoarse-grained pipelines on a 16 node cluster. We theniterated through each of the mean throughputs in theworkload and sample from a Gamma distribution withCV 1.0 for 30 seconds. We use the first 15 minutes ofthe trace for planning, and the remaining 45 minutes forevaluation.

8

7 Experimental EvaluationIn this section we evaluate InferLine’s end-to-end per-formance. We start with an all-in validation that afine-grained approach to heterogeneous ML inferencepipeline configuration is needed, both for proactiveplanning and reactive control (§7.1). We then per-form a sensitivity analysis through a series of micro-benchmarks aimed at gauging InferLine’s robustness tothe dynamics of the arrival process (§7.2). We find thatInferLine is robust to unplanned changes in the arrivalrate as well as unexpected inter-arrival bursts. We thenperform an ablation study to show that the system ben-efits from both (a) fine-grain proactive planning and (b)fine-grain reactive control (§7.3). We conclude by show-ing that InferLine generalizes to other prediction servingsystems (§7.4).

7.1 End-to-end EvaluationWe first establish that fine-grain control for InferLine’sproactive (§7.1.1) and reactive (§7.1.2) components out-performs state of the art coarse-grain pipeline-level con-figuration alternatives in an end-to-end evaluation. In-ferLine is able to achieve the same throughput at sig-nificantly lower cost, while maintaining zero or closeto zero latency SLO miss rate. Coarse-grain replicationnecessarily amplifies the cost of any imbalances in thepipeline.

7.1.1 Proactive fine-grain control

In the absence of a globally reasoning planner (§4), theoptions are limited to either (a) provisioning for the peak(CG Peak), or (b) provisioning for the mean (CG Mean).We compare InferLine to these two end-points of theconfiguration continuum across 2 pipelines (Fig. 5 andFig. 6). InferLine meets latency SLOs at the lowest cost.CG Peak meets SLOs, but at much higher cost, particu-larly for burstier workloads. And CG Peak is not pro-visioned to handle bursty arrivals and results in highSLO miss rates. This result is exacerbated by higherburstiness and lower SLO. Furthermore, the InferLineplanner consistently finds lower cost configurations thanboth coarse-grained provisioning strategies and is ableto achieve up to a 7.6x reduction in cost by minimizingpipeline imbalance.

7.1.2 Reactive fine-grain control

InferLine is also able to (1) maintain a negligible SLOmiss rate, and (2) and reduce cost by up to 4.2xwhen compared to the state-of-the-art approach [10]when handling unexpected changes in the arrival rateand burstiness. In Fig. 7 we evaluate the Social Me-dia pipeline on 2 traces derived from real workloads

100 150 200 250 300 (qps)

0

20

Cos

t ($/

hr) CG-Mean

CG-PeakInferLine

100 150 200 250 300 (qps)

0.0

0.5

1.0

SLO

Mis

s Rat

e

(a) SLO 150ms, CV 1.0

100 150 200 250 300 (qps)

0

20

Cos

t ($/

hr)

100 150 200 250 300 (qps)

0.0

0.5

1.0

SLO

Mis

s Rat

e

(b) SLO 150ms, CV 4.0

Figure 5: Proactive Planning for Image Processing.

100 200 300 400 (qps)

0

25

50

Cos

t ($/

hr) CG-Mean

CG-PeakInferLine

100 200 300 400 (qps)

0.0

0.5

1.0

SLO

Mis

s Rat

e

(a) SLO 150ms, CV 1.0

100 200 300 400 (qps)

0

25

50

Cos

t ($/

hr)

100 200 300 400 (qps)

0.0

0.5

1.0

SLO

Mis

s Rat

e

(b) SLO 150ms, CV 4.0

Figure 6: Proactive Planning for Video Monitoring. CG-Peak was not evaluated on λ > 300 because the configurationsexceeded cluster capacity.

studied in [10]. InferLine finds a configuration morethan 5x cheaper than the coarse-grained provision-ing (Fig. 7(a)). Both systems achieve near-zero SLOmiss rates throughout most of the workload, when thebig spike occurs we observe that the InferLine’s reac-tive controller works as described in §5. As soon as thespike dissipates, InferLine scales the pipeline down tomaintain a cost-efficient configuration. In contrast, thecoarse-grained reactive controller operates much slowerand, therefore, is ill-suited for reacting to rapid changesin the request rate of the arrival process.

In Fig. 7(b), InferLine scales up the pipeline smoothlyand recovers rapidly from an instantaneous spike, unlikethe CG baseline. Furthermore, as the workload dropsquickly after 1000s, InferLine rapidly responds by shut-ting down replicas to reduce cluster cost. At the end ofthe workload InferLine and the coarse-grained pipelinesconverge to similar costs due to the low terminal requestrate which hides the effects of pipeline imbalance.

We further evaluate the differences between the In-ferLine and coarse-grained reactive controllers on aset of synthetic workloads with increasing arrival rates

9

0 1000 2000 3000

Time (s)0

100

200

300R

eque

st R

ate

(qps

)

0 1000 2000 3000

Time (s)0

20

40

Cos

t ($/

hr)

0 2000

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

Inferline CG Peak

(a) AutoScale Big Spike

0 1000 2000 3000

Time (s)0

100

200

300

Req

uest

Rat

e (q

ps)

0 1000 2000 3000

Time (s)0

20

40

Cos

t ($/

hr)

0 1000 2000

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

Inferline CG Peak

(b) AutoScale Slowly Varying

Figure 7: Performance comparison of the reactive control algorithms on traces derived from real workloads. These are thesame workloads evaluated in [10] which forms the basis for the coarse-grained baseline.

in Fig. 8. We observe that the arrival curve monitoringdescribed in §5 enables InferLine to detect the increasein arrival rate earlier and therefore scale up the pipelinein time to maintain a low SLO miss rate. In contrast, thecoarse-grained reactive controller only reacts to the in-crease in request rate at the point when the pipeline isoverloaded and therefore reacts when the pipeline is al-ready in an infeasible configuration. The effect of thisdelayed reaction is compounded by the longer provi-sioning time needed replicate an entire pipeline, result-ing in the coarse-grained baselines being unable to re-cover before the experiment ends. They will eventuallyrecover as we see in Fig. 7 but only after suffering a pe-riod of 100% SLO miss rate.

7.2 Sensitivity AnalysisWe evaluate InferLine’s sensitivity and robustness inits two major building blocks: its proactive Planner

and the Reactive Controller. We show that thePlanner is robust to varying arrival rates, latencySLOs, and burstiness factors. For the reactive controller,we analyze InferLine’s sensitivity to changes in the ar-rival process.

7.2.1 Proactive: Optimizer Sensitivity

InferLine’ Planner performs well under varying load,burstiness, and end-to-end latency SLOs. We observethree important trends in Fig. 9. First, increasing bursti-ness (from CV=1 to CV=4) requires more costly con-figurations as the optimizer provisions more capacity toensure that transient bursts do not cause the queues todiverge more than the SLO allows. We see this trendover a range of different arrival throughputs. We also seethe cost gap narrowing between CV=1 and CV=4 as the

SLO increases. As the SLO increases, additional slackin the deadline can absorb more variability in the arrivalprocess. Second, the cost generally decreases as a func-tion of the latency SLO, enabling deadline vs. cost trade-offs at the application level by using InferLine Plannerto quantify this tradeoff. While this downward cost trendgenerally holds, the optimizer occasionally finds sub-optimal configurations, as it makes locally optimal de-cisions to change a resource assignment. Third, the costincreases as a function of expected arrival rate.

7.2.2 Reactive: Arrival Rate Sensitivity

A common type of unpredictable behavior is a change inthe arrival rate. We compare the behavior of the systemwith and without its reactive controller enabled as thearrival process changes from the provisioned 150qps to250qps. We vary the rate of arrival throughput changeτ . InferLine is able to maintain the SLO miss rate closeto zero while matching or beating two alternatives: (a)a proactive-only planner given the oracular knowledgeof the whole arrival trace a priori, and (b) a proactive-only planner that doesn’t respond to change. Indeed, inFig. 10, InferLine continues to meet the SLO, and in-creases the cost of the allocation only for the durationof the unexpected burst. The oracular proactive plannerprovisions the whole pipeline with the knowledge of thebursty behavior and is equipped with ability to tune allthree control knobs (§3.1). The proactive-only plannerwithout oracular knowledge starts missing latency SLOsas soon as the ingest rate increases.

7.2.3 Reactive: Burstiness Sensitivity

A less obvious but potentially debilitating change in thearrival process is an increase in its burstiness, whilemaintaining the same mean arrival rate. This type of ar-

10

0 50 100 150 200

Time (s)0

100

200

Req

uest

Rat

e (q

ps)

0 50 100 150 200 250

Time (s)0

2

4

6

Cos

t ($/

hr)

100 200

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

InferlineCG Mean

CG Peak

(a) Image Processing

0 50 100 150 200

Time (s)0

100

200

Req

uest

Rat

e (q

ps)

0 100 200 300

Time (s)0

20

40

Cos

t ($/

hr)

0 100 200 300

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

InferlineCG Mean

CG Peak

(b) Social Media SLO

Figure 8: Performance comparison of the reactive control algorithms on synthetic traces with increasing arrival rates.

0.1 0.2 0.3SLO (seconds)

0

20

40

Cos

t

= 100.0CV=1.0CV=4.0

0.1 0.2 0.3SLO (seconds)

0

20

40

Cos

t

= 200.0CV=1.0CV=4.0

Figure 9: Planner sensitivity: Variation in configurationcost across different arrival processes and latency SLOs.

0 100 200 300 400 500

150

200

250

Req

uest

Rat

e (q

ps)

0 100 200 300 400 500

Time (s)

10

15

Cos

t ($/

hr)

0 200 400

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

Proactive BaseProactive Oracle

= 3.33

= 10.00= 1.67

Figure 10: Sensitivity to arrival rate changes.

rival process change is also harder to detect, as the com-mon practice is to look at moments of the distribution,such as the mean or 99th percentile latency. In Fig. 11we show that InferLine is able to detect deviation fromexpected arrival burstiness and react to meet the latencySLOs.

7.3 Attribution of BenefitInferLine benefits from (a) fine-grain proactive plan-ning and (b) fine-grain reactive planning. Thus, weevaluate the following comparison points: coarse grainproactive (CG Pro), fine-grain proactive (FG Pro), fine-

0 100 200 300 400100

150

200

Req

uest

Rat

e (q

ps)

0 100 200 300 400

Time (s)

10

15

Cos

t ($/

hr)

0 200 400

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

Proactive BaseProactive OracleReactive

Figure 11: Sensitivity to arrival burstiness changes.

grain proactive with coarse-grain reactive (FG Pro +CG React), and fine-grain proactive with fine-grain re-active (FG Pro + FG React), building up from pipeline-level configuration to the full feature set InferLineprovides. Fine-grained proactive planner reduces thecost of the initial pipeline configuration by more than3x (Fig. 12), but starts missing latency SLOs when therequest rate increases. Adding the reactive controller(FG Pro + CG React) adapts the configuration, but toolate to completely avoid SLO misses. It recovers fasterthan proactive-only options. The fine-grain reactive con-troller meets all SLOs, illustrating the need for both thePlanner for initial cost-efficient pipeline configuration,and the Reactive Controller to promptly and cost-efficiently adapt.

7.4 GeneralityThe contributions of this work extend generalize to dif-ferent physical serving engines. Here we layer the In-ferLine planning and queuing layers on top of Tensor-flow Serving (TFS)—a state-of-the-art model servingframework developed at Google. In this experiment, we

11

0 100 200 300 400

Time (s)0

100

200

Req

uest

Rat

e (q

ps)

0 100 200 300 400 500

Time (s)0

1

2

3

Cos

t ($/

hr)

0 200 400

Time (s)0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

SLO

Mis

s Rat

e

CG ProFG Pro

FG Pro + CG ReactFG Pro + FG React

Figure 12: Attribution of benefit between the InferLine re-active and proactive components.

0 50 100 150 200 250

Throughput (qps)

0

2

4

6

Cos

t ($/

hr)

0 50 100 150 200 250

Throughput (qps)

0.0

0.5

1.0

SLO

Mis

s Rat

e

InferLineTFS-InferLine

(a) SLO 0.15

0 50 100 150 200 250

Throughput (qps)

0

2

4

6

Cos

t ($/

hr)

0 50 100 150 200 250

Throughput (qps)

0.0

0.5

1.0

SLO

Mis

s Rat

e

InferLineTFS-InferLine

(b) SLO 0.3

Figure 13: Comparison of InferLine’s planner running onour serving engine and on TensorFlow-Serving.

achieve the same low latency SLO miss rate as Infer-Line. This indicates the generality of the planning al-gorithms used to configure individual models in Infer-Line. In Fig. 13 we show both the SLO attainment ratesand the cost of pipeline provisioning when running In-ferLine on our in-house physical execution engine andon TFS. The cost of the latter is slightly worse due tothe additional overhead of TFS RPC serialization. TFSuses gRPC [11] for communication which requires anadditional copy of the inputs and outputs of a model,while our serving engine uses an optimized zero-copyRPC implementation.

8 Related WorkA number of recent efforts study the design of genericprediction serving systems [4, 8, 32]. TensorFlow Serv-ing [32] is a commercial grade prediction serving sys-tem primarily designed to support prediction pipelinesimplemented using TensorFlow [31]. Unlike InferLine,TensorFlow Serving adopts a monolithic design withthe pipeline orchestration living within a single pro-cess. Thus, TensorFlow Serving is able to introduceperformance optimizations like operator fusion across

computation stages to reduce coordination betweenthe CPU and GPU at the expense of fine-grain, in-dependent model configuration. Baylor et al. [4] ex-tend TensorFlow-Serving to support the larger machinelearning life-cycle, adding more support for basic datatransformations and multitenancy in the serving system.However, neither TensorFlow-Serving nor [4] supportSLO constraints.

Alternatively, Clipper [8] adopts a more distributeddesign, similar to InferLine. Like InferLine, each modelin Clipper is individually managed, configured, and de-ployed in a separate Docker container. This design im-proves isolation across models at the expense of greaterdata movement. However, Clipper does not directly sup-port prediction pipelines or reasoning about latencydeadlines across model composition. It also does notdynamically scale deployments in response to changingworkloads.

Mirhoseini et al. [23] use RL to automatically opti-mize hardware placement for deep learning in heteroge-neous hardware environments. However, this work fo-cuses on model placement for training, not predictionserving. These techniques could potentially be extendedto prediction serving, but require substantial training.

Zhang et al. [35] explore the design of a stream-ing video processing system in the VideoStorm project.VideoStorm adopts a distributed design with pipelineoperators provisioned across compute nodes and ex-plores the combinatorial search space of hardware andmodel configurations. However, VideoStorm relies onthe pipeline developer to supply their own model con-figuration control parameters while InferLine automati-cally determines resource type and batching.

A large body of prior work leverages profilingto inform better scheduling. Workflow-aware schedul-ing, however, is a relatively recent phenomenon, in-cluding SLURM-integrated HPC scheduler publishedin 2017 [27] and Morpheus [18]. In contrast, Infer-Line heavily exploits the compute-intensive and side-effect free nature of ML models to estimate end-to-endpipeline performance based on individual model pro-files.

Autoscale [10] offers a comprehensive literature re-view of a body of work aimed at automatically scalingthe number of servers reactively, subject to changingload in the context of web services. Autoscale workswell for single model replication without batching. Itassumes bit-at-a-time instead of batch-at-a-time queryprocessing. We borrow the Autoscale state-of-the-artscaling mechanism for configuring the replication fac-tor for coarse-grain reactive baselines. We find that the

12

fine-grained reactive control proposed far outperformsthe coarse-grain reactive pipeline replication based onAutoscale (§7.1.2).

9 ConclusionConfiguring and serving sophisticated predictionpipelines across heterogeneous parallel hardware whileminimizing cost and ensuring SLO attainment presentscategorically new challenges in system design. Inthis paper we introduce InferLine—a system whichefficiently provisions and executes prediction pipelinessubject to end-to-end latency constraints by proactivelyoptimizing and reactively controlling per-model con-figurations. We leverage the predictability performancescaling characteristics of machine learning models toenable accurate end-to-end performance estimation andproactive cost minimizing pipeline configuration. Inaddition, by exploiting the side-effect free, functionalnature of machine learning models we are able to re-actively adapt pipeline configurations to accommodatechanges in the query workload. InferLine builds onthree main contributions: (a) the fine-grain proactiveplanning algorithm for pipeline configuration (§4),(b) the fine-grain, robust reactive algorithm to adaptto unexpected changes in load (§5), (c) the SLO-and heterogeneity-aware batched queuing layer of thephysical execution engine (§3), to achieve the com-bined effect of cost-efficient heterogeneous inferencepipelines that can be deployed to serve applicationswith a range of tight end-to-end latency objectives. Asa result, we achieve up to 7.6x improvement in costand 32x improvement in SLO attainment for the samethroughput and latency objectives over state-of-the-artalternatives.

References[1] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Deep

compositional question answering with neural modulenetworks. CoRR, abs/1511.02799, 2015.

[2] A. Angelova, A. Krizhevsky, V. Vanhoucke, A. S. Ogale,and D. Ferguson. Real-Time Pedestrian Detection withDeep Network Cascades. BMVC, pages 32.1–32.12,2015.

[3] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire.The nonstochastic multiarmed bandit problem. SIAM J.Comput., 32(1):48–77, Jan. 2003.

[4] D. Baylor, E. Breck, H.-T. Cheng, N. Fiedel, C. Y.Foo, Z. Haque, S. Haykal, M. Ispir, V. Jain, L. Koc,C. Y. Koo, L. Lew, C. Mewald, A. N. Modi, N. Poly-zotis, S. Ramesh, S. Roy, S. E. Whang, M. Wicke,J. Wilkiewicz, X. Zhang, and M. Zinkevich. TFX:A tensorflow-based production-scale machine learningplatform. In Proceedings of the 23rd ACM SIGKDDInternational Conference on Knowledge Discovery andData Mining, KDD ’17, pages 1387–1395. ACM, 2017.

[5] A. Beck. Simulation: the practice of model developmentand use. Journal of Simulation, 2(1):67–67, Mar 2008.

[6] L. Breiman. Bagging predictors. Mach. Learn.,24(2):123–140, Aug. 1996.

[7] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J.Stone. Classification and Regression Trees. Statistic-s/Probability Series. Wadsworth Publishing Company,Belmont, California, U.S.A., 1984.

[8] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E.Gonzalez, and I. Stoica. Clipper: A low-latency on-line prediction serving system. In 14th USENIX Sympo-sium on Networked Systems Design and Implementation(NSDI 17), pages 613–627, Boston, MA, 2017. USENIXAssociation.

[9] J. Fowers, K. Ovtcharov, M. Papamichael, T. Massen-gill, M. Liu, D. Lo, S. Alkalay, M. Haselman, L. Adams,M. Ghandi, S. Heil, P. Patel, A. Sapek, G. Weisz,L. Woods, S. Lanka, S. K. Reinhardt, A. M. Caulfield,E. S. Chung, and D. Burger. A configurable cloud-scalednn processor for real-time ai. In Proceedings of the 45thAnnual International Symposium on Computer Architec-ture, ISCA ’18, pages 1–14, Piscataway, NJ, USA, 2018.IEEE Press.

[10] A. Gandhi, M. Harchol-Balter, R. Raghunathan, andM. A. Kozuch. Autoscale: Dynamic, robust capacitymanagement for multi-tier data centers. ACM Trans.Comput. Syst., 30(4):14:1–14:26, Nov. 2012.

[11] GRPC. https://grpc.io/.

[12] J. Guan, Y. Liu, Q. Liu, and J. Peng. Energy-efficientAmortized Inference with Cascaded Deep Classifiers.arXiv.org, Oct. 2017.

13

[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep resid-ual learning for image recognition. arXiv preprintarXiv:1512.03385, 2015.

[14] A. W. S. Inc. Amazon sagemaker: Developer guide.http://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-dg.pdf, 2017.

[15] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Con-volutional architecture for fast feature embedding. InProceedings of the ACM International Conference onMultimedia, pages 675–678. ACM, 2014.

[16] M. J. Jones and P. Viola. Robust real-time object detec-tion. Workshop on Statistical and Computational Theo-ries . . . , 2001.

[17] N. P. Jouppi, C. Young, N. Patil, D. Patterson,G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden,A. Borchers, R. Boyle, P.-l. Cantin, C. Chao, C. Clark,J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V.Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann,C. R. Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz,A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan, D. Kille-brew, A. Koch, N. Kumar, S. Lacy, J. Laudon, J. Law,D. Le, C. Leary, Z. Liu, K. Lucke, A. Lundin, G. MacK-ean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan,R. Narayanaswami, R. Ni, K. Nix, T. Norrie, M. Omer-nick, N. Penukonda, A. Phelps, J. Ross, M. Ross,A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snel-ham, J. Souter, D. Steinberg, A. Swing, M. Tan, G. Thor-son, B. Tian, H. Toma, E. Tuttle, V. Vasudevan, R. Wal-ter, W. Wang, E. Wilcox, and D. H. Yoon. In-datacenterperformance analysis of a tensor processing unit. In Pro-ceedings of the 44th Annual International Symposium onComputer Architecture, ISCA ’17, pages 1–12, 2017.

[18] S. A. Jyothi, C. Curino, I. Menache, S. M. Narayana-murthy, A. Tumanov, J. Yaniv, R. Mavlyutov, I. n. Goiri,S. Krishnan, J. Kulkarni, and S. Rao. Morpheus: Towardsautomated slos for enterprise clusters. In Proceedingsof the 12th USENIX Conference on Operating SystemsDesign and Implementation, OSDI’16, pages 117–134,Berkeley, CA, USA, 2016. USENIX Association.

[19] J.-Y. Le Boudec and P. Thiran. Network Calculus: A The-ory of Deterministic Queuing Systems for the Internet.Springer-Verlag, Berlin, Heidelberg, 2001.

[20] L. Li, W. Chu, J. Langford, and R. E. Schapire. Acontextual-bandit approach to personalized news articlerecommendation. In WWW, 2010.

[21] M. Malinowski, M. Rohrbach, and M. Fritz. Ask yourneurons: A neural-based approach to answering ques-tions about images. 2015 IEEE International Conferenceon Computer Vision (ICCV), pages 1–9, 2015.

[22] M. McGill and P. Perona. Deciding How to Decide: Dy-namic Routing in Artificial Neural Networks. arXiv.org,Mar. 2017.

[23] A. Mirhoseini, H. Pham, Q. Le, M. Norouzi, S. Ben-gio, B. Steiner, Y. Zhou, N. Kumar, R. Larsen, andJ. Dean. Device placement optimization with reinforce-ment learning. 2017.

[24] Open ALPR. https://www.openalpr.com/.

[25] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang,Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, andA. Lerer. Automatic differentiation in pytorch. 2017.

[26] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. Youonly look once: Unified, real-time object detection. InProceedings of the IEEE conference on computer visionand pattern recognition, pages 779–788, 2016.

[27] G. P. Rodrigo, E. Elmroth, P.-O. Östberg, and L. Ramakr-ishnan. Enabling workflow-aware scheduling on hpc sys-tems. In Proceedings of the 26th International Sym-posium on High-Performance Parallel and DistributedComputing, HPDC ’17, pages 3–14, New York, NY,USA, 2017. ACM.

[28] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips,D. Ebner, V. Chaudhary, and M. Young. Machine learn-ing: The high interest credit card of technical debt. InSE4ML: Software Engineering for Machine Learning(NIPS 2014 Workshop), 2014.

[29] E. Sparks. End-to-End Large Scale Machine Learningwith KeystoneML. PhD thesis, EECS Department, Uni-versity of California, Berkeley, Dec 2016.

[30] Y. Sun, X. Wang, and X. Tang. Deep Convolutional Net-work Cascade for Facial Point Detection. CVPR, pages3476–3483, 2013.

[31] TensorFlow. https://www.tensorflow.org.

[32] TensorFlow Serving. https://tensorflow.github.io/serving.

[33] X. Wang, Y. Luo, D. Crankshaw, A. Tumanov, and J. E.Gonzalez. IDK cascades: Fast deep learning by learningnot to overthink. CoRR, abs/1706.00885, 2017.

[34] Y. Yang, D.-C. Zhan, Y. Fan, Y. Jiang, and Z.-H. Zhou.Deep learning for fixed model reuse, 2017.

[35] H. Zhang, G. Ananthanarayanan, P. Bodik, M. Philipose,P. Bahl, and M. J. Freedman. Live video analytics atscale with approximation and delay-tolerance. In 14thUSENIX Symposium on Networked Systems Design andImplementation (NSDI 17), pages 377–392, Boston, MA,2017. USENIX Association.

14