Model-driven QoS Provisioning for DistributedReal-time and Embedded Systems

Jaiganesh Balasubramanian†, Sumant Tambe†, Balakrishnan Dasarathy‡, Aniruddha Gokhale†,Douglas C. Schmidt†, and Shrirang Gadgil‡

†Department of EECS, Vanderbilt University, Nashville, TN, USA‡Telcordia Technologies, Piscataway, NJ, USA

Abstract— Distributed real-time and embedded (DRE)systems are composed of applications with diverse CPUutilization requirements. These applications participate inmany end-to-end application flows with diverse networkbandwidth requirements. Prior research has advanced thestate-of-the-art on (1) bin-packing algorithms, such asfirst fit decreasing (FFD), to allocate per-application CPUresources, and (2) network-layer quality-of-service (QoS)mechanisms, such as differentiated services (DiffServ), tomanage per-flow network resources. Relatively little work,however, has focused on how applications can be deployedand configured to leverage such advances for addressingtheir end-to-end QoS requirements.

This paper provides two contributions to the studyof middleware that supports QoS-aware deployment andconfiguration of applications in DRE systems. First, wedescribe how our NetQoPE model-driven component mid-dleware framework shields applications from the complex-ities of lower-level CPU and network QoS mechanismsto simplify (1) the specification of per-application CPUand per-flow network QoS requirements, (2) resourceallocation and validation decisions (such as admissioncontrol), and (3) the enforcement of per-flow networkQoS at runtime. Second, we empirically evaluate how wellNetQoPE provides QoS assurance for applications in DREsystems. Our results demonstrate that NetQoPE providesflexible QoS configuration and provisioning capabilities byleveraging CPU and network QoS mechanisms withoutmodifying application source code.

I. INTRODUCTION

Emerging trends. Component middleware, suchas CORBA Component Model (CCM), J2EE, and

This work is supported in part or whole by DARPA Adaptive andReflective Middleware Systems Program Contract NBCH-C-03-0132.Any opinions, findings, conclusions or recommendations expressedin this material are those of the author(s) and do not necessarilyreflect the views of the Department of the Interior, National BusinessCenter, Acquisition Services Division, Southwest Branch or DARPA.An earlier version of this paper appeared at the Proceedings of the14th IEEE Real-time and Embedded Technology and ApplicationsSymposium (RTAS’2008) [1]

.NET, is increasingly being used to develop anddeploy next-generation distributed real-time and em-bedded (DRE) systems, such as shipboard com-puting environments [2], inventory tracking sys-tems [3], avionics mission computing systems [4],and intelligence, surveillance and reconnaissancesystems [5]. These systems consist of applicationsthat participate in different end-to-end applicationflows and operate in dynamic environments withvarying levels of CPU, network connectivity, andbandwidth availability. CPU and network resourcesin such target environments must be configured sothat DRE applications can have their requirementssatisfied end-to-end.

Network quality of service (QoS) mechanisms,such as integrated services (IntServ) [6] and dif-ferentiated services (DiffServ) [7], support a rangeof network service levels for applications in DREsystems. To leverage the services of these QoSmechanisms, however, applications have conven-tionally used relatively low-level APIs provided bythe switching elements.

Moreover, to ensure end-to-end QoS, applicationsmust be deployed in appropriate end hosts so thatrequired CPU and network resources for the appli-cation flow between those hosts are provided. Toconfigure required CPU resources for applications,prior work has focused on resource allocation algo-rithms [8], [9] that satisfy timing requirements ofapplications in a DRE system. After the requiredresources are provisioned and the networking ele-ments are configured correctly, applications invokeremote operations by adding a service level-specificidentifier (e.g., DiffServ codepoint (DSCP)) to theIP packets. DiffServ-enabled network routers parsethe IP packets and provide the appropriate servicelevel-specific packet forwarding behavior.

2

Addressing limitations of current approaches.Although the network and CPU QoS mechanismsdescribed above are powerful, it is tedious and error-prone to develop applications that interact directlywith low-level QoS mechanism APIs written imper-atively in third-generation languages, such as C++or Java. For example, applications must make mul-tiple invocations on network QoS mechanisms toaccomplish key network QoS activities, such as QoSmapping, admission control, and packet marking.To address part of this problem, middleware-basednetwork QoS provisioning solutions [10], [11], [12],[13] have been developed that allow applicationsto specify their coordinates (source and destinationIP and port addresses) and per-flow network QoSrequirements via higher-level frameworks. The mid-dleware frameworks—rather than the applications—are thus responsible for converting high-level spec-ifications of QoS intent into low-level network QoSmechanism APIs.

Although middleware frameworks alleviate manyaccidental complexities of low-level network QoSmechanism APIs, they can still be hard to evolveand extend. In particular, application source codechanges may be necessary whenever changes oc-cur to the deployment contexts (e.g., source anddestination nodes of applications), per-flow require-ments, IP packet identifiers, or middleware APIs.Moreover, applications must explicitly determinethe optimal source and destination nodes before theycan obtain network performance assurances via theunderlying network QoS mechanisms.

To address the limitations with current ap-proaches described above, therefore, what is neededare higher-level integrated CPU and network QoSprovisioning technologies that can completely de-couple application source code from the variabil-ities (e.g., different source and destination nodedeployments, different QoS requirement specifica-tions) associated with their QoS provisioning needs.This decoupling enhances application reuse acrossa wider range of deployment contexts (e.g., differ-ent instance deployments each with different QoSrequirements), thereby increasing deployment flexi-bility.

Solution approach → A model-driven de-ployment and configuration middleware frame-work called Network QoS Provisioning Engine(NetQoPE) that integrates CPU and network QoSprovisioning via declarative domain-specific mod-

eling languages (DSML) [14] to raise the levelof abstraction of DRE system design higher thanusing imperative third-generation programming lan-guages. NetQoPE allows system engineers and soft-ware developers to perform deployment-time anal-ysis (such as schedulability analysis [15]) of non-functional system properties (such as network QoSassurances for end-to-end application flows). Theresult is enhanced deployment-time assurance thatapplication QoS requirements will be satisfied.

NetQoPE deploys and configures componentmiddleware-based applications in DRE systems andenforces their network and CPU QoS requirementsusing the four-stage (i.e., design-, pre-deployment-, deployment-, and run-time) approach shown inFigure 1. The innovative elements of NetQoPE’s

1: NetQoPE’s Four-stage Architecture

four-stage architecture include the following:• The Network QoS Specification Language

(NetQoS), which is a DSML that supports design-time specification of per-application CPU resourcerequirements, as well as per-flow network QoSrequirements, such as bandwidth and delay across aflow. By allowing application developers to focus onfunctionality—rather than the different deploymentcontexts (e.g., different CPU, bandwidth, and delayrequirements) where they will be used—NetQoSsimplifies the deployment of applications in contextsthat have different CPU and network QoS needs,e.g., different bandwidth requirements.• The Network Resource Allocation Frame-

work (NetRAF), which is a middleware-based re-source allocator framework that uses the networkQoS requirements captured by NetQoS as input atpre-deployment time to help guide QoS provision-ing requests on the underlying network QoS mecha-

3

nism at deployment time. By providing application-transparent, per-flow resource allocation capabil-ities at pre-deployment-time using a BandwidthBroker [16], NetRAF provides network bandwidthguarantees for application flows.

• The Network QoS Configurator (NetCON),which is a middleware-based network QoS con-figurator that provides deployment-time configura-tion of component middleware containers. NetCONadds flow-specific identifiers (e.g., DSCPs) to IPpackets at runtime when applications invoke remoteoperations. By providing container-mediated andapplication-transparent capabilities to enforce run-time network QoS, NetCON allows DRE systemsto leverage the QoS services of configured routerswithout modifying application source code.

Figure 1 shows how the output of each stage inNetQoPE serves as input for the next stage, whichhelps automate the deployment and configuration ofDRE applications with network QoS support.

Paper organization. The remainder of the paperis organized as follows: Section II describes a casestudy that motivates common requirements associ-ated with provisioning network QoS for DRE sys-tems; Section III explains how NetQoPE addressesthose requirements via its model-driven componentmiddleware framework; Section IV evaluates thecapabilities provided by NetQoPE; Section V com-pares our work on NetQoPE with related research;and Section VI presents concluding remarks andlessons learned.

II. MOTIVATING NETQOPE’S QOSPROVISIONING CAPABILITIES

To demonstrate and evaluate NetQoPE’s model-driven, middleware-guided network and CPU QoSprovisioning capabilities this section presents a casestudy of a representative DRE system in an officeenterprise security and hazard sensing environment,as which is shown in Figure 2. Enterprises of-ten transport network traffic using an IP networkover high-speed Ethernet. Network traffic in anenterprise can be grouped into several classes, in-cluding (1) e-mail, videoconferencing, and normalbusiness traffic, and (2) sensory and imagery traf-fic of the safety/security hardware (such as fire/-smoke sensors) installed on office premises. Ourcase study assumes that safety/security traffic ismore critical than other traffic, and thus focuses on

2: Network Configuration in an Enterprise Securityand Hazard Sensing Environment

how NetQoPE’s model-driven, middleware-guidedmechanisms help ensure the specified QoS for thistype of traffic in the presence of other traffic thatshares the same network.

Our case study uses software controllers to man-age hardware devices, such as sensors and monitors,shown in Figure 2. Each sensor/camera softwarecontroller filters the sensory/imagery informationand relays them to the monitor software controllersthat display the information. Modern office enter-prises deploy many sensors and monitors, whichmay have different CPU utilization requirements.Moreover, communication between sensors mayhave various network QoS requirements, such asdifferent network bandwidth requirements, e.g., 20Mbps vs 5 Mbps.

The software controllers in our case study weredeveloped using Lightweight CCM (LwCCM) [17],which is described in Sidebar 1. Moreover, the traf-fic between these software controllers uses a Band-width Broker [16] to manage network resourcesusing DiffServ network QoS mechanisms. Althoughthe case study in this paper focuses on LwCCMand DiffServ, NetQoPE is designed for use withother network QoS mechanisms (e.g., IntServ) andcomponent middleware technologies (e.g., J2EE).Component-based applications in our case studyuse Bandwidth Broker services via the followingmiddleware-guided steps: (1) network QoS require-ments are specified on each application flow, alongwith information on the source/destination IP/portaddresses that were determined by bin packing al-gorithms, (2) the Bandwidth Broker is invoked to re-serve network resources along the network paths foreach application flow, configure the corresponding

4

Sidebar 1: Overview of LightweightCORBA Component Model

Our case study is based on Lightweight CCM(LwCCM). Application functionality in LwCCM is pro-vided through components that collaborate with othercomponents via ports to create component assemblies.There are four types of CCM ports: facet, receptacle,event source, and event sink. CCM components provideservices to other components by either synchronouscommunication in the form of operations on facet/-receptable ports or asynchronous communication in theform of eventtypes exchanged between event source/sinkports.

Assemblies in LwCCM are described using XMLdescriptors (mainly the deployment plan descriptor) de-fined by the OMG D&C [18] specification. The de-ployment plan includes details about the components,their implementations, and their connections with othercomponents. The deployment plan also has a placeholderconfigProperty that is associated with elements (e.g.,components, connections) to specify their properties(e.g., priorities) and resource requirements. Componentsare hosted in containers that provide the runtime op-erating environment (e.g., load balancing, security, andevent notification) for components to invoke remoteoperations.

network routers, and obtain per-flow DSCP valuesto help enforce network QoS, and (3) remote op-erations are invoked with appropriate DSCP valuesadded to the IP packets so that configured routerscan provide per-flow differentiated performance.Section III describes the challenges we encounteredwhen implementing these steps in the context ofour case study and shows how NetQoPE’s four-stage architecture in Figure 1 helps resolve thesechallenges.

III. NETQOPE’S MULTISTAGE NETWORK QOSPROVISIONING ARCHITECTURE

As discussed in Section I, conventional tech-niques for CPU allocation and providing networkQoS to applications incur several limitations, includ-ing modifying application source code to specifydeployment context-specific network QoS require-ments and integrate functionality from network QoSmechanisms at runtime. This section describes howNetQoPE addresses these limitations via its model-driven, middleware-guided network QoS provision-ing architecture.

A. Challenge 1: Alleviating Complexities in CPUand Network QoS Requirements Specification

Context.: Every component’s CPU utilizationrequirements guide the selection of the physicalnode to which it is deployed. After identifying thenodes on which to deploy the components, eachapplication flow in a DRE system can specify arequired level of service (e.g., high priority vs.low priority), the source and destination IP andport addresses, and ingress and egress bandwidthrequirements. NetQoPE uses this information toconfigure network resources between two endpointnodes to provide the required QoS.Problem. Multiple feasible CPU allocations arepossible with a given set of component CPU re-quirements. For example, a component could bedeployed on any node that has the capacity availableto satisfy the component’s CPU utilization require-ment. Depending on the node chosen to deploy thecomponent, the deployment of other components inthe system can also change. Manually enumeratingall these possible CPU allocation permutations ishard.

Network QoS requirements can affect CPU al-locations. For example, a deployment plan for acomponent with multiple CPU allocations (e.g.,component A could be deployed on either node X ornode Y) must ensure the deployment also satisfiesnetwork QoS requirements. Source and destinationIP address are therefore needed to specify networkQoS requirements properly.

Network QoS requirements can also change de-pending on the deployed context. For example,in our case study from Section II, multiple firesensors are deployed at different importance levelsand each sensor sends its sensory information to itscorresponding monitors. Fire sensors deployed inthe parking lot have a lower importance than thosein the server room. The sensor to monitor flowsthus have different network QoS requirements, eventhough the reusable software controllers managingthe fire sensor and the monitor have the samefunctionality.

Conventional techniques, such as hard-coded APIapproaches [13], require application source codemodifications for each context. Writing this codemanually to specify both CPU and network QoS re-quirements is tedious, error-prone, and non-scalable.In particular, it is hard to envision at development

5

time all the contexts in which source code will bedeployed.Solution approach → Model-driven declarativeCPU and network requirements specification.NetQoPE provides a DSML called the Network QoSSpecification Language (NetQoS), which is builtusing Generic Modeling Environment (GME) [19]and the Platform Independent Component ModelingLanguage (PICML) [20]. DRE system developerscan use NetQoS to (1) model application elements,such as interfaces, components, connections, andcomponent assemblies, (2) specify CPU utilizationof components, and (3) specify the network QoSclasses, such as HIGH PRIORITY (HP), HIGH RELI-ABILITY (HR), MULTIMEDIA (MM), and BEST EF-FORT (BE), bi-directional bandwidth requirementson the modeled application elements. NetQoS’snetwork QoS classes correspond to the DiffServlevels of service provided by our Bandwidth Bro-ker [21].1 For example, the HP class represents thehighest importance and lowest latency traffic (e.g.,fire detection reporting in the server room) whereasthe HR class represents traffic with low drop rate(e.g., surveillance data).

In LwCCM, components communicate usingports (described in Sidebar 1) that provideapplication-level communication endpoints. NetQoSprovides capabilities to annotate communicationports with the network QoS requirement capabilitiesdescribed above. In certain application flows (e.g.,a monitor requesting location coordinates from afire sensor in our case study) clients control thenetwork priorities at which requests/replies are sent.In other application flows (e.g., a temperature sensorsends temperature sensory information to monitors),servers control the reception and processing of clientrequests. To support both models of communication(i.e., whether clients vs. servers control networkQoS for a flow), NetQoS supports annotating eachbi-directional flow using either:

• The CLIENT_PROPAGATED network prioritymodel, which allows clients to request real-time network QoS assurance even in the pres-ence of network congestion or

• The SERVER_DECLARED network prioritymodel, which allows servers to dictate the ser-vice that they wish to provide to the clients to

1NetQoS’s DSML capabilities can also be extended to providerequirements specification conforming to other network QoS mech-anisms, such as IntServ.

prevent clients from wasting network resourceson non-critical communication.

Defining network QoS specifications in sourcecode or through NetQoS is a human-intensive pro-cess. Errors in these specifications may remainundetected until later stages of development, such asdeployment and runtime, when they are much morecostly to identify and fix. To identify common errorsin network QoS requirement specification early inthe development phase, NetQoS uses built-in con-straints specified via the OMG Object ConstraintLanguage (OCL) that check the application modelannotated with network priority models.

For example, NetQoS detects and flags specifi-cation errors, such as negative or zero bandwidth.It also enforces the semantics of network prioritymodels via syntactic constraints in its DSML. Forexample, the CLIENT_PROPAGATED model can beassociated with ports in the client role only (e.g., re-quired interfaces), whereas the SERVER_DECLARED

model can be associated with ports in the serverrole only (e.g., provided interfaces). Figure 3 showsother examples of network priority models supportsby NetQoS.

3: Network QoS Models Supported by NetQoSA server using the SERVER_DECLARED network

priority model can also dictate that the total ingressbandwidth from all communicating clients cannotexceed a designated network bandwidth (e.g., 30Mbps). NetQoS checks the aggregation of egressbandwidth requested using all clients that commu-nicate with the server and raise an error if the totalexceeds the preferred total bandwidth. Without thiscapability, applications could fail at runtime whereclients invoke remote operations on servers afterreserving more network bandwidth than the server’sreply will use, which wastes available networkbandwidth that could be used by other applicationflows. NetQoS provides this capability so applica-tion deployers can provision the underlying network

6

QoS mechanisms efficiently and flexibly.After a model has been created and checked for

type violations using built-in constraints, networkresources must be allocated, which requires iden-tifying component source and destination nodes.NetQoS allows the specification of CPU utilizationrequirements of each component and also the tar-get environment where components are deployed.NetQoS’s model interpreter traverses CPU require-ments of each application component and generatesa set of feasible deployment plans (described inSidebar 1) using CPU allocation algorithms, suchas first fit, best fit, and worst fit, as well as max anddecreasing variants of these algorithms. NetQoS canbe used to choose the desired CPU allocation algo-rithm and to generate the appropriate deploymentplans automatically, thereby shielding developersfrom tedious and error-prone manual component-to-node placements.

To perform network resource allocations (de-scribed in Section III-B), NetQoS’s model inter-preter captures the details about the components,their deployment locations (determined by the CPUallocation algorithms), and the network QoS re-quirements for each application flow they are partof the deployment plan configProperty tags (seeSidebar 1). Section III-B describes how a laterstage in NetQoPE allocates network resources basedon requirements specified in the deployment plandescriptor.Application to the case study. Figure 4 showsa NetQoS model that highlights many of the ca-pabilities described above. In this model, multiple

4: Applying NetQoS Capabilities to the Case Studyinstances of the same reusable application compo-nents (e.g., FireSensorParking and FireSensorServercomponents) are annotated with different QoS at-tributes using an intuitive drag and drop technique.

Specifying QoS requirements via NetQoS is muchsimpler than modifying application code for eachdeployment context, as shown in Section IV-C.

Our case study also has scores of applicationflows with different client- and server-dictated net-work QoS specifications, which are modeled usingCLIENT_PROPAGATED and SERVER_DECLARED

network priority models, respectively. The well-formedness of these specifications are checked usingNetQoS’s built-in constraints. In addition, the sameQoS attribute (e.g., HR_1000 in Figure 4) can bereused across multiple connections, which increasesthe scalability of expressing requirements for thenumber of connections prevalent in large-scale DREsystems, such as our enterprise office environmentcase study.

B. Challenge 2: Alleviating Complexities in Net-work Resource Allocation and Configuration

Context. After deciding where to deploy compo-nents on source and destination nodes, DRE systemsmust communicate with a network QoS mechanismAPI (e.g., Bandwidth Broker for DiffServ networks)to allocate and configure network resources basedon the network QoS requirements specified on theapplication flows.Problem. It is often undesirable to tightly coupleapplication components (e.g., the temperature sensorsoftware controller code in our case study) with anetwork QoS mechanism API. This coupling com-plicates deploying the same application componentin a different context (e.g., the temperate sensorsoftware controllers for sensing the temperature atthe server room and the conference room) withdifferent network QoS requirements. Manually pro-gram application components to handle all possiblecombinations of network resources is tedious anderror-prone.

Moreover, network QoS mechanism APIs thatallocate network resources require IP addresses forhosts where the resources are allocated. Compo-nents that require network QoS must therefore knowthe placement of the components with which theycommunicate. This component deployment infor-mation may unknown at development time sincedeployments are often not finalized until CPU place-ment algorithms decide them. Maintaining suchdeployment information at the source code levelor querying it at runtime is unnecessarily complex.

7

Ideally, network resources should be allocated with-out modifying application source code and shouldhandle difficulties associated with specifying appli-cation source and destination nodes, which couldvary depending on the deployment context.Solution approach → Middleware-based Re-source Allocator Framework. NetQoPE’s Net-work Resource Allocator Framework (NetRAF) isa resource allocator engine that allocates networkresources for DRE systems using DiffServ networkQoS mechanisms. As shown in Figure 5, input toNetRAF is the set of feasible deployment plansgenerated by NetQoS model interpreter, which alsoembeds per-flow network QoS requirements.

5: NetRAF’s Network Resource Allocation Capa-bilities

The modeled deployment context could havemany instances of the same reusable source code,e.g., the temperature sensor software controllercould be instantiated two times: one for the serverroom and one for the conference room. When usingNetQoS, however, application developers annotateonly the connection between the instance at theserver room and the monitor software controller.Since NetRAF operates on the deployment planthat captures this modeling effort, network QoSmechanisms are used only for the connection onwhich QoS attributes are added. NetRAF thus im-proves conventional approaches [11] that modifyapplication source code to work with network QoSmechanisms, which become complex when sourcecode is reused in a wide range of deploymentcontexts.

NetRAF’s Network Resource Allocator Manageraccepts application QoS requests at pre-deployment-time. It processes these requests in conjunctionwith a DiffServ Allocator, using deployment specificinformation (e.g., source and destination nodes) ofcomponents and per-flow network QoS require-ments embedded in the deployment plan created by

NetQoS. This capability shields applications frominteracting directly with complex APIs of networkQoS mechanisms, thereby enhancing the flexibil-ity NetQoPE for range of deployment contexts.Moreover, since NetRAF provides the capabilityto request network resource allocations on behalfof components, developers need not write sourcecode to request network resource allocations forall applications flows, which simplifies the creationand evolution of application logic, as shown inSection IV-C.

While interacting with network QoS mechanismspecific allocators (e.g., a Bandwidth Broker), Ne-tRAF’s Network Resource Allocator Manager mayneed to handle exceptional conditions, such as fail-ures in resource allocation. Failures during alloca-tion may occur due to insufficient network resourcesbetween the source and destination nodes hostingthe components. Although NetQoS checks the well-formedness of network requirement specifications atapplication level, it cannot identify every situationthat may lead to failures during actual resourceallocation.

To handle failure scenarios gracefullly, NetRAFprovides hints to regenerate CPU placements forcomponents using the CPU allocation algorithmselected by application developers using NetQoS.For example, if network resource allocations failsfor a pair of components deployed in a particu-lar source and destination node, NetRAF requestsrevised CPU placements by adding a constraint tonot deploy the components in the same source anddestination nodes. After the revised CPU placementsare computed, NetRAF will (re)attempt to allocatenetwork resources for the components.

NetRAF automates the network resource alloca-tion process by iterating over the set of deploymentplans until a deployment plan is found that satisfiesboth types of requirements (i.e., both the CPU andnetwork resource requirements), thereby simplifyingsystem deployment via the following two-phaseprotocol:

1) It first invokes the API of QoS mechanism-specific allocator, providing it one flow ata time without actually reserving networkresources.

2) It then commits the network resources if andonly if the first phase is completely successfuland resources for all the flows can be success-fully reserved.

8

This protocol prevents the delay that would other-wise be incurred if resources allocated for a subsetof flows must be released due to failures occurring ata later allocation stage. If no deployment plan yieldsa successful resource allocation, the network QoSrequirements of component flows must be reducedusing NetQoS.Application to the case study. Since our case studyis based on DiffServ, NetRAF uses the DiffServAllocator to allocate network resources, which inturn invokes the Bandwidth Broker’s admission con-trol capabilities [21] by feeding it one applicationflow at a time. If all flows cannot be admitted, Ne-tRAF provides developers with an option to modifythe deployment context since applications have notyet been deployed. Example modifications includechanging component implementations to consumefewer resources or change the source/destinationnodes. As shown in Section IV-C, this capabilityhelps NetRAF incur lower overhead than conven-tional approaches [10], [11] that perform validationdecisions when applications are deployed and oper-ated at runtime.

NetRAF’s DiffServ Allocator instructs the Band-width Broker to reserve bi-directional resources inthe specified network QoS classes, as described inSection III-A. The Bandwidth Broker determinesthe bi-directional DSCPs and NetRAF encodesthose values as connection attributes in the deploy-ment plan. In addition, the Bandwidth Broker usesits Flow Provisioner [16] to configure the routersto provide appropriate per-hop behavior when theyreceive IP packets with the specified DSCP values.Section III-C describes how component containersare auto-configured to add these DSCPs when ap-plications invoke remote operations.

C. Challenge 3: Alleviating Complexities in Net-work QoS Settings Configuration

Context.: After network resources are allo-cated and network routers are configured, applica-tions in DRE systems need to invoke remote oper-ations using the chosen network QoS settings (e.g.,DSCP markings) so the network can differentiateapplication traffic and provision appropriate QoS toeach flow.Problem. Application developers have historicallywritten code that instructs the middleware to providethe appropriate runtime services, e.g., DSCP mark-ings in IP packets [12]. For example, fire sensors in

our case study from Section II can be deployed indifferent QoS contexts that are managed by reusablesoftware controllers. Modifying application code toinstruct the middleware to add network QoS settingsis tedious, error-prone, and non-scalable because (1)the same application code could be used in differentcontexts requiring different network QoS settingsand (2) application developers might not (and ide-ally should not) know the different QoS contexts inwhich the applications are used during the develop-ment process. Application-transparent mechanismsare therefore needed to configure the middleware toadd these network QoS settings depending on theapplication deployment context.Solution approach → Deployment and run-time component middleware mechanisms. Side-bar 1 describes how LwCCM containers provide aruntime environment for components.2 NetQoPE’sNetwork QoS Configurator (NetCON) can auto-configure these containers by adding DSCPs to IPpackets when applications invoke remote operations.NetRAF performs network resource allocations, de-termines the bi-directional DSCP values to use foreach application flow and encodes those DSCPvalues in the deployment plan, as shown in Figure 6.

6: NetCON’s Container Auto-configurations

During deployment, NetCON parses the deploy-ment plan and its connection tags to determine (1)source and destination components, (2) the networkpriority model to use for their communication, (3)the bi-directional DSCP values, and (4) the targetnodes on which the components are deployed. Net-CON deploys the components on their respectivecontainers and creates the associated object refer-ences for use by clients in a remote invocation.When a component invokes a remote operation in

2Other component middleware provides similar capabilities viacontainers, e.g., EJB applications interact with containers to obtainthe right runtime operating environment.

9

LwCCM, its container’s context information pro-vides the object reference of the destination com-ponent.

NetCON’s container programming model cantransparently add DSCPs and enforce the networkpriority models described in Section III-A. Tosupport the SERVER_DECLARED network prioritymodel, NetCON encodes a SERVER_DECLARED

policy and the associated request/reply DSCPs onthe server’s object reference. When a client invokesa remote operation with this object reference, theclient-side middleware checks the policy on theobject reference, decodes the request DSCP, andincludes it in the request IP packets. Before sendingthe reply, the server-side middleware checks thepolicy again and the reply DSCP is added to theassociated IP packets.

To support the CLIENT_PROPAGATED networkpriority model, NetCON configures the containersto apply a CLIENT_PROPAGATED policy at thepoint of binding an object reference with the client.In contrast to the SERVER_DECLARED policy, theCLIENT_PROPAGATED policy can be changed atruntime and different clients can access the serverswith different network priorities. When the sourcecomponent invokes a remote operation using thepolicy-applied object reference, NetCON adds theassociated forward and reverse DSCP markings onthe IP packets, thereby providing network QoS tothe application flow. A NetQoPE-enabled containercan therefore transparently add both forward and re-verse DSCP values when components invoke remoteoperations using the container services.Application to the case study. In our case studyshown in Figure 4, the FireSensor software con-troller component is deployed in two different in-stances to control the operation of the fire sensorsin the parking lot and the server room. Thereis a single MonitorController software component(MonitorController3 in Figure 4) that communicateswith the deployed FireSensor components. Due todifferences in importance of the FireSensor com-ponents deployed, however, the MonitorControllersoftware component the uses CLIENT_PROPAGATED

network priority model to communicate with theFireSensor components with different network QoSrequirements.

After software components are modeled us-ing NetQoS—and the required network resourcesare allocated using NetRAF—NetCON config-

ures the container hosting the MonitorController3component with the CLIENT_PROPAGATED policy,which corresponds to the CLIENT_PROPAGATED

network priority model defined on the componentby NetQoS. This capability is provided automati-cally by containers to ensure that the appropriateDSCP values are added to both forward and re-verse communication paths when the MonitorCon-troller3 component communicates with either theFireSensorParking or FireSensorServer componentat runtime. Communication between the Monitor-Controller3 and the FireSensorParking or FireSen-sorServer components thus receives the requirednetwork QoS since NetRAF configures the routersbetween the MonitorController3 and FireSensor-Parking components with the source IP address,destination IP address, and DSCP tuple.

NetCON therefore allows DRE system developersto focus on their application component logic (e.g.,the MonitorController component in the case study),rather than wrestling with low-level mechanismsfor provisioning network QoS. Moreover, NetCONprovides these capabilities without modifying appli-cation code, thereby simplifying development andminimizing runtime overhead, as shown in Sec-tion IV-C.1.

IV. EVALUATING NETQOPE

This section empirically evaluates the flexibilityand overhead of using NetQoPE to provide CPU andnetwork QoS assurance to end-to-end applicationflows. We first validate that NetQoPE’s automatedmodel-driven approach can provide differentiatednetwork performance for a variety of applicationsin DRE systems, such as our case study. We thendemonstrate how NetQoPE’s network QoS provi-sioning capabilities can significantly reduce applica-tion development effort compared with conventionalapproaches.

A. Evaluation Scenario

Our empirical evaluation of NetQoPE was con-ducted at ISISlab (www.dre.vanderbilt.edu/ISISlab), which consists of (1) 56 dual-CPU bladesrunning 2.8 Gz XEONs with 1 GB memory, 40GB disks, and 4 NICs per blade, and (2) 6 Cisco3750G switches with 24 10/100/1000 MPS ports perswitch. As shown in Figure 7, our experiments wereconducted on 15 of dual CPU blades in ISISlab,

10

where (1) 7 blades (A, B, D, E, F, G, and H) hostedour modern office enterprise case study softwarecomponents (e.g., a fire sensor software controller)and (2) 8 other blades (P, Q, R, S, T, U, V, and W)hosted Linux router software.

7: Experimental Setup

The software controller components were devel-oped using the CIAO middleware, which is an open-source LwCCM implementation developed atop theTAO real-time CORBA object request broker [22].Our evaluations used DiffServ QoS and the asso-ciated Bandwidth Broker [21] software was hostedon blade C. All blades ran Fedora Core 4 Linuxdistribution configured using the real-time schedul-ing class. The blades were connected over a 1 GbpsLAN via virtual 100 Mbps links.

In our evaluation scenario, a number of sensoryand imagery software controllers sent their mon-itored information to monitor controllers so thatappropriate control actions could be performed byenterprise supervisors monitoring abnormal events.For example, Figure 7 shows two fire sensor con-troller components deployed on hosts A and B.These components sent their monitored informationto monitor controller components deployed on hostsD and F. Communication between these softwarecontrollers used one of the traffic classes (e.g., HIGH

PRIORITY (HP)) defined in Section III-A with thefollowing capacities on all links: HP = 20 Mbps, HR

= 30 Mbps, and MM = 30 Mbps. The BE class usedthe remaining available bandwidth in the network.

To emulate the network behavior of the soft-ware controllers when different network QoSrequirements are provisioned, we created theTestNetQoPE performance benchmark suite.3 We

3TestNetQoPE can be downloaded as part of the CIAO open-source middleware available at (www.dre.vanderbilt.edu/CIAO).

used TestNetQoPE to evaluate the flexibility, over-head, and performance of using NetQoPE to providenetwork QoS assurance to end-to-end applicationflows. In particular, we used TestNetQoPE to spec-ify and measure diverse CPU and network QoSrequirements of the different software componentsthat were deployed via NetQoPE, such as the appli-cation flow between the fire sensor controller com-ponent on host A and the monitor controller com-ponent on host D. These tests create a session forcomponent-to-component communication with con-figurable bandwidth consumption. High-resolutiontimer probes were used to measure roundtrip latencyaccurately for each client invocation.

B. Experimental Results and Analysis

We now describe the experiments performed us-ing the ISISlab configuration described in Sec-tion IV-A and analyze the results.

C. Evaluating NetQoPE’s Model-driven QoS Pro-visioning Capabilities

Rationale. As discussed in Section III, NetQoPE isdesigned to provision application CPU and networkQoS mechanisms in an extensible manner. This ex-periment evaluates the effort application developersspend using NetQoPE to (re)deploy applications andprovision QoS and compares this effort against theeffort needed to provision QoS for applications viaconventional approaches.Methodology. We first identified four flows fromFigure 7 whose network QoS requirements aredescribed as follows:

• A fire sensor controller component on host Auses the high reliability (HR) class to sendpotential fire alarms in the parking lot to themonitor controller component on host D.

• A fire sensor controller component on hostB uses the high priority (HP) class to sendpotential fire alarms in the server room to themonitor controller component on host F.

• A camera controller component on host E usesthe multimedia (MM) class and sends imageryinformation from the break room to the monitorcontroller component on host G.

• A temperature sensor controller component onhost A uses the best effort (BE) class and sendstemperature readings to the monitor controllercomponent on host F.

11

The clients dictated the network priority for requestsand replies in all flows except for the tempera-ture sensor and monitor controller component flow,where the server dictated the priority. TCP was usedas the transport protocol and 20 Mbps of forwardand reverse bandwidth was requested for each typeof network QoS traffic.

We also define a taxonomy for evaluating tech-nologies that provide network QoS assurancesto end-to-end DRE application flows to com-pare NetQoPE’s methodology of provisioning net-work QoS for these flows with conventional ap-proaches, including (1) object-oriented [23], [11],[10], [12], (2) aspect-oriented [24], and (3) compo-nent middleware-based [13], [5] approaches. Belowwe describe how each approach provides the follow-ing functionality needed to leverage network QoSmechanism capabilities:

• QoS Requirements specification. In conven-tional approaches applications use (1) middleware-based APIs [23], [10], (2) contract definition lan-guages [11], [12], (3) runtime aspects [24], or (4)specialized component middleware container inter-faces [13] to specify network QoS requirements.These approaches do not, however, provide capabil-ities to specify both CPU and network requirementsand assume that physical node placement for allcomponents are decided before the network resourceallocations are requested.

Moreover, application source code must changewhenever the deployment context (e.g., differentphysical node placements, component deploymentfor a different usecase) and the associated QoSrequirements (e.g., CPU or network resource re-quirements) changes, which limits reusability. Incontrast, NetQoS provides domain-specific, declar-ative techniques that increase reusability across dif-ferent deployment contexts and alleviate the needto specify QoS requirements programmatically, asdescribed in Section III-A.

• Network resource allocation. Conventionalapproaches require application deployment beforetheir per-flow network resource requirements canbe provisioned by network QoS mechanisms. Ifthe required resources cannot be allocated for theseapplications, however, the following steps occur:

1) They must be stopped2) Their source code must be modified to specify

new resource requirements (e.g., either sourceand destination nodes of the components can

be changed or for the same pair of sourceand destination nodes the network resourcerequirements could be changed) and

3) The resource reservation process must berestarted.

This approach is tedious since applications maybe deployed and re-deployed multiple times, po-tentially on different nodes. In contrast, NetRAFhandles deployment changes via NetQoS models(see Section III-B) at pre-deployment, i.e., beforeapplications have been deployed, thereby reducingthe effort needed to change deployment topology orapplication QoS requirements.• Network QoS enforcement. Conventional ap-

proaches modify application source code [12] orprogramming model [13] to instruct the middlewareto enforce runtime QoS for their remote invocations.Applications must therefore be designed to handletwo different usecases—to enforce QoS and whenno QoS is required—thereby limiting applicationreusability. In contrast, NetCON uses a containerprogramming model that transparently enforces run-time QoS for applications without changing theirsource code or programming model, as described inSection III-C.

We now compare the effort required to provisionnetwork QoS to the 4 end-to-end application flowsdescribed above using conventional manual ap-proaches vs. the NetQoPE model-driven approach.We decompose this effort across the following gen-eral steps: (1) implementation, where software de-velopers write code, (2) deployment, where systemdeployers map (or stop) application components ontheir target nodes, and (3) modeling tool use, whereapplication developers use NetQoPE to model aDRE application structure and specify per-flow QoSrequirements. In our evaluation, a complete QoSprovisioning lifecycle consists of specifying require-ments, allocating resources, deploying applications,and stopping applications when they are finished.

To compare NetQoPE with manual efforts, wedevised a realistic scenario for the 4 end-to-endapplication flows described above. In this scenario,three sets of experiments were conducted with thefollowing deployment variants:• Baseline deployment. This variant configured

all 4 end-to-end application flows with the networkQoS requirements as described above. The manualeffort required using conventional approaches forthe first deployment involved 10 steps: (1) modify

12

source code for each of the 8 components to specifytheir QoS requirements (8 implementation steps),(2) deploy all components (1 deployment step),and (3) shutdown all components (1 deploymentstep). In contrast, the effort required using NetQoPEinvolved the following 4 steps: (1) model the DREapplication structure of all 4 end-to-end applicationflows using NetQoS (1 modeling step), (2) annotateQoS specifications on each end-to-end applicationflow (1 modeling step), (3) deploy all components (1deployment step), and (4) shutdown all components(1 deployment step).

• QoS modification deployment. This variantdemonstrated the effect of changes in QoS re-quirements on manual efforts by modifying thebandwidth requirements from 20 Mbps to 12 Mbpsfor each end-to-end flow. As with baseline vari-ant above, the effort required using a conventionalapproach for the second deployment was 10 stepssince source code modifications were needed asthe deployment contexts changed (in this case, thebandwidth requirements changed across 4 differentdeployment contexts). In contrast, the effort requiredusing NetQoPE involved 3 steps: (1) annotate QoSspecifications on each end-to-end application flow(1 modeling step), (2) deploy all components (1deployment step), and (3) shutdown all components(1 deployment step). Application developers alsoreused NetQoS’s application structure model createdfor the initial deployment, which helped reduce therequired efforts by a step.

• Resource (re)reservation deployment. Thisvariant demonstrated the effect of changes in QoSrequirements and resource (re)reservations takentogether on manual efforts. We modified bandwidthrequirements of all flows from 12 Mbps to 16Mbps. We also changed temperature sensor con-troller component to use the high reliability (HR)class instead of the best effort BE class. Finally, weincreased the background HR class traffic across thehosts so that the resource reservation request for theflow between temperature sensor and monitor con-troller components fails. In response, deploymentcontexts (e.g., bandwidth requirements, source anddestination nodes) were changed and resource re-reservation was performed.

The effort required using a conventional approachfor the third deployment involved 13 steps: (1)modify source code for each of the 8 components tospecify their QoS requirements (8 implementation

steps), (2) deploy all components (1 deploymentstep), (3) shutdown the temperature sensor com-ponent (1 deployment step – resource allocationfailed for the component), (4) modify source codeof temperature sensor component back to use BE

network QoS class (deployment context change) (1implementation step), (5) redeploy the temperaturesensor component (1 deployment step), and (6)shutdown all components (1 deployment step).

In contrast, the effort required using NetQoPE forthe third deployment involved 4 steps: (1) annotateQoS specifications on each end-to-end applicationflow (1 modeling step), (2) begin deployment ofall components, though NetRAF’s pre-deployment-time allocation capabilities determined the resourceallocation failure and prompted the NetQoPE appli-cation developer to change the QoS requirements(1 pre-deployment step), (3) re-annotate QoS re-quirements for the temperature sensor componentflow (1 modeling step) (4) deploy all components (1deployment step), and (5) shutdown all components(1 deployment step).

Table I summarizes the step-by-step analysis de-scribed above. These results show that conventional

Approaches # Steps in Experiment VariantsFirst Second Third

NetQoPE 4 3 5Conventional 10 10 13

I: Comparison of Manual Efforts Incurred in Con-ventional and NetQoPE Approachesapproaches incurred roughly an order of magnitudemore effort than NetQoPE to provide network QoSassurance for end-to-end application flows. Closerexamination shows that in conventional approaches,application developers spend substantially more ef-fort developing software that can work across dif-ferent deployment contexts. Moreover, this processmust be repeated when deployment contexts andtheir associated QoS requirements change. In ad-dition, conventional implementations are complexsince the requirements are specified directly usingmiddleware [10] and/or network QoS mechanismAPIs [6].

Application (re)deployments are also requiredwhenever reservation requests fail. In this experi-ment, only 1 flow required re-reservation and thatincurred additional effort of 3 steps. If there arelarge number of flows—and enterprise DRE systemslike our case study often have scores of flows—conventional approaches require significantly more

13

effort.In contrast, NetQoPE’s ability to “write once,

deploy multiple times for different QoS require-ments” increases deployment flexibility and exten-sibility in environments that deploy many reusablesoftware components. To provide this flexibility,NetQoS generates XML-based deployment descrip-tors that capture context-specific QoS requirementsof applications. For our experiment, communicationbetween fire sensor and monitor controllers wasdeployed in multiple deployment contexts, i.e., withbandwidth reservations of 20 Mbps, 12 Mbps, and16 Mbps. In DRE systems like our case study,however, the same communication patterns betweencomponents could occur in many deployment con-texts.

For example, the same communication patternscould use any of the four network QoS classes (HP,HR, MM, and BE). The communication patterns thatuse the same network QoS class could make dif-ferent forward and reverse bandwidth reservations(e.g., 4, 8, or 10 Mbps). As shown in Table II,NetQoS auto-generates over 1,300 lines of XMLcode for these scenarios, which would otherwisebe handcrafted by application developers. These

Deployment contextsNumber of communications2 5 10 20

1 23 50 95 1855 47 110 215 425

10 77 185 365 72520 137 335 665 1325

II: Generated Lines of XML Code

results demonstrate that NetQoPE’s model-drivennetwork QoS provisioning capabilities significantlyreduce application development effort comparedwith conventional approaches. Moreover, NetQoPEalso provides increased flexibility when deployingand provisioning multiple application end-to-endflows in multiple deployment and network QoScontexts.

1) Evaluating the Overhead of NetQoPE forNormal Operations: Rationale. NetQoPE providesnetwork QoS to applications via the four-stage ar-chitecture shown in Figure 1. This experiment eval-uates the runtime performance overhead overheadof using NetQoPE to enforce network QoS.Methodology. DRE system developers can useNetQoPE at design time to specify network QoS re-quirements on the application flows, as described inSection III-A. Based on the specified network QoS

requirements, NetRAF interacts with the BandwidthBroker to allocate per-flow network resources atpre-deployment time. By providing design- and pre-deployment-time capabilities, NetQoS and NetRAFthus incur no runtime overhead. In contrast, Net-CON configures component middleware containersat post-deployment-time by adding DSCP markingsto IP packets when applications invoke remote op-erations (see Section III-C). NetCON may thereforeincur runtime overhead, e.g., when containers applya network policy models to provide the source ap-plication with an object reference to the destinationapplication.

To measure NetCON’s overhead, we conductedan experiment to determine the runtime overheadof the container when it performs extra work toapply the policies that add DSCPs to IP packets.This experiment had the following variants: (1)the client container was not configured by Net-CON (no network QoS required), (2) the clientcontainer was configured by NetCON to apply theCLIENT_PROPAGATED network policy, and (3) theclient container was configured by NetCON to applythe SERVER_DECLARED network policy. This ex-periment had no background network load to isolatethe effects of each variant.

Our experiment had no network congestion, soQoS support was thus not needed. The network pri-ority models were therefore configured with DSCPvalues of 0 for both the forward and reverse direc-tion flows. TestNetQoPE was configured to make200,000 invocations that generated a load of 6Mbps and average roundtrip latency was calculatedfor each experiment variant. The routers were notconfigured to perform DiffServ processing (providerouting behavior based on the DSCP markings), sono edge router processing overhead was incurred.We configured the experiment to pinpoint only theoverhead of the container no other entities in thepath of client remote communications.Analysis of results. Figure 8 shows the averageroundtrip latencies experienced by clients in thethree experiment variants (in this figure CP is theCLIENT_PROPAGATED network priority model andSD is the SERVER_DECLARED model). To honor thenetwork policy models, the NetQoPE middlewareadded the request/reply DSCPs to the IP packets.The latency results shown in Figure 8 are all similar,which shows that NetCON is efficient and adds neg-ligible overhead to applications. If another variant

14

99 percentMeanMax

16

18

20

22

24

SDCPNo−QoS

Lat

ency

(m

ilise

cond

s)

8: Overhead of NetQoPE’s Policy Framework

of the experiment was run with background networkloads, network resources will be allocated and theappropriate DSCP values used for those applicationflows. The NetCON runtime overhead will remainthe same, however, since the same middlewareinfrastructure is used, only with different DSCPvalues.

2) Evaluating NetQoPE’s QoS CustomizationCapabilities: Rationale. NetQoPE’s model-drivenapproach enhances flexibility by enabling the reuseof application source code in different deploymentcontexts. It can also address the QoS needs of awide variety of applications by supporting multipleDiffServ classes and network priority models. Thisexperiment evaluates the benefits of these capabili-ties empirically.Methodology. We identified four flows from Fig-ure 7 and modeled them using NetQoS as follows:

• A fire sensor controller component on bladeA uses the high reliability (HR) class to sendpotential fire alarms in the parking lot to themonitor controller component on blade D.

• A fire sensor controller component on bladeB uses the high priority (HP) class to sendpotential fire alarms in the server room to themonitor controller component on blade F.

• A camera controller component on blade Euses the multimedia (MM) class and sendsimagery information of the break room to themonitor controller component on blade G.

• A temperature sensor controller component onblade A uses the best effort (BE) class andsends temperature readings to the monitor con-troller component on blade F.

The CLIENT_PROPAGATED network policy wasused for all flows, except for the temperature sensorand monitor controller component flow, which usedthe SERVER_DECLARED network policy.

We performed two variants of this experiment.The first variant used TCP as the transport protocoland requested 20 Mbps of forward and reverse band-

width for each type of QoS traffic. TestNetQoPEconfigured each application flow to generate a loadof 20 Mbps and the average roundtrip latencyover 200,000 iterations was calculated. The secondvariant used UDP as the transport protocol andTestNetQoPE was configured to make oneway in-vocations with a payload of 500 bytes for 100,000iterations. We used high-resolution timer probes tomeasure the network delay for each invocation onthe receiver side of the communication.

At the end of the second experiment we recorded100,000 network delay values (in ms) for eachnetwork QoS class. Those network delay valueswere then sorted in increasing order and everyvalue was subtracted from the minimum value inthe whole sample, i.e., they were normalized withrespect to the respective class minimum latency. Thesamples were divided into fourteen buckets basedon their resulting values. For example, the 1 msbucket contained only samples that are <= to 1 msin their resultant value, the 2 ms bucket containedonly samples whose resultant values were <= 2 msbut > 1 ms, etc.

To evaluate application performance in the pres-ence of background network loads, several otherapplications were run in both experiments, as de-scribed in Table III (in this table TS stands for “tem-perature sensor controller,” MS stands for “monitorcontroller”, FS stands for “fire sensor controller,”and CS stands for “camera controller”). NetRAF

Background Traffic in MbpsTraffic TypeBE HP HR MM

BE (TS - MS) 85 to 100HP (FS - MS) 30 to 40 28 to 33 28 to 33HR (FS - MS) 30 to 40 12 to 20 14 to 15 30 to 31MM (CS - MS) 30 to 40 12 to 20 14 to 15 30 to 31

III: Application Background Trafficallocated the network resources for each flow anddetermined which DSCP values to use. After de-ploying the applications, NetCON configured thecontainers to use the appropriate network prioritymodels to add DSCP values to IP packets whenapplications invoked remote operations.Analysis of results. Figure 9a shows the results ofexperiments when the deployed applications wereconfigured with different network QoS classes andsent TCP traffic. This figure shows that irrespectiveof the heavy background traffic, the average latencyexperienced by the fire sensor controller componentusing the HP network QoS class is lower than the

15

0

50,000

100,000

150,000

200,000

BEMMHRHP

Lat

ency

(m

icro

seco

nds)

Network QoS classes

88,939 97,300

132,644

187,805

(a) Average Latency under Different Network QoS Classes (b) Jitter Distribution under Different Network QoS Classes

9: Performance of NetQoPE

average latency experienced by all other compo-nents. In contrast, the traffic from the BE class is notdifferentiated from the competing background trafficand thus incurs a high latency (i.e., throughput isvery low). Moreover, the latency increases whileusing the HR and MM classes when compared tothe HP class.

Figure 9b shows the (1) cardinality of the networkdelay groupings for different network QoS classesunder different ms buckets and (2) losses incurredby each network QoS class. These results showthat the jitter values experienced by the applicationusing the BE class are spread across all the buckets,i.e., are highly unpredictable. When combined withpacket or invocation losses, this property is undesir-able in DRE systems. In contrast, the predictabilityand loss-ratio improves when using the HP class, asevidenced by the spread of network delays acrossjust two buckets. The application’s jitter is almostconstant and is not affected by heavy backgroundtraffic.

The results in Figure 9b also show that theapplication using the MM class experienced morepredictable latency than applications using BE andHR class. Approximately 94% of the MM classinvocations had their normalized delays within 1ms. This result occurs because the queue size at therouters is smaller for the MM class than the queuesize for the HR class, so UDP packets sent by theinvocations do not experience as much queueingdelay in the core routers as packets belonging tothe HR class. The HR class provides better loss-ratio, however, because the queue sizes at the routersare large enough to hold more packets when thenetwork is congested.

These results demonstrate that NetQoPE’s au-tomated model-driven middleware-guided mecha-nisms (1) support the needs of a wide variety ofapplications by simplifying the modeling of QoS

requirements via various DiffServ network QoSclasses and (2) provide those modeled applicationswith differentiated network performance validatingthe automated network resource allocation and con-figuration process. By using NetQoPE, therefore,applications can leverage the capabiltiies of networkQoS mechanisms with minimal effort, as describedin Section IV-C.

These results also demonstrate the following QoScustomization possibilities for a set of applicationcommunications (e.g., fire sensor and monitor con-troller component):

• Different network QoS performance, e.g., HP

communication between blades A and D, andHR communication between blades B and F.

• Different transport protocols for communica-tion, e.g., TCP and UDP.

• Different network access models, e.g., monitorcontroller components were accessed using theCLIENT_PROPAGATED network priority modeland the SERVER_DECLARED network prioritymodel.

These results show how NetQoPE’s ability to “writeonce, deploy multiple times for different QoS re-quirements” increased deployment flexibility andextensibility for environments where many reusablesoftware components are deployed. To provide thisflexibility, NetQoS generates XML-based deploy-ment descriptors that capture context-specific QoSrequirements of applications. For our experiment,communication between fire sensor and monitorcontrollers was deployed in multiple deploymentcontexts, i.e., HR and HP QoS requirements.

V. RELATED WORK

This section compares our R&D activities onNetQoPE with related work on middleware-basedQoS management and model-based design tools.

16

Network QoS management in middleware.Prior work on integrating network QoS mechanismswith middleware [10], [11], [12], [23] focused onproviding middleware APIs to shield applicationsfrom directly interacting with complex network QoSmechanism APIs. Middleware frameworks trans-parently converted the specified application QoSrequirements into lower-level network QoS mech-anism APIs and provided network QoS assurances.These approaches, however, modified applicationsto dictate QoS behavior for the various flows.NetQoPE differs from these approaches by provid-ing application-transparent and automated solutionsto leverage network QoS mechanisms, thereby sig-nificantly reducing manual design and developmenteffort to obtain network QoS.

QoS management in middleware. Prior researchhas focused on adding various types of QoS capa-bilities to middleware. For example, [25] describesJ2EE container resource management mechanismsthat provide CPU availability assurances to appli-cations. Likewise, 2K [26] provides QoS to appli-cations from varied domains using a component-based runtime middleware. In addition, [13] ex-tends EJB containers to integrate QoS features byproviding negotiation interfaces which the appli-cation developers need to implement to receivedesired QoS support. Synergy [27] describes adistributed stream processing middleware that pro-vides QoS to data streams in real time by effi-cient reuse of data streams and processing com-ponents. These approaches are restricted to CPUQoS assurances or application-level adaptations toresource-constrained scenarios. NetQoPE differs byproviding network QoS assurances in a application-agnostic fashion.

Deployment-time resource allocation. Priorwork has focused on deploying applications at ap-propriate nodes so that their QoS requirements canbe met. For example, prior work [28], [29] hasstudied and analyzed application communicationand access patterns to determine collocated place-ments of heavily communicating components. Otherresearch [8], [9] has focused on intelligent compo-nent placement algorithms that maps componentsto nodes while satisfying their CPU requirements.NetQoPE differs from these approaches by leverag-ing network QoS mechanisms to allocate networkresources at pre-deployment-time and enforcing net-work QoS at runtime.

Model-based design tools. Prior work has beendone on model-based design tools. PICML [14] en-ables DRE system developers to define componentinterfaces, their implementations, and assemblies,facilitating deployment of LwCCM-based applica-tions. VEST [30] and AIRES [15] analyze domain-specific models of embedded real-time systems toperform schedulability analysis and provides au-tomated allocation of components to processors.SysWeaver [31] supports design-time timing behav-ior verification of real-time systems and automaticcode generation and weaving for multiple targetplatforms. In contrast, NetQoPE provides model-driven capabilities to specify network QoS require-ments on DRE system application flows, and subse-quently allocate network resources automatically us-ing network QoS mechanisms. NetQoPE thus helpsassure that application network QoS requirementsare met at deployment-time, rather than design-timeor runtime.

VI. CONCLUDING REMARKS

This paper describes the design and evaluationof NetQoPE, which is a model-driven componentmiddleware framework that manages CPU and net-work QoS for applications in distributed real-timeand embedded (DRE) systems. The lessons welearned developing NetQoPE and applying it toa representative DRE system case study thus farinclude:• NetQoPE’s domain-specific modeling lan-

guages (e.g., NetQoS) help capture per-deploymentQoS requirements of applications so that CPU andnetwork resources can be allocated appropriately.Application business logic consequently need notbe modified to specify deployment-specific QoSrequirements, thereby increasing software reuse andflexibility across a range of deployment contexts, asshown in Section III-A.• Programming network QoS mechanisms di-

rectly in application code requires the deploymentand execution of applications before they can deter-mine if the required network resources are availableto meet QoS needs. Conversely, providing these ca-pabilities via NetQoPE’s model-driven, middlewareframework helps guide resource allocation strategiesbefore application deployment, thereby simplifyingvalidation and adaptation decisions, as shown inSection III-B.

17

• NetQoPE’s model-driven deployment and con-figuration tools help configure the underlying com-ponent middleware transparently on behalf of ap-plications to add context-specific network QoS set-tings. These settings can be enforced by NetQoPE’sruntime middleware framework without modifyingthe programming model used by applications. Ap-plications therefore need not change how they com-municate at runtime since network QoS settings canbe added transparently, as shown in Section III-C.

• NetQoPE’s strategy of allocating network re-sources to applications before deployment may betoo limiting for certain types of DRE systems. Inparticular, applications in open DRE systems [32]might not consume all their resource allotment atruntime, in which case NetQoPE may underuti-lize system resources. Our future work is there-fore extending NetQoPE to overprovision resourcesfor applications on the assumption that not allapplications will use their allotment. If runtimeresource contentions occur, we are also integratingdynamic resource management algorithms [33] withNetQoPE to provide predictable network perfor-mance for applications in open DRE systems.

All of NetQoPE’s model-driven middleware plat-forms and tools—except the Bandwidth Broker—described in this paper and used in the ex-periments are available in open-source formatfrom www.dre.vanderbilt.edu/cosmic and inthe CIAO component middleware available at www.dre.vanderbilt.edu.

REFERENCES

[1] J. Balasubramanian, S. Tambe, B. Dasarathy, S. Gadgil,F. Porter, A. Gokhale, and D. C. Schmidt, “NetQoPE: AModel-Driven Network QoS Provisioning Engine for EnterpriseDistributed Real-time and Embedded Systems,” in Proceedingsof the 14th IEEE Real-time and Embedded Technology andApplications Symposium, St. Louis, MO, USA, Apr. 2008.

[2] D. C. Schmidt, R. Schantz, M. Masters, J. Cross, D. Sharp, andL. DiPalma, “Towards Adaptive and Reflective Middleware forNetwork-Centric Combat Systems,” CrossTalk - The Journal ofDefense Software Engineering, Nov. 2001.

[3] A. Nechypurenko, D. C. Schmidt, T. Lu, G. Deng, A. Gokhale,and E. Turkay, “Concern-based Composition and Reuse ofDistributed Systems,” in Proceedings of the 8th InternationalConference on Software Reuse. Madrid, Spain: ACM/IEEE,July 2004.

[4] D. C. Sharp and W. C. Roll, “Model-Based Integration ofReusable Component-Based Avionics System,” in Proceedingsof the Workshop on Model-Driven Embedded Systems in RTAS2003. Washington, DC: IEEE Computer Society, May 2003.

[5] P. Sharma, J. Loyall, G. Heineman, R. Schantz, R. Shapiro, andG. Duzan, “Component-Based Dynamic QoS Adaptations inDistributed Real-time and Embedded Systems,” in Proceedingsof the International Symposium on Distributed Objects andApplications (DOA), Agia Napa, Cyprus, Oct. 2004.

[6] L. Zhang and S. Berson and S. Herzog and S. Jamin, “ResourceReSerVation Protocol (RSVP) Version 1 Functional Specifica-tion,” Network Working Group RFC 2205, pp. 1–112, Sept.1997.

[7] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, andW. Weiss, “An Architecture for Differentiated Services,” In-ternet Society, Network Working Group RFC 2475, pp. 1–36,Dec. 1998.

[8] D. de Niz and R. Rajkumar, “Partitioning Bin-Packing Algo-rithms for Distributed Real-time Systems,” International Jour-nal of Embedded Systems, vol. 2, no. 3, pp. 196–208, 2006.

[9] S. Gopalakrishnan and M. Caccamo, “Task Partitioning withReplication upon Heterogeneous Multiprocessor Systems,” inRTAS ’06, San Jose, CA, USA, 2006, pp. 199–207.

[10] P. Wang, Y. Yemini, D. Florissi, and J. Zinky, “A DistributedResource Controller for QoS Applications,” in Proceedings ofthe Network Operations and Management Symposium (NOMS2000). IEEE/IFIP, Apr. 2000.

[11] R. Schantz, J. Zinky, D. Karr, D. Bakken, J. Megquier, andJ. Loyall, “An Object-level Gateway Supporting Integrated-Property Quality of Service,” ISORC, vol. 00, p. 223, 1999.

[12] R. Schantz and J. Loyall and D. Schmidt and C. Rodriguesand Y. Krishnamurthy and I. Pyarali, “Flexible and AdaptiveQoS Control for Distributed Real-time and Embedded Middle-ware,” in Proc. of Middleware’03. Rio de Janeiro, Brazil:IFIP/ACM/USENIX, June 2003.

[13] M. A. de Miguel, “Integration of QoS Facilities into Com-ponent Container Architectures,” in Proceedings of the FifthIEEE International Symposium on Object-Oriented Real-TimeDistributed Computing (ISORC 2002), 2002.

[14] K. Balasubramanian, J. Balasubramanian, J. Parsons,A. Gokhale, and D. C. Schmidt, “A Platform-IndependentComponent Modeling Language for Distributed Real-time andEmbedded Systems,” Journal of Computer Systems Science,vol. 73, no. 2, pp. 171–185, 2007.

[15] Z. Gu, S. Kodase, S. Wang, and K. G. Shin, “A Model-BasedApproach to System-Level Dependency and Real-time Analysisof Embedded Software,” in RTAS’03. Washington, DC: IEEE,May 2003, pp. 78–85.

[16] B. Dasarathy, S. Gadgil, R. Vaidyanathan, A. Neidhardt,B. Coan, K. Parameswaran, A. McIntosh, and F. Porter, “Adap-tive network qos in layer-3/layer-2 networks for mission-criticalapplications as a middleware service,” Journal of Systems andSoftware: special issue on Dynamic Resource Management inDistributed Real-time Systems, 2006.

[17] Light Weight CORBA Component Model Revised Submission,OMG Document realtime/03-05-05 ed., Object ManagementGroup, May 2003.

[18] Deployment and Configuration of Component-based Dis-tributed Applications, v4.0, Document formal/2006-04-02 ed.,OMG, Apr. 2006.

[19] A. Ledeczi, A. Bakay, M. Maroti, P. Volgysei, G. Nordstrom,J. Sprinkle, and G. Karsai, “Composing Domain-Specific De-sign Environments,” IEEE Computer, pp. 44–51, November2001.

[20] K. Balasubramanian, J. Balasubramanian, J. Parsons,A. Gokhale, and D. C. Schmidt, “A Platform-IndependentComponent Modeling Language for Distributed Real-Time andEmbedded Systems,” in RTAS ’05: Proceedings of the 11th

18

IEEE Real Time on Embedded Technology and ApplicationsSymposium, Los Alamitos, CA, USA, 2005, pp. 190–199.

[21] B. Dasarathy, S. Gadgil, R. Vaidhyanathan, K. Parmeswaran,B. Coan, M. Conarty, and V. Bhanot, “Network QoS Assurancein a Multi-Layer Adaptive Resource Management Scheme forMission-Critical Applications using the CORBA MiddlewareFramework,” in RTAS 2005. San Francisco, CA: IEEE, Mar.2005.

[22] D. C. Schmidt, B. Natarajan, A. Gokhale, N. Wang, and C. Gill,“TAO: A Pattern-Oriented Object Request Broker for Dis-tributed Real-time and Embedded Systems,” IEEE DistributedSystems Online, vol. 3, no. 2, Feb. 2002.

[23] M. A. El-Gendy, A. Bose, S.-T. Park, and K. G. Shin, “Pavingthe First Mile for QoS-dependent Applications and Appliances,”in Proc. of IWQOS’04, Montreal, Canada, June 2004.

[24] G. Duzan, J. Loyall, R. Schantz, R. Shapiro, and J. Zinky,“Building Adaptive Distributed Applications with Middlewareand Aspects,” in Proc. of AOSD ’04, New York, NY, USA,2004, pp. 66–73.

[25] M. Jordan, G. Czajkowski, K. Kouklinski, and G. Skinner,“Extending a J2EE Server with Dynamic and Flexible ResourceManagement,” in Proceedings of the ACM/IFIP/USENIX Inter-national Middleware Conference, Toronto, Canada, 2004.

[26] D. Wichadakul, K. Nahrstedt, X. Gu, and D. Xu, “2K: AnIntegrated Approach of QoS Compilation and Reconfigurable,Component-Based Run-Time Middleware for the Unified QoSManagement Framework,” in Proc. of Middleware’01, 2001.

[27] T. Repantis, X. Gu, and V. Kalogeraki, “Synergy: Sharing-Aware Component Composition for Distributed Stream Process-ing Systems,” in Proc. of Middleware 2006.

[28] D. Llambiri, A. Totok, and V. Karamcheti, “Efficiently Dis-tributing Component-Based Applications Across Wide-AreaEnvironments,” in Proc. of ICDCS’03, 2003.

[29] C. Stewart and K. Shen, “Performance Modeling and SystemManagement for Multi-component Online Services,” in Proc.of NSDI’05, Boston, MA, May 2005, pp. 71–84.

[30] J. A. Stankovic, R. Zhu, R. Poornalingam, C. Lu, Z. Yu,M. Humphrey, and B. Ellis, “Vest: An aspect-based compositiontool for real-time systems,” in Proc. of RTAS’03, Washington,DC, USA, 2003, p. 58.

[31] D. de Niz, G. Bhatia, and R. Rajkumar, “Model-Based Devel-opment of Embedded Systems: The SysWeaver Approach,” inProc. of RTAS’06, Washington, DC, USA, August 2006, pp.231–242.

[32] X. Wang, D. Jia, C. Lu, and X. Koutsoukos, “DEUCON:Decentralized End-to-End Utilization Control for DistributedReal-Time Systems,” Parallel and Distributed Systems, IEEETransactions on, vol. 18, no. 7, pp. 996–1009, 2007.

[33] P. Lardieri, J. Balasubramanian, D. C. Schmidt, G. Thaker,A. Gokhale, and T. Damiano, “A Multi-layered ResourceManagement Framework for Dynamic Resource Managementin Enterprise DRE Systems,” Journal of Systems and Software:Special Issue on Dynamic Resource Management in DistributedReal-time Systems, vol. 80, no. 7, pp. 984–996, July 2007.