A Dynamically Reconfigurable Data Stream
Processing System
Jerzy M. NogiecFermilab
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 2
OutlineOutline
Dynamism in Software Systems …..…
EMS Framework ……………………….
Reconfiguration in EMS ……………….
Dynamism using Scripting ………..…..
Wrap-Up ……………..………….………
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 3
Dynamism in Software Systems
Dynamism in Software Systems
Open dynamic systemChanges at runtime that were not anticipated at build time.
Closed dynamic systemConfiguration changes performed at runtime, but planned at build time.
Tailorable systemSome aspects of the system can be modified at runtime.
Configurable system Many different configurations can be set up.
Static system System behavior fully described at build time.
dynamism
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 4
Dynamic Reconfiguration
Dynamic Reconfiguration
Dynamic reconfiguration is a process of modifying/evolving a running application, in which a composition of interacting components changes at runtime. It could be used to:
Modify behavior of the system why preserving continuity of its execution (mission or safety critical systems)
Apply patches to a running system Extend base application functionality with additional nonfunctional services (end
user customization and extension) Adapt to changing requirements or environment that are difficult to anticipate
(dynamically optimizing system functionality or recovering from errors and failures).
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 5
EMS FrameworkEMS Framework
EMS is a component-based framework for building measurement, data acquisition, and data stream processing systems. The goal of the EMS project was to design, implement, and deploy a system that is extensible, flexible and dynamic. EMS can be used to develop configurable and dynamically reconfigurable systems.
*Framework – the skeleton of an application that can be customized and reused**Component – an independently released software module suitable for composition (together with other components) into multiple applications
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 6
Visual ComponentsVisual Components
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 7
Component InterfacesComponent Interfaces
• properties• state• error status
data data
control
exception debug
Components have properties and state. Typical components input, process, and output data. Their behavior depends on their state and property values. Components can be forced to perform certain actions by sending control events to them. Components also output debug and exception information.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 8
Component ConnectivityComponent Connectivity
splitter
merger
filter
There are several categories of data processing components:
Data sinks Data sources Data filters/processors Data splitters Data mergers
source
sink
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 9
Connecting ComponentsConnecting
Components EMS provides for
independent “wiring” of components for property, data, exception, control, and debug events.
EMS supports unicast, multicast, and broadcast patterns.
Source routing (dynamic, run-time) & routing tables (static, configuration-time).
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 10
<configuration version="0.1" title="Display Test XML">
<!-- Component definitions --> <component id="Producer” class="ems.core.components.SimpleDataGenerator"> <property name="title" value="Data Generator Component"/> </component> <component id="Chart" class="ems.measurement.chart.ChartDataDisplay"> <property name="XPosition" value="0"/> <property name="YPosition" value="200"/> <property name="title" value="Plot Display"/> </component>
<!-- Routing information --> <route type="Data" origin="Producer" destination="Chart" />
<!-- Control signals --> <control signal="init" destination="!"/> <control signal="start" destination="!"/>
</configuration>
EMS XML DialectEMS XML Dialect
Producer Chart
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 11
Reconfiguration in EMS
Reconfiguration in EMS
There is several techniques that can be employed to alter the behavior of the system:
Use of component properties that are modifiable also at runtimeDefinition and modification of event pathsDeactivation/activation of components Deactivation/activation of data paths using gate components“Coloring” data in data driven processing Addition/removal of componentsUse of externally controllable source routing
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 12
Reconfiguration in EMS: Modification of Component Properties
Reconfiguration in EMS: Modification of Component Properties
Each component has a set of properties that both control its behavior and exhibit its state. These properties can be introspected and modified externally at runtime.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 13
Reconfiguration in EMS: Modification of
Data Paths
Reconfiguration in EMS: Modification of
Data Paths Data paths between
components can be manipulated (added, modified or removed) at runtime.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 14
Reconfiguration in EMS: Deactivation/Activation of
Components
Reconfiguration in EMS: Deactivation/Activation of
Components A component can be made
inactive, so it passes the data but does not perform any processing.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 15
Reconfiguration in EMS: Gate Components
Reconfiguration in EMS: Gate Components
A gate component passes or stops all data traffic going through it. Therefore it effectively activates or deactivates the data path that originates at it.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 16
Reconfiguration in EMS: Coloring Data
Reconfiguration in EMS: Coloring Data
A “coloring” component (Marker) is used to mark data. Other components act only on the data marked (“colored”) to be processed by them.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 17
Reconfiguration in EMS: Adding/Removing
Components
Reconfiguration in EMS: Adding/Removing
Components A component can be added to
or removed from the system.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 18
Reconfiguration in EMS: Source Routing
Reconfiguration in EMS: Source Routing
A component using source routing can send data to a recipient defined in one of its properties that can be manipulated by other
(controlling) components.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 19
Reconfiguration Components
Reconfiguration Components
Separation of concerns between functional code and reconfiguration code can be achieved by introducing a dedicated reconfiguration component (-s).
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 20
Reconfiguration with Scripting
Reconfiguration with Scripting
Scripting allows for automation of processing and for quick construction of new tests or data processing logic.
Scripting allows for easy implementation of reconfiguration in closed dynamic systems.The script control panel controlling execution of a
Jython script.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 21
WrapUpWrapUpEMS is a flexible component-based framework for building
dynamic systems.The system has been applied to building measurement and data
acquisition systems where continuous data streams are processed in real-time.
EMS allows for high level of reuse.The architecture supports separation of concerns between
reconfiguration and functional aspects.EMS features allow for agile development of new applications and
easy adaptation of existing applications to changing requirements.
A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 22
ResourcesResources EMS web site http://sdsg.fnal.gov/emsweb
EMS on-line documentationhttp://wwwtsmtf.fnal.gov/ems_javadoc2.X