NRI-Small: Improved safety and reliability of robotic systems byfaults/anomalies detection from uninterpreted signals of computation graphs

Andrea Censi (coPI) Richard M. Murray (PI)

California Institute of Technology

One of the main challenges to designing robots that can operate around humans is to createsystems that can guarantee safety and effectiveness, while being robust to the numerous “nuisances”of unstructured environments, from hardware faults to software issues, erroneous calibration, andless predictable anomalies, such as tampering and sabotage. Perception algorithms work assuminga simplified model of the world and are notoriously fragile to such disturbances; because actionsare based on perception, this creates safety issues. In principle, each nuisance can be detectedusing established techniques, but, in practice, this amounts to a huge design effort, and it is notpossible to anticipate everything. However, the fact that the streams of observations and commandspossess coherence properties suggests that it would be possible to detect and mitigate many of thesedisturbances using relatively simple models that can be learned from the data. Currently, roboticsystems are developed as a directed “computation graph” using middleware such as ROS or Orocos.In these architectures, every signal is available for logging and learning. The goal of this project isto develop theoretical methods, an applicable design, and a reference implementation of a genericfaults/anomalies detection mechanism, based on black-box modeling of robotic sensorimotor signals.The system, without any prior information about the robot configuration, should be able to learna model of the robot and the environment by passive observations of the signals exposed in thecomputation graph, and, based on this model, instantiate faults/anomalies detection componentsin an augmented computation graph. The system should need zero or minimal configuration and itshould be possible to plug-and-play into any existing robotic system.

Intellectual merit To achieve this project’s goals, it will be necessary to push the state ofthe art of machine learning and system identification. While machine learning methods are nowcommonly used with massive datasets, they are not designed for high-dimensional, heterogeneousreal-time data streams, where most information is present in the dynamics of the data and thecausal relation between actions and changes in the observations. Moreover, only weak guaranteescan be given for the inference procedure. System identification, as studied in the control theoryfield, provides hard bounds and guarantees, but for very limited classes of systems that are notsuitable to represent robotic sensorimotor cascades. We aim at developing theoretically-groundedgeneric black-box sensorimotor learning methods, using the problem of faults/anomalies detectionas a litmus test of their generality across classes of sensors and actuators.

Broader impact The immediate broad impact will be a standard plug-and-play faults/anomaliesdetection layer, hopefully becoming a standard part of any robotic system. The sets of benchmarksand datasets collected along the way are expected to have a broader impact beyond the robotics field,hopefully inviting non-roboticists (statisticians, neurobiologists, cognitive development experts) tolook at realistic instances of embodied learning scenarios and problems relevant to robotics. Thepractical activities will involve the participation of undergraduate students, as part of class projectsand as paid research assistants.

1

1 Motivation

State of the art for autonomous robotics deployment in the civil sector Based on theprogress mainly seen in sensing and perception, the last years have witnessed the first commercialdeployments of completely autonomous robotics systems in the civil sector. Examples of systemscurrently deployed include Kiva’s autonomous storage room [1], autonomous waterfront opera-tions [2] and mining [3]. In these applications, the robots and the humans do not share the sameenvironment. In fact, in the Autostrad autonomous waterfront system, composed by dozens of au-tonomous trucks, the entire plant is shut down whenever an intruder crosses the barrier, primarilyfor safety purposes. We have also seen the first commercially successful domestic robots, such asiRobot’s Roomba [4], and Evolution Robotics’s Mint [5], which uses exteroceptive sensing to imple-ment advanced navigation strategies. These robots do share people’s environment, because, due tothe small mass and not powerful actuators, they cannot cause any harm (the worst that they cando is scaring the household pets).

Challenges ahead: ensuring safety and reliability in systems heavily relying on per-ception The challenge for the future is to improve the safety properties of current prototypesystems to cooperate safely with humans, sharing the same unstructured environments. Examplesinclude autonomous cars: the VisLab team [6] has managed to drive autonomously from Italy toChina using mainly vision-based algorithms; the Google cars (whose perception is based on activelidar sensing) have reportedly accumulated hundreds of thousands of miles of autonomous driving.Willow Garage’s PR2 promises to be a prototype for the future robotic housekeeper. When evalu-ating the safety of these systems, it is instructive to “think of the children”: eventually, we wouldlike to have enough faith in these systems to have an autonomous school bus, or an autonomoushousekeeper allowed to clean while the kids are playing around.

The difficulty in ensuring the safe operation of these systems depends heavily on a correctperception of the external world based on non-trivial data processing; if the perception mechanismis “confused”, the robot might perform an unsafe action. By contrast, for industrial manipulatorsone has accurate models of the system, rich proprioception, and powerful actuators that make itpossible to quickly detect and mitigate unsafe situations [7].

Dealing with a world full of unexpected nuisances Assuring safety of these systems will bea grand enterprise; in this proposal, we focus on one particular aspect: dealing with unexpectednuisances to sensory data. What makes the system fragile is the fact that a perception algorithmtypically uses a (possibly implicit) model of the world that is a simplified version of reality. Designerstry to anticipate possible nuisances based on their previous experience, but it is certainly not possibleto anticipate every possible nuisance that makes data unreliable.

For example, consider a localization/mapping algorithm using camera data and based on theassumption of a planar world. If the robot drives over a bump, the modeling assumption is violated.However, mapping algorithms are designed in such a way to minimize the effect of these rarerandom events (perhaps by relaxing the odometry error model). This is an example of nuisancethat is anticipated and mitigated. In the same scenario, imagine a fly landing over the camera. Thefly creates a very salient and very stable feature for the robot to track, therefore possibly causinga failure of the localization/mapping algorithms. Yet, it is clear that a human observing the datawould be able to decide that those pixels are unreliable, based on simple considerations of spatial

2

and temporal coherence with respect to the nearby pixels and the commands given to the actuators.We want to equip robotic systems with this sort of low level “common sense” that might allow totolerate such nuisances.

Examples of faults and anomalies in robotic systems There are a variety of phenomena,here collected under the generic labels of “faults” and “anomalies”, that would create nuisances forperception algorithms that do not take special steps to anticipate them.

1. Static calibration and configuration issues. An example would be the presence of some fixtureon the robot obstructing a sensor in a way that was not considered at design time; or anerroneous calibration procedure.

2. Hardware faults/deterioration. Eventually all mechanical components of a robot deteriorate,either suddenly or gradually. The ability to detect such events is critical to ensure the safetyof the system.

3. Innocent interference. An unexpected sensor occlusion due to an external agent (animal/child)is perhaps the most intuitive example of a nuisance that is likely to confuse the robot’sperception of the world.

4. Intentional tampering/sabotage. This is likely to be an important issue with the eventualwidespread deployment of autonomous systems—what better publicity for a neo-Luddite anti-robot group than a series of inexplicable incidents?

We remark that, in theory, all of these nuisances could be identified using standard methods, such asBayesian estimation. For example, one could certainly design a fly detector that continuously checksfor the presence of flies on the sensor frame. However, this would imply an exorbitant design effortand computational cost if done for all possible nuisances—and by definition, one cannot expect theunexpected.

The basic premise of our work is that faults/anomalies detection can be done with minimaldesign effort and in a very general way, because most nuisances significantly alter the sensorimotorproperties of the robot, and such changes can be detected with relatively simple and efficient models,directly learned from data.

2 Opportunity

Modern robotic software systems are described by computation graphs Modern roboticsystems are designed as sets of software components glued together by robotics-specific middle-ware (Fig. 1a) such as ROS [8] and Orocos [9]. The robotic system can be considered a directedgraph, where nodes are software components and labeled edges represent information flow betweensoftware components. We call this graph the “computation graph” of the robotic system. This com-putation graph abstracts middleware-specific metaphors (such as “topics”, “ports”, “sinks/sources”,“blackboards”, etc.). The computation graph is typically specified by a configuration file that de-scribes the software components configuration parameters, their interconnection, and, in the caseof distributed systems, the allocation of components to specific computation nodes.

Some of the signals in the computation graph represent sensor data (they originate from sourcesrepresenting the low-level sensor drivers), some represent commands towards the actuators (they

3

sensor

sensor

sensor

actuator

actuator

actuator

world

(a) Robotics systems can be abstracted as “computation graphs” where nodes are components and edges are signals.

sensor

sensor

sensor

actuator

actuator

actuator

y ulearned sensorimotor

model

(b) In a computation graph, every signal is available for monitoring and learning at no additional effort.

original system

augmentedsystem

(c) Computation graphs have a formal description that can be programmatically manipulated.

Figure 1: Designing robotic systems as a computation graph of software components opens up newpossibilities that are at the basis of this project.

4

go towards sinks representing the low-level actuators drivers), while the rest of the signals areintermediate results. The external world can be considered as an additional component that closesthe loop between sensors and actuators.

Every signal of interest in the computation graph is now available for learning Themain motivations for describing a robotic application as a graph of software components wereits advantages for modularity, code reuse, ease of configuration, etc. A serendipitous side effectis that in modern architectures all signals of interests are easily accessible for monitoring andlearning (Fig. 1b).

Our assumption is that, when considered as a whole, the signals of the computation graphcontain all necessary information to detect a wide class of faults and anomalies, and relativelysimple models are needed to interpret such information.

The computation graphs can be programmatically manipulated The configuration filescontain a formal description of the computation graph. Therefore, it is possible to treat the compu-tation graph as an entity that can be programmatically manipulated (Fig. 1c). This allows to reasonabout robotic systems at a more abstract level, and it is a key enabler for realizing a completelygeneric plug-and-play solution for fault detection.

Opportunity: system-level learning and inference These recent changes in the design ofrobotic systems enable us to think a new level of applications, whose primary object that theymanipulate is the robotic system as a whole: by observing the signals in the computation graph,one can learn about the robot sensorimotor cascade; by manipulating the computation graph, onecan add new functionality to the system. In this project, we apply these ideas to the case offault/anomalies detection.

5

3 System overview

Our goal is developing theoretical methods, a reference design, and an example implementation ofa generic learning-based faults/anomalies detection and mitigation system. The main design goalis to not impose any additional design or configuration effort on the user.

3.1 User manual

Ideally, using the system would be as easy as 1-2-3:

1. Collect logs of the system in nominal conditions.

2. Run the following command:

$ robustify -i system.launch –logs collected_logs/ -o robust_system.launch

where system.launch is a configuration file describing the original computation graph; collected_logscontains the collected logs; and the output file robust_system.launch describes an augmentedsystem, with added software components for faults/anomalies detection.

3. Enjoy a more robust robotic system. The added components automatically detect faults/anomalies,and censor/replace the affected data, or otherwise mitigate the effects on the rest of the system.

A diagram of how this is supposed to work is shown in Fig. 2.

originalgraph new graph

learnedmodels

world

robustifylogs

original system

augmented system

introduced components for faults/anomalies detection

Figure 2: Prototype idea

3.2 Behind the scenes

The theory and methods to create such a system do not exist yet; we briefly describe the mainchallenges that must be confronted.

1) Learning from uninterpreted signals of computation graphs The inputs to the systemare a description of the robot’s computation graph and a set of logs taken during the operationof the robot. Based on these logs, and no other prior information about the robot, the systemestimates a model of the robot sensorimotor cascades; i.e., how the robot commands (u) influencethe observations (y). The model is used for fault detection in the following phase.

The challenges of this phase include:

6

• Selecting a generic class of models that can represent a large class of sensor and actuators. Forthis, we will be extending our previous work on bootstrapping [10, 11, 12], which showed thatrelatively simple models can approximate the dynamics of a variety of sensors such as range-finders, cameras, and field samplers. The main extensions needed are dealing with articulatedbodies and second-order dynamics (for example, to apply the methods to systems such asquadrotors).

• Selecting which subset of signals to consider for modeling the sensor-sensor correlations orthe sensori-motor interaction. It is likely to be unfeasible to consider all observation signals yas candidates for learning, as the data volume can be in the range of ≈1Gb/s for roboticplatforms with many rich sensors. However, the data available for learning is expected to behighly redundant. For example, a dead pixel can be detected both from the raw camera dataas well as the undistorted camera image, and both signals are part of y. We will explore aprincipled way to choose a relevant subsets of signals to use, with the goal of having somethingtreatable, for example by using submodular optimization [13].

2) Augmenting the computation graph with fault/anomalies detection components Inthis second phase, based on the models learned, the initial computation graph is modified as toinclude monitoring and filtering modules. Each monitoring module is assigned one of more signalsto monitor, and outputs a “reliability” signal which describes whether the data follows the modelpreviously learned. A filtering module is assigned to a particular signal and it has the ability to markdata as invalid, in a way compatible with the semantics of the data. The preexisting componentsare reconfigured to read from the filtered streams.

The challenges of this phase include:

• Given a global CPU budget, describing the computational resources the system designer de-cides to be worth using for faults/anomaly detection, decide in a principled way which signalsare worth monitoring.

• Distribute the computation across different hosts in a way compatible with communicationconstraints and local CPU budgets.

3) Faults/anomalies mitigation Assume that monitoring components are embedded in thecomputation graph and correctly detecting faults, we have to decide what to do with them. Up tonow, all of the techniques described did not actually modify any signal or influence the behavior ofthe robot. There are two ways in which this could happen. The simplest way is for the filter modulesto mark the data as invalid based on the reliability measure given by the monitoring components.For this to happen, the existing messages in the computation graph must support the semantics of“invalid data”.

More generally, one would want to activate some emergency plan. While it is certainly possibleto identify in the computation graph which messages drive the actuators, it is not safe in generalto just shut off the actuators. For example, if the robot platform was a quadrotor, one might wantto activate some safe landing if a fault is detected. In conclusion, while the previous phases can beengineered considering the system a black box, and no knowledge of the actual robot or task, thisfunctionality cannot be realized in a pure “plug-and-play” manner as it requires a slight modificationof the existing components.

7

The main challenges of this phase are:

• Defining conventions for expressing the concept of “invalid data” for all common messages,and understand what is the right level of granularity, both in space (should all the messagebe marked as invalid, or is it useful to mark only part of the sensor data as invalid?) and intime (should one reason message-by-message or work at a coarser time scale?).

• Define messages and protocols that allow to trigger a “safe shutdown” or “emergency mode”,in such a way that is maximally compatible with the conventions in the existing systems.

4 Technical approach

Our design goal is to implement a “one-click” system that adds faults/anomalies detection andmitigation to an arbitrary robotic system. We do not want the user to spend additional design orconfiguration effort. This ambitious goal dictates that various forms of learning to be a central partof this project. At the same time, because we want to improve the safety of robotic systems, itis crucial to establish some form of theoretical guarantee regarding the faults/anomalies that canbe detected. To succeed, we will have to integrate the flexibility of machine learning methods, therigor of control theory, and the ambitions of developmental learning/epigenetic robotics.

4.1 Related work

Classical control theory approaches to faults/anomalies detection and isolation In theestablished approach to fault detection and isolation (see, e.g., [14, 15]) one defines faults of sen-sors and actuators as deviation of their behavior from a nominal model, assumed to represent the“healthy” system, which can be given or identified from the data. Classical control theory primarilystudies very limited classes of models, such as linear time-invariant systems, for which theoreticalguarantees can be given, in terms of the convergence of the learning procedure as well as rejectingthe effects of nuisances [16]. Unfortunately, these models are not powerful enough to represent thedynamics of robotic sensorimotor cascades.

Universal approximators in machine learning and robotics Machine learning providesmany examples of methods whose goal is to be “universal approximators”. For example, variousflavors of deep belief networks [17, 18] are designed as universal approximators of data distributions,but they do not deal with actions. Established techniques such as reinforcement learning [19] gives amathematical framework to deal with actions in unknown state spaces, but it does not explain howto build stateful representations from raw sensory values and partially observable environments.Predictive State Representations [20] is an effort to introduce a flexible idea of “state” induced bytests, which are manually designed features of the system’s dynamics. Learning the dynamics of anarbitrary dynamical system is extremely difficult in general, with no additional constraints on thesystem, as shown by the bounds of the complexity of the optimal agent given by Hutter [21].

In the robotics community, there have been several attempts to learning dynamics with methodsmore or less tailored to the robotic context [22, 23, 24, 25]. All of these methods cannot start fromlow-level sensorimotor interaction (i.e., the “pixel level”) and include a good amount of domainknowledge or previous knowledge of the body schema.

8

Related work in developmental learning & epigenetic robotics A somewhat distinct lineof research is present in the developmental learning and epigenetic robotics community. The statedgoal is to obtain generic artificial intelligence agents that learn to act in the world starting fromlow-level sensorimotor interaction, and unknown sensors and actuators. The focus is to obtainexplicit generative models of the actuator/world/sensor dynamics, rather than simply an optimalpolicy; in this context, direct learning of a policy as in reinforcement learning would be seen as aninadmissible shortcut.

A bootstrapping agent builds progressively more complicated models of the world starting fromscratch [26]. The first steps consist in understanding from the observed statistics of the data what is aplausible grouping of observations by sensor [27], and recovering the geometry of the sensors [28, 29];with these steps one can transform scrambled streams into spatially coherent sensor snapshots. Thenext steps consist in understanding what is the effect of actions on the sensor readings, by progres-sively abstracting the continuous dynamics into (possibly discrete) primitives [30, 31]. The agent isoften assumed to be able to explore the world, according to some form of intrinsic motivation [32].

While several aspects of what would be a complete bootstrapping architecture have been demon-strated anecdotally, no agent that can provably work for general robotic sensors/actuators has beendesigned. Part of our long-term research plan is “adding the control theory” in bootstrapping, toobtain reliable methods that can be routinely used by roboticists. Faults/anomalies detection is alow hanging fruit for bootstrapping, which allows us to harden the theory while being immediatelyuseful to the robotics community.

Other related work in faults/anomalies detection in robotics Relevant techniques were in-vestigated in two European projects under the 6th and 7th Framework Programme, called Phriends [33](Physical Human-Robot Interaction: DepENDability and Safety), and its successor Saphari [34](Safe and Autonomous Physical Human-Aware Robot Interaction). In these projects, the researchersfocused on the safe physical interaction of a human and a manipulator in an industrial context. Thecore of the issue is detecting potentially dangerous events, such as the collision of the robotic armwith the human body. The collisions are monitored using anomaly detection, by monitoring ifthe joint measurements differ with respect to the nominal dynamics. If a collision is detected, therobot task is suspended, and the the dynamics of the arm is changed to absorb the collision energy.This is possible as industrial manipulators have very precise models, and full observations of thestate through the joint measurements. Ideally, our generic faults/anomalies detection scheme wouldobtain similar performance, but relying on models which are entirely learned from scratch.

4.2 Premise: the dynamics of heterogeneous sensorimotor cascades are moresimilar than one might expect

The key observations that allows generic learning models is that the dynamics of robotic sensori-motor cascades (the series of actuators and sensors, considered together) are more similar than onemight expect. In previous work [10], we have shown this with reference to three “canonical roboticsensors”: field-samplers, range-finders, and cameras. Let ys be the raw readings of such sensors;where s ∈ S represents an index over the sensels (short for sensory elements) which ranges over thesensel space S, and ys is, respectively, field intensity, distance reading, and luminance at sensel s.At this low level, the three sensor dynamics are quite similar, as shown in Table 1, which gives anexpression for ys as a function of the robot velocities.

9

This similarity justifies looking for models that can represent the dynamics of potentially anyrobotic sensor. Learning with no prior information is what enables a zero-configuration approach tofault detection: all information about the system is in the data itself.

Table 1: Continuous dynamics of canonical robotic sensors

sensor S continuous dynamics (far from occlusions)field sampler R3 ys =

∑i∇iy

svi +∑

i (s×∇ys)iωi

camera S2 ys = µs∑

i∇iysvi +

∑i (s×∇ys)iωi

range-finder S2 ys =∑

i(∇i log ys − s∗i )vi +

∑i (s×∇ys)iωi

In this table, v ∈ R3 and ω ∈ R3 are the robot linear and angular velocities; s is a continuous index ranging over the“sensel space” S; y(s) is the raw sensor data (field intensity value, pixel luminance, or range reading); ∇i is the i-thcomponent of the spatial gradient with respect to s; µ(s) is the nearness (inverse of the distance to the obstacles).

4.3 Bootstrapping models of sensorimotor cascades

In a series of previous works, we have studied several different models that could represent thedynamics of canonical robotic sensors. These models occupy different points of the design space,as they have various tradeoffs in terms of the computational complexity and the required priorknowledge about the system.

Bilinear dynamics systems (BDS) In particular, we proposed the class of bilinear dynam-ics systems (BDS) as a possible candidate in [10]. Given a discretized observation vector y ={yi}1≤i≤ny ∈ Rny , and generic commands u ∈ Rnu , the dynamics are assumed to be of the form:

yit =∑

j

∑aM

ijay

jtu

at . (BDS) (1)

The model is parametrized by a ny × ny × nu tensor M ija.

Bilinear gradient dynamics systems (BGDS) In [12] we studied bilinear gradient dynamicssystems (BGDS), a more constrained class of models that uses explicitly the spatial organizationof the sensels. The observations are assumed to be a smooth function from the sensels space:y = {ys}s∈S. The dynamics depend on the spatial gradient ∇y. The model is parametrized by twotensor fields Gds

a and Bsa that model a general bilinear/affine relation between the derivative of the

observations and the commands:

yst =∑

d

∑a(G

dsa ∇dy

st +Bs

a) uat , (BGDS) (2)

This model is slightly more complicated, and less general, but much more efficient, as the complexityis linear in ny instead of quadratic.

Diffeomorphisms dynamical systems (DDS) In [35] we studied diffeomorphisms dynamicalsystems (DDS). These are discrete-time dynamical systems, where the state is yk : S→ R, and thecommands u belong to a finite alphabet U = {u1, . . . ,u|U|}. Each command uj is associated to adiffeomorphism ϕj ∈ Diff(S). The transition function from the state yk at time k to the state yk+1

is given byysk+1 = y

ϕ(s)k , (DDS) (3)

10

Examples of detected faults/anomaliesModels Relevant trade-o!s

Single sensel statistics

Inter-sensel statistics

Sensorimotorinteraction

stateful

instantaneousquantitative

qualitative

More learning data neededMore prior information neededMore powerful

Actuator performancedeterioration

Gross fault of actuators

Dead pixelsLess learning data neededLess prior information neededLess powerful

Random readings

Mirror placed in frontof camera

able to capture the difference between faults/anomalies and the normal regime. In this project,rather than aim for maximum generality, we want to characterize a hierarchy of models, orderedby their representation power (Fig. 3). To the best of our knowledge, formal methods to establishthis (partial) order are not available and they will have to be developed along the way. In parallel,we aim to enumerate classes of faults/anomalies, characterizing what are the simplest models thatallow to detect them.

Very simple models can still be useful Here, we use the word “model” to refer to whateverdescription of the system that allows to make predictions on the observations. We remark thatmodels could be simpler than expected.

As a simple example, let yt = {yit}n

i=1 be the observations at time t, where yit is a single sensor

reading, perhaps the pixel of a camera. Consider the model

yit ∼ Uniform([yi, yi]), (4)

which describes the data as generated independently by a simple random distribution. Learning ofthis model consists in fitting the upper and lower bounds [yi, yi]. This model is clearly not complexenough to fully represent the sensor, however it allows to make prediction and detect simple faults,such as a dead pixel: yi

t = 0 for all t ≥ t0 is a very unlikely sequence to be generated by themodel (4). If, instead of a dead pixel, the camera had a faulty pixel giving randomly oscillatingmeasurements, this model would be not powerful enough. To detect such fault, one could use themodel

corr(yit, y

jt ) = Rij ,

which describe the pairwise correlation between readings. This model is not generative, but justdescriptive of the typical sequences in the data. This model allows to detect a larger class of faults,and implement mitigation strategies, such as in-painting of dead pixels.

More complex faults/anomalies need even more complex models which take into account thesensorimotor interaction. Suppose that the camera is mounted upside down (or a saboteur puts amirror in front of the camera). To detect this anomaly, one should use a predictive model of theobservations based on the actuators commands, such as (1)–(3).

4.5 Fault/anomalies mitigation

Suppose that, based on the learned models, the system has identified a fault or anomaly in the data.We want to address the question of what should be happening now in the system. There are severalpossibilities that could be implemented. To mitigate faults/anomalies concerning the sensor data,the simplest actions would consist in altering the information flow:

• Censoring the data. For temporary faults, perhaps the data can be simply censored by drop-ping the messages (estimation algorithms are usually designed to handle large temporal gapsin the data).

• Marking the data. A “reliability” measure could be attached to the data. This could be doneboth at the level of the messages or with more granularity, for example by marking as invalidthe single reading of the sensor. This implies that the messages are augmented to add thisinformation, and that the original system components can use this reliability indication.

11

able to capture the difference between faults/anomalies and the normal regime. In this project,rather than aim for maximum generality, we want to characterize a hierarchy of models, orderedby their representation power (Fig. 3). To the best of our knowledge, formal methods to establishthis (partial) order are not available and they will have to be developed along the way. In parallel,we aim to enumerate classes of faults/anomalies, characterizing what are the simplest models thatallow to detect them.

Very simple models can still be useful Here, we use the word “model” to refer to whateverdescription of the system that allows to make predictions on the observations. We remark thatmodels could be simpler than expected.

As a simple example, let yt = {yit}n

i=1 be the observations at time t, where yit is a single sensor

reading, perhaps the pixel of a camera. Consider the model

yit ∼ Uniform([yi, yi]), (4)

which describes the data as generated independently by a simple random distribution. Learning ofthis model consists in fitting the upper and lower bounds [yi, yi]. This model is clearly not complexenough to fully represent the sensor, however it allows to make prediction and detect simple faults,such as a dead pixel: yi

t = 0 for all t ≥ t0 is a very unlikely sequence to be generated by themodel (4). If, instead of a dead pixel, the camera had a faulty pixel giving randomly oscillatingmeasurements, this model would be not powerful enough. To detect such fault, one could use themodel

corr(yit, y

jt ) = Rij ,

which describe the pairwise correlation between readings. This model is not generative, but justdescriptive of the typical sequences in the data. This model allows to detect a larger class of faults,and implement mitigation strategies, such as in-painting of dead pixels.

More complex faults/anomalies need even more complex models which take into account thesensorimotor interaction. Suppose that the camera is mounted upside down (or a saboteur puts amirror in front of the camera). To detect this anomaly, one should use a predictive model of theobservations based on the actuators commands, such as (1)–(3).

4.5 Fault/anomalies mitigation

Suppose that, based on the learned models, the system has identified a fault or anomaly in the data.We want to address the question of what should be happening now in the system. There are severalpossibilities that could be implemented. To mitigate faults/anomalies concerning the sensor data,the simplest actions would consist in altering the information flow:

• Censoring the data. For temporary faults, perhaps the data can be simply censored by drop-ping the messages (estimation algorithms are usually designed to handle large temporal gapsin the data).

• Marking the data. A “reliability” measure could be attached to the data. This could be doneboth at the level of the messages or with more granularity, for example by marking as invalidthe single reading of the sensor. This implies that the messages are augmented to add thisinformation, and that the original system components can use this reliability indication.

11

This similarity justifies looking for models that can represent the dynamics of potentially anyrobotic sensor. Learning with no prior information is what enables a zero-configuration approach tofault detection: all information about the system is in the data itself.

Table 1: Continuous dynamics of canonical robotic sensors

sensor S continuous dynamics (far from occlusions)field sampler R3 ys =

�i ∇iy

svi +�

i (s ×∇ys)i ωi

camera S2 ys = µs�

i ∇iysvi +

�i (s ×∇ys)i ω

i

range-finder S2 ys =�

i(∇i log ys − s∗i )vi +

�i (s ×∇ys)i ω

i

In this table, v ∈ R3 and ω ∈ R3 are the robot linear and angular velocities; s is a continuous index ranging over the“sensel space” S; y(s) is the raw sensor data (field intensity value, pixel luminance, or range reading); ∇i is the i-thcomponent of the spatial gradient with respect to s; µ(s) is the nearness (inverse of the distance to the obstacles).

4.3 Bootstrapping models of sensorimotor cascades

In a series of previous works, we have studied several different models that could represent thedynamics of canonical robotic sensors. These models occupy different points of the design space,as they have various tradeoffs in terms of the computational complexity and the required priorknowledge about the system.

Bilinear dynamics systems (BDS) In particular, we proposed the class of bilinear dynam-ics systems (BDS) as a possible candidate in [10]. Given a discretized observation vector y ={yi}1≤i≤ny ∈ Rny , and generic commands u ∈ Rnu , the dynamics are assumed to be of the form:

yit =

�j

�aM

ijay

jt u

at . (BDS) (1)

The model is parametrized by a ny × ny × nu tensor M ija.

Bilinear gradient dynamics systems (BGDS) In [12] we studied bilinear gradient dynamicssystems (BGDS), a more constrained class of models that uses explicitly the spatial organizationof the sensels. The observations are assumed to be a smooth function from the sensels space:y = {ys}s∈S. The dynamics depend on the spatial gradient ∇y. The model is parametrized by twotensor fields Gds

a and Bsa that model a general bilinear/affine relation between the derivative of the

observations and the commands:

yst =

�d

�a(G

dsa ∇dy

st + Bs

a) uat , (BGDS) (2)

This model is slightly more complicated, and less general, but much more efficient, as the complexityis linear in ny instead of quadratic.

Diffeomorphisms dynamical systems (DDS) In [35] we studied diffeomorphisms dynamicalsystems (DDS). These are discrete-time dynamical systems, where the state is yk : S → R, and thecommands u belong to a finite alphabet U = {u1, . . . , u|U|}. Each command uj is associated to adiffeomorphism ϕj ∈ Diff(S). The transition function from the state yk at time k to the state yk+1

is given byys

k+1 = yϕ(s)k , (DDS) (3)

9

Second phase (approximately 1.5 years) In the second phase, there will be explicit effortsto publicize the project with the ROS community; we will create an official website for increasedvisibility. A data repository will be setup on Caltech-owned servers.

Post-project (5 years) At the end of the project, we will freeze the complete software envi-ronment used as VMWare images for long-term preservation. The software is likely to be activelymaintained/extended in the near future, and the collected data kept available on Caltech servers.

Permanent storage and availability Caltech maintains a secure institutional repository thatprovides access to material that is of long-term value to the academic community, including datasets.The Caltech Collection of Open Digital Archives (CODA) (http://coda.caltech.edu) makes use ofthe open source EPrints software that is installed on three high-availability Linux servers managed bythe Library and utilizing fiber-attached SAN. CODA metadata conform to international standards.Documents and metadata are backed up to disk and tape, with off-site tape storage in case ofdisaster. All records are given Persistent URLs that do not change with system upgrades. CaltechCODA is indexed by all major Internet search engines. All publications (articles, theses, technicalreports, conference papers) and software packages produced as part of the project will be archivedin Caltech CODA persistently.

uj �= 0 ⇒ yj �= 0

2

Figure 3: Hierarchy of models

where ϕ is the diffeomorphism associated to the command given at time k. Learning genericdiffeomorphisms is typically much more expensive; the method that we use in [35] has computationalcomplexity which is cubic in the sensor resolution.

Extensions needed for this project We have shown that these models are useful to representthe dynamics of a variety of robotic sensors. Their biggest limitation so far is how they representthe robot platform dynamics. They work well if the commands u are kinematic velocities appliedto the same rigid body where the sensor is attached. They cannot represent second-order dynamics(necessary for flying vehicles) nor articulated bodies (necessary for personal robots). These arenon-trivial extensions that we intend to pursue in this project.

4.4 Hierarchies of models

Selection criteria for models to be used in faults/anomalies detection Choosing the rightclasses of models is a central point of this project. In machine learning and system identification,one usually classifies classes of models according to two criteria: representation power (a moregeneral class is better), and the computational resources necessary for learning and inference. Inthis application the evaluation criteria are slightly different. For example, a more powerful model ismore likely to require more data to be learned; for example, learning the models (2)–(3) requires datastreams whose length is on the order of hours to learn camera models from scratch (i.e. withoutcalibration information). Here, the problem is not the complexity of the learning phase, whichcould possibly happen offline, but rather the storage of a large amount of data. The computationalconstraints are stricter for what regards the inference procedure, as it must happen online, in parallelwith the original process.

Characterizing hierarchy of models by their representation power Therefore, one shouldselect the simplest model possible, that can be consistently learned from small amounts of data, andthat supports a computationally inexpensive inference procedure. At the same time, the models

11

must be complex enough to represent the phenomena of interest; in this case, the models must beable to capture the difference between faults/anomalies and the normal regime. In this project,rather than aim for maximum generality, we want to characterize a hierarchy of models, orderedby their representation power (Fig. 3). To the best of our knowledge, formal methods to establishthis (partial) order are not available and they will have to be developed along the way. In parallel,we aim to enumerate classes of faults/anomalies, characterizing what are the simplest models thatallow to detect them.

Very simple models can still be useful Here, we use the word “model” to refer to whateverdescription of the system that allows to make predictions on the observations. We remark thatmodels could be simpler than expected.

As a simple example, let yt = {yit}ni=1 be the observations at time t, where yit is a single sensorreading, perhaps the pixel of a camera. Consider the model

yit ∼ Uniform([yi, yi]), (4)

which describes the data as generated independently by a simple random distribution. Learning ofthis model consists in fitting the upper and lower bounds [yi, yi]. This model is clearly not complexenough to fully represent the sensor, however it allows to make prediction and detect simple faults,such as a dead pixel: yit = 0 for all t ≥ t0 is a very unlikely sequence to be generated by themodel (4). If, instead of a dead pixel, the camera had a faulty pixel giving randomly oscillatingmeasurements, this model would be not powerful enough. To detect such fault, one could use themodel

corr(yit, yjt ) = Rij ,

which describe the pairwise correlation between readings. This model is not generative, but justdescriptive of the typical sequences in the data. This model allows to detect a larger class of faults,and implement mitigation strategies, such as in-painting of dead pixels.

More complex faults/anomalies need even more complex models which take into account thesensorimotor interaction. Suppose that the camera is mounted upside down (or a saboteur puts amirror in front of the camera). To detect this anomaly, one should use a predictive model of theobservations based on the actuators commands, such as (1)–(3).

4.5 Fault/anomalies mitigation

Suppose that, based on the learned models, the system has identified a fault or anomaly in the data.We want to address the question of what should be happening now in the system. There are severalpossibilities that could be implemented. To mitigate faults/anomalies concerning the sensor data,the simplest actions would consist in altering the information flow:

• Censoring the data. For temporary faults, perhaps the data can be simply censored by drop-ping the messages (estimation algorithms are usually designed to handle large temporal gapsin the data).

• Marking the data. A “reliability” measure could be attached to the data. This could be doneboth at the level of the messages or with more granularity, for example by marking as invalidthe single reading of the sensor. This implies that the messages are augmented to add thisinformation, and that the original system components can use this reliability indication.

12

• Filling-in the data. For some kinds of faults one can fill-in the data. For example, if a flydoes land on the camera sensor, the interested area could be filled in by using the learnedmodels, either considering sensel-sensel correlation (this is called “in-painting” in computervision) or using the learned dynamics to predict what the data would have been based onprevious images and the observed commands.

In principle, the faults/anomalies detection system can also manipulate the commands. For example,it can freeze the robot should there be a major fault detected. However, this is not always a safechoice—think for example to the case of a flying platform. More generally, one might want someplatform-specific fault recovery mode to be triggered. This necessarily requires some modificationsof the original system components. A systems engineering focus of this project is how to design sucha system; how messages could be augmented to include reliability data, and what minimal interfaceshould planning modules expose to participate in this faults/anomalies mitigation system.

5 Preliminary results

We have already implemented some of these ideas in a prototype system for faults detection [36].

Intrinsic fault detection using sensel usefulness In this work, we explored the idea thatsome hardware faults can be described even without reference to a nominal model. The principleis that a sensel is “faulty” if it is “useless”, in the sense that it gives no information on the stateof the system. For example, consider a sensor with some obstructed readings. The obstructedreadings can be considered “faulty” (even if, to be precise, they are working) because they give noinformation about the robot motion (Fig. 4). For the i-th reading, we can formalize a notion of“usefulness”, as the distance between p(u|y) and p(u|y\{yi}). If the distance is 0, the i-th readingcan be considered faulty.

Example of computation graph manipulation Figure 5 shows an example of the manipulationof a computation graph with minimal impact on the original components. The computation graphis augmented with a bootstrapping agent, estimating a model of the sensorimotor cascade accordingto the BDS model (1), and computing the usefulness signal. A filtering component is introducedbetween the robot sensors/actuators drivers and the controller. Based on the estimated usefulnesssignal, it marks some of the readings as invalid. The original controller receives the sanitizedreadings.

commands not influencing the observations

observations not revealing

the commands

x uaya

ub......yb

“world”

y

Figure 4: Some faults in sensorimotor cascades can be given a characterization independent of anominal model, from an information-theory perspective. Completely “faulty” sensels (yb in thefigure) are those that do not provide any information about the commands applied to the system.

13

sanitized y

y yu

controller

robot

filteringmodule

bootstrappingagent

panicsignal

senselusefulness

u uy

robot

controller

Figure 5: Example manipulation of the computation graph

Prototype demonstration Figure 6 shows an example demonstration of the system. The read-ings of a range-finder on a mobile robot are partially obstructed by the robot’s antennas. Thebootstrapping agent can detect those anomalies, as those readings are not predictive of the robotmotion. Based on the computed sensel usefulness, some of the readings are marked as invalid. Toavoiding committing to hard thresholds, there is an adaptive threshold. The controller communi-cates to the filtering component with a “panic signal” that the data does not allow any feasibleplan; the filtering component progressively lowers the threshold on the sensel usefulness until allobstructed readings are recognized as invalid.

wi-fi antennascover the range finder

antennas vibrate during motion

antenna antenna

IR readings

sensel #

Sensel usefulness

(a) Robotic platform and sensel usefulness

sensels

time (s)

senselsmarked

as invalid

panic mode normal operation

adaptive sensel blocking starts

operating

time (s)

(b) Demonstration of faults mitigation

Figure 6: Prototype of generic anomalies/faults detection mechanism from zero prior informationabout the system.

Previous accomplishments based on NSF support

Control Design for CyberPhysical Systems Using Slow Computing (0931746). RichardMurray is the PI for a 2009-2012 CPS grant. The goal of this project is to develop new, systematicmethods for the design of control systems that can work in the presence of slow computing elements.In the first two years of the project we have developed new techniques for "bootstrapping" controllersbased on experimental data [10, 11, 12], designing delay-based controllers that approximate arbitrary

14

transfer functions [37], and exploring the use of learned data for generating trajectories for complexsystems [38, 39]. Two graduate students have been supported under this grant to date.

The “Preliminary results” presented in the previous section were developed with the support ofthis grant.

6 Methodology

Robotic systems and collected data We plan to use a variety of robotic systems to demonstratethe generality of the approach. The Kuka YouBot is the simplest robot that supports mobilemanipulation. It will be equipped with cameras and range-finders. The Skybotix helicopter is aninexpensive flying platform. A flying platform gives the added difficulty of having to use second-order models, and for which the fault recovery program cannot be to just stop the actuators. TheTurtleBot, equipped with a Kinect camera, is meant to be a platform with which undergraduateproject participants can get started quickly.

These platforms will be used to gather a large set of heterogeneous data logs, both in nominalconditions, as well with various faults/anomalies introduced. These logs will be eventually madeavailable as part of a benchmark, with annotated ground-truth, in ROS format as well as otherformats more suitable to long-term preservation.

Software products Several software components will be produced; these can be divided inroughly three independent groups:

• Components for learning and inference from sensorimotor data. This will be a library ofmodels for low-level sensorimotor learning. This part of the software is meant to be largelyindependent of ROS, with standard dependencies, and reusable as part of future projects.

• Plug-and-play fault/anomalies detection system (i.e., the robustify command introduced inSection 3.1). This part is supposed to be largely ROS-specific and have a simple, well-defineduser interface that requires no expert knowledge of the system’s internal workings.

• Benchmarks, data annotation and evaluation scripts. A lasting contribution of this projectwill be the creation of benchmarks for the problem of faults/anomalies detection. This in-cludes annotation tools for attaching ground-truth faults/anomaly data, utilities for generatingsynthetic faults, and evaluation scripts.

The evaluation of a particular model will involve learning from the variety of collected logs andtesting the model prediction against the annotated ground-truth. The number of platformsand volume of data makes it necessary to process the data in a distributed way by cloudcomputing. We are currently evaluating services such as PiCloud that offer a simple API toaccess the Amazon EC2 platform.

As discussed in the Data Management Plan, we plan to distribute software and data as free software(e.g., GPL/BSD), and to package it according to the conventions used by the ROS project.

15

7 Plan & milestones

Year 1

Theory Definition of a reference universe of models of interest. Develop a theory to frame theintuition of “hierarchy of models”. Develop a parallel hierarchy of reference faults/anomaliesto consider in the analysis.

Platforms and data Assembling robotic platforms and implementation of reasonably complexbehaviors based on already available software components, partly as group projects of anadvanced robotics class. Initial collection of data logs in nominal and faulty conditions.

Software Prototype of a library of generic learning modules based on the proposers’ previous re-search, as well as established machine learning methods. Prototype of fault/anomaly detectionbenchmarks.

Year 2

Theory Extension of the previous theory on bootstrapping to cover all systems of interest (e.g., in-clude articulated bodies and second-order systems). Extend/adapt relevant machine learningmethods. Give a solid theoretical foundations to benchmarks and performance measures.

Platforms and data Consolidation of the datasets informed by the theory developments. Firstend-to-end demonstrations of the system, with the first users/testers being the students of theyearly robotics class.

Software Prototype of plug-and-play fault detection/mitigation system. Design extensions to com-mon data formats/messages necessary for fault mitigation.

Dissemination The prototype software and benchmarks are available on the project’s website.Involve the robotics community for early feedback on design.

Year 3

Theory Work out the theoretical guarantees that the system offers in terms of data needed forlearning and bounds on the faults/anomalies detection reliability.

Platforms and data Update datasets as appropriate given the theory developments.

Software Plug-and-play fault/anomaly detection system ready for public release.

Dissemination Organize workshops in robotics conferences (ICRA, IROS, RSS) to disseminatethe theory and technology developed in the project. Publicize the faults/anomaly detectionproblem and the availability of benchmarks/datasets to the machine learning communitythrough participation in relevant conferences.

16

References

[1] “Kiva’s automated material handling order fulfillment system.”. (link).

[2] H. Durrant-Whyte, D. Pagac, B. Rogers, M. Stevens, and G. Nelmes, “An autonomous strad-dle carrier for movement of shipping containers: From research to operational autonomoussystems,” IEEE Robotics and Automation Magazine, vol. 14, no. 3, 2007. DOI.

[3] S. Vasudevan, F. Ramos, E. Nettleton, and H. Durrant-Whyte, “A mine on its own,” IEEERobotics and Automation Magazine, vol. 17, no. 2, 2010. DOI.

[4] “iRobot Roomba.”. (link).

[5] “Evolution Robotics Mint.”. (link).

[6] M. Bertozzi, A. Broggi, E. Cardarelli, R. Fedriga, L. Mazzei, and P. Porta, “The VIAC expe-dition toward autonomous mobility,” 2011. DOI.

[7] S. Haddadin, A. Albu-Schaffer, and G. Hirzinger, “Requirements for Safe Robots: Measure-ments, Analysis and New Insights,” The International Journal of Robotics Research, vol. 28,no. 11-12, 2009. DOI.

[8] M. Quigley, K. Conley, B. P. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng,“ROS: an open-source Robot Operating System,” in ICRA Workshop on Open Source Software,2009.

[9] “The Orocos Project.”. (link).

[10] A. Censi and R. M. Murray, “Bootstrapping bilinear models of robotic sensorimotor cascades,”in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),2011. (link).

[11] A. Censi and R. M. Murray, “Uncertain semantics, representation nuisances, and necessaryinvariance properties of bootstrapping agents,” in Joint IEEE International Conference onDevelopment and Learning and Epigenetic Robotics, 2011.

[12] A. Censi and R. M. Murray, “Bootstrapping sensorimotor cascades: a group-theoretic perspec-tive,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011.(link).

[13] A. Krause and C. Guestrin, “Submodularity and its applications in optimized informationgathering,” ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 4, 2011.

[14] S. Simani, C. Fantuzzi, and R. J. Patton, Model-based Fault Diagnosis in Dynamic SystemsUsing Identification Techniques. Advances in Industrial Control, Springer, 2002.

[15] S. X. Ding, Model-based fault diagnosis techniques. Springer, 2008.

[16] P. Ioannou, Adaptive Control Tutorial (Advances in Design and Control). SIAM, 2006.

[17] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,”Neural Computation, vol. 18, no. 7, 2006. DOI.

17

[18] Y. Bengio, “Learning deep architectures for AI,” Foundations and Trends in Machine Learning,2009. DOI.

[19] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998.

[20] B. Boots and G. J. Gordon, “Predictive state temporal difference learning,” in Advances inNeural Information Processing Systems (NIPS), 2011. (link).

[21] M. Hutter, Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Proba-bility. Berlin: Springer, 2004. (link).

[22] J. Bongard, V. Zykov, and H. Lipson, “Resilient Machines Through Continuous Self-Modeling,”Science, vol. 314, no. 5802, 2006. DOI.

[23] Z. Kira, “Modeling cross-sensory and sensorimotor correlations to detect and localize faults inmobile robots,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS), 2007. DOI.

[24] J. Sturm, C. Plagemann, and W. Burgard, “Body schema learning for robotic manipulatorsfrom visual self-perception,” Journal of Physiology, 2009.

[25] J. Ko and D. Fox, “Learning GP-BayesFilters via Gaussian process latent variable models,”Autonomous Robots, vol. 30, no. 1, 2010. DOI.

[26] B. Kuipers, “An intellectual history of the Spatial Semantic Hierarchy,” Robotics and cognitiveapproaches to spatial mapping, vol. 38, 2008.

[27] D. Pierce and B. Kuipers, “Map learning with uninterpreted sensors and effectors,” ArtificialIntelligence, vol. 92, no. 1-2, 1997. DOI.

[28] J. Stober, L. Fishgold, and B. Kuipers, “Sensor map discovery for developing robots,” in AAAIFall Symposium on Manifold Learning and Its Applications, 2009. (link).

[29] J. Modayil, “Discovering sensor space: Constructing spatial embeddings that explain sensorcorrelations,” in Proceedings of the International Conference on Development and Learning(ICDL), 2010. DOI.

[30] J. Stober and B. Kuipers, “From pixels to policies: A bootstrapping agent,” in Proceedings ofthe International Conference on Development and Learning (ICDL), 2008. DOI.

[31] J. Stober, L. Fishgold, and B. Kuipers, “Learning the sensorimotor structure of the foveatedretina,” in Proceedings of the International Conference on Epigenetic Robotics (EpiRob), 2009.(link).

[32] S. Singh, R. L. Lewis, A. G. Barto, and J. Sorg, “Intrinsically Motivated Reinforcement Learn-ing: An Evolutionary Perspective,” IEEE Transactions on Autonomous Mental Development,vol. 2, no. 2, 2010. DOI.

[33] “PHRIENDS: Physical Human-Robot Interaction: DepENDability and Safety.”. (link).

[34] “The Saphari project: Safe and Autonomous Physical Human-Aware Robot Interaction.”.(link).

18

[35] A. Censi and R. M. Murray, “Learning diffeomorphism models of robotic sensorimotor cas-cades,” in ICRA, 2012. (submitted). (link).

[36] A. Censi, M. Hakansson, and R. M. Murray, “Fault detection and isolation from uninterpreteddata in robotic sensorimotor cascades,” in ICRA, 2012. (submitted). (link).

[37] S. Sojoudi, J. Lavaei, and R. M. Murray, “Fault-tolerant controller design with applications inpower systems and synthetic biology,” in American Control Conference (ACC), 2011.

[38] A. Censi, S. Han, S. B. Fuller, and R. M. Murray, “A bio-plausible design for visual attitudestabilization,” in Proceedings of the 48th IEEE Conference on Decision and Control, 2009. DOI.

[39] S. Han, A. Censi, A. D. Straw, and R. M. Murray, “A bio-plausible design for visual pose sta-bilization,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),2010. (Link is to the extended technical report). DOI.

19