Vision-based Scene Analysis on Mobile Robots using Layered POMDPs

Shiqi Zhang and Mohan SridharanDepartment of Computer Science

Texas Tech University, USA{s.zhang,mohan.sridharan}@ttu.edu

Abstract

Mobile robots are increasingly being deployed in different ap-plications as a result of progress in sensor technology and thedevelopment of state of the art algorithms for processing sen-sory inputs. A key challenge to the widespread deploymentof robots is the ability to autonomously tailor sensing and in-formation processing to the task at hand. In this paper, wedescribe on-going work towards such general-purpose pro-cessing of visual input. We pose visual processing manage-ment as an instance of probabilistic sequential decision mak-ing, and specifically as a three-layered hierarchical partiallyobservable Markov decision process (POMDP). We use atwo-layered POMDP hierarchy to enable a robot to plan a se-quence of visual operators in order to reliably and efficientlyanalyze the state of the world represented by salient regionsof interest in images (Sridharan, Wyatt, and Dearden 2008;2009). We then incorporate scene analysis as the highestlayer, where the goal is to maximize the known informationabout the environment. The POMDP models at the differentlayers are tailored to the task at hand by enabling the robotto learn the model parameters and trade-off computationalefficiency and reliability. The proposed approach is testedon robots and simulated agents in human robot interactionscenarios. Keywords: Scene analysis, Processing manage-ment, Hierarchical POMDPs, Human robot interaction.

MotivationIn recent times, high-fidelity sensors have become avail-able at moderate costs (Hokuyo 2008; Videre Design 2010),and sophisticated algorithms have been developed to processsensory inputs. Mobile robots equipped with multiple sen-sors are hence being used in applications such as disasterrescue. navigation and medicine (Casper and Murphy 2003;Pineau and Thrun 2002; Thrun 2006). However, the abil-ity to accurately sense the environment and interact reliablywith other robots and humans remains an open challenge.The following features and requirements characterize mo-bile robots operating in dynamic environments:• Features:• Non-deterministic actions: robot sensing and actuation

are unreliable.• Partial observability: the robot cannot directly observe

the state of the world. It can only update its belief of

ICAPS 2010 POMDP Practitioners Workshop, May 12, 2010,Toronto, Canada.

the state of the world by applying operators on sensoryinputs and observing the outcomes.

• Computationally intensive: algorithms processing sen-sory inputs are often computationally expensive.

• Requirements:• Performance: robots interacting with a dynamic envi-

ronment have to respond in real-time.• Reliability: though individual actions are unreliable,

high overall reliability is required.These features and requirements make it all the more chal-lenging to process images from a camera, which constitute arich and high-bandwidth source of information in compari-son to other popular sensors such as range finders. However,visual input is more sensitive to environmental changes suchas illumination, and operators (i.e. algorithms) that processvisual input are typically computationally expensive. In ad-dition, a robot can process the images using a variety of al-gorithms, each with a different level of uncertainty and com-putational complexity. One appealing solution is to retainoperators to support many tasks and then tailor the sensingand processing to the task at hand i.e. use a subset of opera-tors that can accomplish the task reliably and efficiently.

An approach based on probabilistic sequential decisionmaking, or more specifically partially observable Markovdecision processes (POMDPs) (Kaelbling, Littman, andCassandra 1998), elegantly captures the non-determinismand the partial observability of robot domains. However,POMDP formulations of many practical problems havea state space that increases exponentially, and the worstcase time complexity of POMDPs is exponential in thestate space dimensions. Even state of the art approxi-mate POMDP solvers (Smith and Simmons 2005; Ross etal. 2008) are computationally expensive for robots de-ployed in dynamic domains. Promising results have beenobtained by introducing a hierarchy in the state or actionspaces of the POMDP formulation (Pineau and Thrun 2002;Foka and Trahanias 2005). However, many such methodsrequire manual supervision for creating the hierarchy or thecorresponding POMDP models.

This paper builds on our prior work on hierarchical im-age processing (Sridharan, Wyatt, and Dearden 2008) to de-scribe a three-layered hierarchy for scene analysis. The toplayer determines where to look i.e. which scene of the 3Dspace to focus on; the intermediate layer determines what to

process i.e. which regions of images of the chosen scene toanalyze; and the bottom layer focuses on how to process i.e.what visual operators to apply on the chosen image regions.This paper therefore makes the following contributions:• It proposes a functional hierarchical decomposition for

visual processing, which (partially) ameliorates the com-putational complexity challenge and enables the robot tobetter exploit the visual input.

• It enables the robot to automatically create the POMDPmodels required at the different levels, which are thensolved to generate the overall policy.

The remainder of this paper is organized as follows. We be-gin with a brief review of some related work, followed bya description of the overall goal and the specific task ad-dressed in this paper. We then present the proposed lay-ered POMDP hierarchy. Finally, we present some proof-of-concept evaluation results on physical robots and simulatedagents in human-robot interaction scenarios.

Related WorkComputer vision research has produced several methods forusing a user-specified high-level goal to plan a pipeline ofvisual operators. However, many of these methods use de-terministic action models, with the action pre-conditions andeffects being propositions that are required to be true a pri-ori, or are made true by the application of the operator. Un-satisfactory results are handled by re-planning the operatorsequence or modifying operator parameters (Clouard et al.1999). However, the state of the system is not directly ob-servable in robot domains and actions are unreliable.

Recent research in AI planning has focused on relaxingthe limitations of classical planners to make them suitablefor practical domains. The PKS planner (Petrick and Bac-chus 2004) uses a first-order language to describe actionsin terms of their effect on the agent’s knowledge, ratherthan their effect on the world. The model is hence non-deterministic because the true state of the world is deter-mined uniquely by the actions performed, but the agent’sknowledge of that state is not. PKS captures the initial stateuncertainty and constructs conditional plans based on theagent’s knowledge. The Continual Planner (CP) (Brennerand Nebel 2008), on the other hand, interleaves planning,plan execution and monitoring, in order to postpone reason-ing about uncertain states until more information is avail-able. CP asserts that action preconditions will be met whenthat point is reached during plan execution. If the precondi-tions are not met during execution, or are met earlier, replan-ning is triggered. In robot domains, it may be necessary toaccumulate evidence by applying operators more than once,which cannot be done using PKS or CP.

Unlike the approaches based on classical planners, Li etal. (2003) modeled image interpretation as a Markov Deci-sion Process (MDP). Human-annotated images are used todetermine the reward structure, explore the state space andcompute value functions that are extrapolated to the entirestate space. Online operation involves action choices thatmaximize the value of the learned functions. Darrell (1997)used manual feedback, memory-based reinforcement learn-ing and POMDPs to learn what foveation actions to execute,

and when to execute the terminal recognition action, in orderto foveate salient body parts in an active gesture recognitionsystem. Manual feedback and extensive training trials arehowever infeasible in many mobile robot domains.

The POMDP formulation is appropriate for domains withpartially observable state and unreliable actions. However,a POMDP formulation is intractable for many practical do-mains. Pineau and Thrun (2002) proposed an elegant hierar-chical POMDP approach for behavior control of a robot as-sistant. In their hierarchy, the top level action is a collectionof simpler actions that are represented by smaller POMDPsand solved completely; planning is done in a bottom-upmanner, combining individual policies to provide the totalpolicy. All model parameters are defined over the same statespace, but the relevant space is abstracted for each POMDPusing a dynamic belief network. Similar approaches havebeen used for robot navigation (Foka and Trahanias 2005)but a significant amount of data for the hierarchy and modelcreation is hand-coded. POMDPs have also be used forvisual search based on models of the human visual sys-tem (Butko and Movellan 2008). In parallel, faster solutionmethods (Ross et al. 2008) and schemes for hierarchy dis-covery in POMDPs have been developed (Hansen and Zhou2003; Toussaint, Charlin, and Poupart 2008). POMDPs havealso been used on robots that sense and interact in spe-cific real-world tasks such as grasping (Hsiao, Kaelbling,and Lozano-Perez 2007; Saxena, Driemeyer, and Ng 2008).However, most of these methods are computationally expen-sive for applications with large state spaces. Instead, we pro-pose a three-layered hierarchy for visual scene analysis, withthe model parameters being learned by the robot.

Domain DescriptionFigure 1(a) describes the long-term goal of the on-goingproject: reliable and autonomous human-robot interaction inindoor scenarios. Achieving this goal requires, among otherthings, autonomous learning and processing management.Here, we focus on autonomous visual processing manage-ment. Consider, for instance, the task of finding a red bowli.e. addressing the query: “where is the red bowl?” or com-mand: “fetch the red bowl”. Ideally, the robot should focuson a suitable scene of the environment as shown in the topright of Figure 1(a), resulting in images such as Figure 1(b).

The images can be processed to obtain salient regions ofinterest (ROIs). The robot can process such image ROIs us-ing one or more of a large set of visual operators, for taskssuch as segmentation, object recognition and scene recon-struction. However, only a small subset of these operatorsare appropriate for any specific task—in the current exam-ple, we need operators to identify a ROI that is red andcontains a bowl. In this paper, we model sensing and in-formation processing operators as actions and use the terms“operators”, “routines” and “actions” interchangeably. Wefocus on tasks that require a sequence of visual operators tobe applied on specific images of specific scenes, so that ahuman and a robot can converse about (text-based and notlanguage-based) objects of interest.

The experimental platforms include a humanoid robot:Aldebaran Nao (Nao 2008) and a wheeled robot: Videre

(a) Overall scenario. (b) Example image of a desk. (c) Robot platforms.

Figure 1: Visual processing management: (a) Overall scenario; (b) Example image of a portion of the scene with regions-of-interest (ROIs)bounded by rectangles; (c) Robot platforms.

Design ERA-Mobi (Videre Design 2010), as shown in Fig-ure 1(c). The Nao is 58cm tall and equipped with colorcameras, ultrasound sensors and touch sensors. The ERA-Mobi is quipped with stereo and monocular cameras, andlaser range finders. These platforms have on-board process-ing (500MHz, 2.2GHz) and wi-fi capabilities, in additionto algorithms for extracting different types of informationfrom the sensory inputs. Next, we describe the hierarchicalPOMDP formulation for scene analysis.

Hierarchical POMDP FormulationThe proposed hierarchical POMDP formulation has threelayers to match the functional and cognitive requirementsof visual scene analysis. The goal here is not to create aPOMDP solver but to enable reliable and efficient visualprocessing using operators that are individually unreliable.As shown in Figure 2(a), the top layer of the hierarchy isthe high-level POMDP (“HL-POMDP”) that addresses thequestion: where to look? i.e. it determines the specificscene of the 3D space under consideration that is to be ana-lyzed next. The intermediate layer (“IL-POMDP”) analyzesimages of the chosen scene by determining the image re-gions to be processed next (what to process?). Finally, eachsalient image region (i.e. ROI) is analyzed using a lower-level POMDP (“LL-POMDP”) that determines how to pro-cess? i.e. the sequence of visual operators to be applied inorder to address the specific task (e.g. finding the red bowl).The layers of the hierarchy are described below.

Each LL-POMDP operates on a specific image ROI (i.e.salient region) and computes a sequence of visual operatorsthat would identify the desired object reliably and efficiently.Without loss of generality, consider the specific task of lo-cating all red bowls i.e. addressing the query: where arethe red bowls? In addition, assume that the robot can usethe following operators: a color operator (based on colorhistograms) that classifies the dominant color of the ROI, ashape operator (based on shape moments) that classifies thedominant shape within the ROI, and a sift operator (basedon the SIFT features (Lowe 2004)) that detects the presenceof one of the previously trained object models.

Since the true state of the system cannot be observed, therobot maintains a probability distribution over the underly-ing state (belief state). Each action considers the true under-lying state to be composed of the class labels (e.g. red(R),

green(G), blue(B) for color; circle(C), triangle(T), square(S)for shape; bowl, box, book for sift), a label to denote the ab-sence of any valid object class—empty (φ), and a label todenote the presence of multiple classes (M ). The model foreach action’s outcomes provides a probability distributionover the set composed of the corresponding class labels, thelabel empty (φ) which implies that the match probability cor-responding to the class labels is very low, and unknown (U )which means that multiple class labels are equally likely. Uis an observation whereas M is part of the state: though theyare correlated they are not the same.

Since visual operators only update belief states, we in-clude task-specific “special actions” that indicate the pres-ence or absence of the target object, or identify which un-derlying state is most likely to be the true state. Such actionscause a transition to a terminal state where no further actionsare applied. Below, without loss of generality and for easeof explanation , consider two operators: color and shapedenoted by subscripts c, s respectively; other operators areincluded in the experiments. States and observations are de-noted by superscripts a, o respectively. The POMDP tuple〈S,A, T ,Z,O,R〉 for a single image ROI is:• S : Sc × Ss ∪ term, the set of states, is a Cartesian

product of the variables describing different aspects ofthe underlying state (e.g. color, shape). It also includesa terminal state (term). Sc : {φa

c , Rac , Ga

c , Bac ,Mc},

Ss : {φas , Ca

s , T as , Sa

s ,Ms}.• A : {color, shape, AS} is the set of actions. The

first two entries are the visual operators. The rest arespecial actions. For a query such as: “is there a cir-cle in the scene?”, AS = {sFound, sNotFound} de-scribes the presence/absence of the target object, whilethe query: “what is the color of the ROI?” has AS ={sRed, sGreen, sBlue}. Special actions lead to term.

• T : S × A × S ′ → [0, 1] represents the state transitionfunction. For e.g. it is an identity matrix for visual opera-tors that do not change the state, such as color and shape.For special actions it represents a transition to term.

• Z : {φoc , R

oc , G

oc , B

oc , Uc, φ

os, C

os , T o

s , Sos , Us} is the set of

observations of all operators.• O : S × A × Z → [0, 1] is the observation function. It

is learned by the robot for the visual actions and it is auniform distribution for the special actions.

• R : S × A → <, specifies the reward, a mapping from

the state-action space to real numbers.∀s ∈ S, R(s, operators) = −β · f(ROI size) (1)

R(s, special actions) = ±100 · αFor visual operators, the cost depends on the ROI-sizeand the relative computational complexity (β for coloris twice that for shape). Special actions are assigned alarge positive (negative) reward for giving the correct (in-correct) answer. Parameter α trades-off computation andreliability. Values for α, β are tuned experimentally.

Given the belief state (i.e. a probability distribution over theunderlying state) at time t, Bt, the belief update proceeds as:

Bt+1(s′) =O(s′, at, ot+1)

∑s∈S T (s, at, s

′) ·Bt(s)P (ot+1|at, bt)

(2)

The visual planning for a single ROI involves solving thisPOMDP to find a policy (π : Bt 7→ at+1) that maximizesreward over a range of belief states. Plan execution corre-sponds to using the policy to repeatedly choose the actionwith the highest value at the current belief state, and updat-ing the belief after executing that action and getting an ob-servation. However, for a single ROI with m features (color,shape etc.) each with n values (e.g. R, G, B for color), thePOMDP has an underlying space of nm + 1; for k ROIs itis: nmk + 1. For instance, with three ROIs and two actionswe get ≈ 15000 states i.e. the planning task soon becomesintractable, even with sophisticated approximate solvers.

The proposed approach therefore models each ROI witha lower-level (LL) POMDP as described above, and intro-duces a IL-POMDP that chooses, at each step, the ROIwhose policy (generated by solving the corresponding LL-POMDP) is to be executed. The overall problem is then de-composed into one POMDP with state space 2k + 1, andk POMDPs with state space nm + 1. Without loss of gen-erality, consider an input image with two ROIs, whose IL-POMDP is given by the tuple 〈SI ,AI , T I ,ZI ,OI ,RI〉:• SI = {R1 ∧ ¬R2,¬R1 ∧ R2,¬R1 ∧ ¬R2, R1 ∧ R2} ∪

termI is the set of states. It represents the presence orabsence of the object in one or more of the ROIs, andincludes a terminal state (termI ).

• AI = {u1, u2, AIS} are the actions, where (ui) denotes

the choice of executing LL ROI Ri’s policy. For queriessuch as “is there a circle in the scene?”, the special ac-tion set AI

S = {sFoundI , sNotFoundI}. However, forqueries such as “where is the circle?”, AI

S represents thefact that one of the entries of SI is the answer. All specialactions lead to termI .

• T I is the state transition function that leads to termI forspecial actions and is an identity matrix otherwise.

• ZI = {FR1,¬FR1, FR2,¬FR2} is the set of observa-tions, which represents finding or not-finding the desiredobject when each ROI’s policy is executed.

• OI : SI ×AI ×ZI → [0, 1], the observation function, isa uniform matrix for special actions. For other actions, itis learned from the LL-POMDP policy trees.

• RI is the reward specification. For a special action, itis a large positive (negative) value if it predicts the statecorrectly (incorrectly). For other actions, it is a “cost”computed from the LL policy trees.

The required LL observation functions and rewards arelearned in an initial training phase (see the next section).Given a task (i.e. query) and images of the scene, the robotcreates a LL-POMDP for each ROI based on the availablevisual operators. These POMDPs are created in the formatused by the ZMDP package (ZMDP Code 2008) and solvedusing a point-based solver in the package (Smith and Sim-mons 2005). The LL policy trees and the query are used tocompute the appropriate observation functions and rewardsfor the IL-POMDP model at run-time. The IL-POMDP isthen solved to generate the policy that operates on the entireimage, and analyzes relevant ROIs using the correspondingLL policies, in order to identify the target object. More de-tails on the automatic belief propagation between layers canbe found in (Sridharan, Wyatt, and Dearden 2008).

Images of real-world scenes contain overlapping objectsthat would be considered as a single ROI. Such situationscan be handled using operators that split an existing ROIbased on one of more underlying features such as color orshape. However, planning with such operators is difficultbecause they change the perceptual state of the system bycreating new ROIs. Our approach plans the effects of suchROI-splitting operators by replacing the solution of severalnew POMDPs with one in which we compute the likelihoodthat one of the resultant ROIs has the desired feature value.During execution, we replan with a new POMDP after asplit. Details are not provided here due to space limitationsbut are available in (Sridharan, Wyatt, and Dearden 2009)—an execution example is provided below.

In practical settings, the target object can exist in differentlocations in the room or even in different rooms. The robotmay need to move and analyze different scenes to locate thedesired target. We therefore introduce a HL-POMDP as thetopmost layer of the hierarchy. The HL-POMDP is basedon recent work by Butko and Movellan (2008), where visualsearch was posed as the task of maximizing information gaini.e. reducing the entropy in the belief state. We represent the3D area under consideration using a discrete 2D occupancygrid—Figure 2(b). Each grid cell is associated with a proba-bility that represents the likelihood of occurrence of the tar-get object(s). For a grid with N cells, the definition of theHL-POMDP tuple 〈SI ,AH , T H ,ZH ,OH ,RH〉 is:• SH : si, i ∈ [1, N ] is the state vector, where entry si

corresponds to the event that a target is in grid cell i.• AH : ai, i ∈ [1, N ] is the set of actions, where entry ai

corresponds to the event that the robot analyzes the scenecorresponding to grid cell i.

• T H : SH × AH × S ′H → [0, 1] is the state transitionfunction. For the actions that only observe the state, itis an identity matrix. It can be extended to actions thatchange the state.

• ZH : {present, absent} is the set of observations that in-dicates the presence or absence of the target object in thegrid cell being analyzed.

• OH : SH × AH × ZH → [0, 1] is the observation func-tion. It is determined automatically as given below.

Such a formulation can represent uniform initial belief andalso situations when some prior knowledge of object loca-tion is available. The key difference is that the action utili-

(a) Three-layered hierarchy. (b) HL scenario and observation models. (c) Example image with two ROIs.

Figure 2: Visual processing management: (a) Proposed hierarchical decomposition; (b) HL grid cells and observation Gaussians; and (c)Example image with three objects enveloped in two ROIs.

ties (i.e. rewards) are based on the information likely to begained by executing that action. For the belief state Bt attime t, we use an approach similar to (Butko and Movellan2008) to define reward RH as the InfoMax objective func-tion i.e. the negative entropy of the belief:

RH(Bt) := −∑

i

Bitlog(Bi

t) (3)

The goal then is to learn a policy: π : Bt 7→ at+1 that de-creases the entropy in Bt over a planning horizon of T steps.The other major feature is the observation function specifica-tion. We set it to be a function of the expected performanceof the lower levels of the hierarchy:

OH(zi, sj , ak) = p(zi|sj , ak) ∼ N (µ, σ2) (4)

µ = fµ(sj , ak), σ2 = fσ2(O,OI |sj , ak)

where the probability of obtaining a specific observation zi

given the state sj (i.e. desired target is in grid cell j) andaction ak (i.e. focus on grid cell k) is given by a Gaussiandistribution. The mean of the Gaussian depends on the targetlocation, the grid cell being examined and the camera’s fieldof view. The variance of the Gaussian denotes the likelihoodthat the IL-POMDP and the LL-POMDPs will identify thetarget: it is hence a function of the corresponding observa-tion functions. Figure 2(b) is a pictorial representation oftargets and observation Gaussians in a 7× 7 grid.

In such a HL-POMDP representation, the number of gridcells can be large for a practical application domain, mak-ing it challenging to find policies. We currently use an ex-isting implementation of policy gradient algorithms (Buffetand Aberdeen 2009) to solve the HL-POMDP. The overalloperation then involves using the HL policy to choose anappropriate portion of the environment for analysis. Imagesof the specific scene are analyzed using the IL-POMDP andLL-POMDPs that are created at run-time. The result of thisanalysis updates the beliefs in the HL, resulting in the choiceof a grid cell for subsequent analysis. The key contributionsare: (a) the extension of the existing hierarchical approachfor image analysis to complex scenes; (b) the incorporationof high-level scene analysis to maximize information gain;and (c) the automatic belief propagation between layers thatoperate over different state spaces. The proposed hierarchi-cal approach is hence called: I-HiPPo.

Experimental Setup and ResultsIn this section, we describe the learning of the LL-POMDPcomponents, followed by the experimental results.

The model creation in the LL-POMDPs requires observa-tion and reward functions. Unlike other POMDP-based ap-plications, we enable the robot to explicitly model the uncer-tainty in the operators. Objects with known labels (e.g. “redbook“, “blue square box”) are placed at positions known tothe robot. The robot executes different visual operators onimages of the scenes to collect statistics of various actionoutcomes. These statistics are used to generate the observa-tion functions, assuming the observations are mutually in-dependent and produced by different actions. The LL ac-tion costs are a function of the relative run-time complexityof the operators and the size of the ROI (Equation 1). Therun-time complexity is experimentally determined based onthe statistics collected during observation function learning.The effect of ROI-size is modeled as a polynomial function:

f(r) = a0 +N∑

k=1

ak · rk (5)

where r is the ROI-size in pixels. The polynomial degreeand coefficients are estimated such that they best fit theperformance statistics collected during observation functionlearning (N = 1, 3, 4 for shape, color, sift respectively).An existing saliency operator (Itti, Koch, and Niebur 1998)is used to determine the ROIs. For simple scenes, the systemcan use background subtraction to extract the ROIs.

We conducted several experiments on robot platforms toevaluate our algorithm. First, we evaluated the ability ofthe lower and intermediate layers of the proposed hierar-chy (i.e. IL-POMDP and LL-POMDPs) on images of ob-jects on a desk. Consider the execution example that an-swers the query: “where are the red bowls?” on the imageshown in Figure 3(a)—this is a pictorial representation ofan actual image captured by the robot. We include the ac-tions: color, shape, sift in addition to the region-splittingactions that split input ROIs based on these features i.e.rSplitcolor, rSplitshape, rSplitsift. As stated earlier, suchsplitting actions would result in a modified state space forthe IL-POMDP. In Figure 3(a) two of the three objects over-lap, resulting in two ROIs. Since both ROIs are equally

(a) Input image. (b) Execution Step 1. (c) Execution Step 2.

(d) Execution Step 3. (e) Execution Step 4. (f) Execution Step 5.

Figure 3: Example query: “Where are the red bowls?” Pictorial representation of actual execution on a robot. Region-split operators enableseparation of overlapping objects.

likely target locations, the IL-POMDP chooses to exam-ine the smaller ROI R2 first, because of its smaller actioncosts—action u2 in Figure 3(b). The corresponding LL-POMDP runs the color operator on the ROI, leading to theoutcome of green. This outcome significantly reduces thelikelihood of finding a red bowl, and a terminal action ischosen as the best action in the next step: sNotFound. TheIL-POMDP receives the input that the target object was notfound in R2, leading to a belief update and subsequent ac-tion selection—action u1 in Figure 3(c). The LL policy ofR1 is invoked, causing color and sift to be applied in turnon the ROI. Both operators come up with outcomes of un-known because the ROI has two different colors and objects.At this point, rSplitsift is chosen as the best action and R1

is split into R1 and R3 on the basis of the sift (i.e. localgradient) features within the ROI—Figure 3(d). Our systemincludes other algorithms that can be invoked to split a ROIon the basis of color histograms or shape contours. The ex-ecution of rSplitsift is followed by the application of sifton each sub-region, leading to the observations: book andbowl in R1 and R3 respectively—Figure 3(c). The currentbeliefs are used to create and solve a new IL-POMDP forthree ROIs. The subsequent action selection in the IL (u3)results in the execution of the LL-policy of R3, whose ini-tial belief reflects the previous application of sift. Hence,color and sift are applied just once before a terminal action(sFound) is chosen—Figure 3(e). The subsequent belief up-date and action choice in the IL leads to the processing ofR1 since the goal is locate all red bowls. The terminal ac-tion sNotFound for R1 results in the action s¬R1∧¬R2∧R3

in the IL—Figure 3(f).

As seen in the example above, some actions are appliedmore than once on a ROI to accumulate evidence. In suchcases, conditional independence of the observations is en-sured by taking a new image of the scene before repeatingthe action. Though the images are not strictly independent,the independence assumption is required for the belief up-date (Equation 2) and works in practice to account for smallchanges in factors such as illumination.

Similar to our prior work, we compared our approachagainst a non-deterministic planner that has been used suc-cessfully in human-robot interaction domains: ContinualPlanning (CP) (Brenner and Nebel 2008). We also consid-ered the case where all operators were applied on the im-age ROIs (“no planning”) until the target was found or allROIs were analyzed. We considered a range of queries onscene properties, object occurrence, object location and ob-ject properties. For each query category, experiments wererun on 20 different scenes, with 20 trials for each query. Thequeries represent combinations of object properties (colors,shapes, sift features). A subset of these experiments (on an-other robot platform) were reported in (Sridharan, Wyatt,and Dearden 2008; 2009).

Figure 4(a) shows that the “joint POMDP” that operates inthe space of all ROIs and actions soon becomes intractable.The plot includes several trials on simulated scenes (withdifferent number of ROIs of varying sizes) in addition to therobot experiments. The efficiency may be improved usingfaster POMDP solvers but the intractability would still beobserved. The hierarchical approach significantly reducesthe planning time—though the computed policies are notoptimal the performance is reliable. Next, Figure 4(b) com-

(a) HiPPo vs. joint POMDP. JointPOMDP soon becomes intractable.

(b) Planning times of HiPPo vs. CP.Policy-caching makes results comparable.

(c) Planning+execution times of HiPPo,CP vs. No planning.

Figure 4: Experimental Results: comparing planning and execution times of I-HiPPo and CP against no planning, on specific scenes. I-HiPPoand CP reduce processing time.

pares the planning times of the hierarchical approach andCP as a function of the number of ROIs. The default hi-erarchical approach needs to compute policies for all ROIs(LL-POMDPs) and is hence more expensive than CP. How-ever, if no prior information is available regarding the con-tents of the different ROIs, the LL-POMDPs only differ inthe action costs. The ROI-sizes are hence discretized, andthe policies of ROIs within specific size ranges are cachedand reused. Though this policy caching results in planningtimes comparable to CP, it introduces value estimation er-rors. We estimate these errors to trade-off reliability andefficiency (Sridharan, Wyatt, and Dearden 2009).

Figure 4(c) compares the planning approaches against theno-planning approach in terms of the combined planningand execution times. Planning provides benefits even onscenes with just two ROIs, and the benefits are much moresignificant as the number of ROIs and visual operators in-crease. In these experiments the robot had modules oper-ating in parallel to analyze the sensory inputs. Therefore,though the individual operators are optimized for perfor-mance, the parallelism results in the times reported above.

As mentioned earlier, the goal is not to create a POMDPsolver but to provide a hierarchy that sequences availableoperators to achieve reliable and efficient visual process-ing. Table 1 summarizes the reliability performance overthe range of tasks, with the ground truth provided manually.With no planning, the average reliability (i.e. classificationaccuracy) is 76.67% i.e. the visual operators provide incor-rect results approximately once every four trials. ThoughCP is very efficient, it does not model the action outcomesand hence cannot achieve higher reliability than the naiveapproach of applying all available operators. Our approachexplicitly models the uncertainty and accumulates evidenceto provide higher reliability.

Approach % ReliabilityNo planning 76.67CP 76.67I-HiPPo 90.75

Table 1: Reliability of visual processingFinally, we ran experiments to evaluate the HL-POMDP’s

capabilities. The robot’s environment was discretized into

grid cells, and the robot had to locate specific objects (e.g.soccer ball, bowl). Since running extensive trials on a phys-ical robot is infeasible, we also ran experiments in a simu-lated domain that mimicked the robot’s sensing capabilities(based on the computed observation functions). The simu-lated grid currently includes the locations of certain knownobstacles (e.g. walls) and objects (e.g. tables) though this canalso be learned by the robot, using existing methods (Thrun,Burgard, and Fox 2005). The different grid sizes correspondto the different regions over which the evaluation was con-ducted. The size of each cell in the grid is set based on thefield of view of the robot’s camera (e.g. 60cm, 100cm).

Grid size % Accuracy5× 5 887× 7 9010× 10 92

Table 2: Accuracy of visual search for different grid resolutions.Table 2 summarizes the results, including ≈ 1000 trials

(for each grid size) in the simulated domain. In a trainingphase, the robot learns the LL observation functions and aHL policy for visual search. Then, the policy is evaluatedby placing target objects in different grid cells. A trial isdeemed to be successful if the policy results in identifyingthe location of the target in the correct cell. In Table 2, mostfailures correspond to cases where the target object is identi-fied but its estimated location is no more than one cell-widthaway from the actual location. Such errors are due to the gridcell resolution or the errors in robot sensing and motion. Inaddition, the policy performs significantly better than a ran-dom selection of target grid cells, converging quickly afterthe cell with the target is analyzed once. The planning timedoes vary from a few minutes to a few hours based on thesize of the grid being analyzed. We are currently investi-gating the use of other POMDP solvers and convolutionalpolicies to speed up the planning process.

ConclusionsMobile robot environments are characterized by partial ob-servability, non-determinism and computational complex-ity. Though POMDP formulations elegantly encapsulate allthese factors, they are intractable for all but the most sim-

ple problems. In this paper, we have summarized our recentwork on using a three-layered hierarchical POMDPs for vi-sual scene analysis. We have presented evaluation results ofour approach on physical robot platforms and simulated do-mains. We are currently in the process of fully integratingthe different components of the proposed approach on mul-tiple robot platforms, in order to extensively evaluate our ap-proach in challenging indoor scenarios. We are also workingon including several additional visual operators, which mayrequire that we factor the state and action spaces. Further-more, we aim to incorporate actions that change the physicalstate of the system (e.g. move an object), in order to reasonabout action affordances.

Our goal here is to enable a robot to fully utilize the richinformation encoded in camera images. Our approach is nota POMDP solver but a scheme to sequence a subset of ex-isting (unreliable) operators to achieve reliable and efficientvisual processing for the task at hand. Our ultimate aim isto enable robots to collaborate autonomously, reliably andefficiently with humans in dynamic scenarios.

AcknowledgmentsThis work was sponsored in part by the ONR awardN00014-09-1-0658.

ReferencesBrenner, M., and Nebel, B. 2008. Continual Planning andActing in Dynamic Multiagent Environments. Journal ofAutonomous Agents and Multiagent Systems.Buffet, O., and Aberdeen, D. 2009. The Factored Policy-Gradient Planner. Artificial Intelligence 173(5-6):722–747.Butko, N. J., and Movellan, J. R. 2008. I-POMDP: AnInfomax Model of Eye Movement. In The IEEE Interna-tional Conference on Development and Learning (ICDL).Casper, J., and Murphy, R. R. 2003. Human-robot In-teractions during Urban Search and Rescue at the WTC.Transactions on SMC.Clouard, R.; Elmoataz, A.; Porquet, C.; and Revenu, M.1999. Borg: A Knowledge-Based System for AutomaticGeneration of Image Processing Programs. IEEE Trans. onPattern Analysis and Machine Intelligence 21(2):128–144.Darrell, T. 1997. Reinforcement Learning of ActiveRecognition Behaviors. Technical report, Interval ResearchTechnical Report 1997-045.Foka, A. F., and Trahanias, P. E. 2005. Real-time Hierar-chical POMDPs for Autonomous Robot Navigation. In IJ-CAI Workshop on Reasoning with Uncertainty in Robotics.Hansen, E. A., and Zhou, R. 2003. Synthesis of Hier-archical Finite-State Controllers for POMDPs. In ICAPS,113–122.Hokuyo. 2008. Hokuyo Laser. http://www.hokuyo-aut.jp/products/.Hsiao, K.; Kaelbling, L. P.; and Lozano-Perez, T. 2007.Grasping POMDPs. In IEEE International Conference onRobotics and Automation (ICRA).Itti, L.; Koch, C.; and Niebur, E. 1998. A Model ofSaliency-Based Visual Attention for Rapid Scene Analy-sis. PAMI 20(11):1254–1259.

Kaelbling, L.; Littman, M.; and Cassandra, A. 1998. Plan-ning and Acting in Partially Observable Stochastic Do-mains. Artificial Intelligence 101:99–134.Li, L.; Bulitko, V.; Greiner, R.; and Levner, I. 2003.Improving an Adaptive Image Interpretation System byLeveraging. In Australian and New Zealand Conferenceon Intelligent Information Systems.Lowe, D. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of ComputerVision (IJCV) 60(2):91–110.Nao. 2008. The Aldebaran Nao Robots. http://www.aldebaran-robotics.com/.Petrick, R., and Bacchus, F. 2004. Extending theKnowledge-Based approach to Planning with IncompleteInformation and Sensing. In ICAPS, 2–11.Pineau, J., and Thrun, S. 2002. High-level Robot BehaviorControl using POMDPs. In AAAI.Ross, S.; Pineau, J.; Paquet, S.; and Chaib-draa, B. 2008.Online Planning Algorithms for POMDPs. Journal of Ar-tificial Intelligence Research 32:663–704.Saxena, A.; Driemeyer, J.; and Ng, A. Y. 2008. RoboticGrasping of Novel Objects using Vision. InternationalJournal of Robotics Research 27(2):157–173.Smith, T., and Simmons, R. 2005. Point-based POMDPAlgorithms: Improved Analysis and Implementation. InUAI.Sridharan, M.; Wyatt, J.; and Dearden, R. 2008. HiPPo:Hierarchical POMDPs for Planning Information Process-ing and Sensing Actions on a Robot. In ICAPS.Sridharan, M.; Wyatt, J.; and Dearden, R. 2009. POMDP-based Planning for Visual Processing Management on aMobile Robot. In Cognitive Vision Workshop at IROS.Thrun, S.; Burgard, W.; and Fox, D. 2005. ProbabilisticRobotics. Cambridge, USA: MIT Press.Thrun, S. 2006. Stanley: The Robot that Won the DARPAGrand Challenge. Field Robotics 23(9):661–692.Toussaint, M.; Charlin, L.; and Poupart, P. 2008. Hierarchi-cal POMDP Controller Optimization by Likelihood Maxi-mization. In Uncertainty in AI (UAI).Videre Design. 2010. Videre Design Robots.http://www.videredesign.com/robots/era mobi.htm.2008. ZMDP Planning Code. http://www.cs.cmu.edu/∼trey/zmdp/.