Proceedings of the SIGDIAL 2018 Conference, pages 151–160,Melbourne, Australia, 12-14 July 2018. c©2018 Association for Computational Linguistics

151

Language-Guided Adaptive Perception for Efficient GroundedCommunication with Robotic Manipulators in Cluttered Environments

Siddharth PatkiUniversity of Rochester

Rochester, NY, 14627, USAspatki@ur.rochester.edu

Thomas M. HowardUniversity of Rochester

Rochester, NY, 14627, USAthoward@ece.rochester.edu

Abstract

The utility of collaborative manipulatorsfor shared tasks is highly dependent onthe speed and accuracy of communicationbetween the human and the robot. Therun-time of recently developed probabilis-tic inference models for situated symbolgrounding of natural language instructionsdepends on the complexity of the repre-sentation of the environment in which theyreason. As we move towards more com-plex bi-directional interactions, tasks, andenvironments, we need intelligent percep-tion models that can selectively infer pre-cise pose, semantics, and affordances ofthe objects when inferring exhaustivelydetailed world models is inefficient andprohibits real-time interaction with theserobots. In this paper we propose a modelof language and perception for the prob-lem of adapting the configuration of therobot perception pipeline for tasks whereconstructing exhaustively detailed modelsof the environment is inefficient and in-consequential for symbol grounding. Wepresent experimental results from a syn-thetic corpus of natural language instruc-tions for robot manipulation in exampleenvironments. The results demonstratethat by adapting perception we get signifi-cant gains in terms of run-time for percep-tion and situated symbol grounding of thelanguage instructions without a loss in theaccuracy of the latter.

1 INTRODUCTION

Perception is a critical component of an intelli-gence architecture that converts raw sensor obser-vations to a suitable representation for the task

that the robot is to perform. Models of environ-ments vary significantly depending on the applica-tion. For example, a robotic manipulator may needto model the objects in its environment with theirsix degree-of-freedom pose for grasping and dex-terous manipulation tasks, whereas a self-drivingcar may need to model the dynamics of the envi-ronment in addition to domain-specific semanticssuch as stop signs, sidewalks and pedestrians etc.to safely navigate through the environment.

The ability of robots to perform complex tasksis linked to the richness of the robot’s worldmodel. As inferring exhaustively detailed worldrepresentations is impractical, it is common to in-fer representations which are highly specific to thetask that the robot is to perform. However, incollaborative domains as we move towards morecomplex bi-directional interactions, manipulationtasks, and the environments, it becomes unclearhow to best represent the environment in order tofacilitate planning and reasoning for a wide distri-bution of tasks. As shown in the Figure 1, mod-eling the affordance between the chips can and itslid would be unnecessary for the task of picking upthe mustard sauce bottle and vice versa. Inferringexhaustively detailed models of all of the objectsin the environment is computationally expensiveand inconsequential for the individual tasks, andinhibits real-time interaction with these collabora-tive robots.

The utility of collaborative manipulators is alsohighly dependent on the speed and accuracy ofcommunication between the human operator andthe robot. Natural language interfaces provide in-tuitive and muti-resolution means to interact withthe robots in shared realms. In this work, we pro-pose learning a model of language and percep-tion that can adapt the configurations of the per-ception pipeline according to the task in order toinfer representations that are necessary and suffi-

152

Figure 1: On the left is an image showing the Baxter Research Robot in a cluttered tabletop environmentin the context of collaborative human-robot tasks. A perception system that does not use the context ofthe instruction when interpreting the observations would inefficiently construct detailed world modelthat is only partially utilized by the symbol grounding algorithm. On the right are the adaptively inferredrepresentations using our proposed language perception model for the instructions, “pick up the leftmostblue gear” and “pick up the largest red object” respectively.

cient to facilitate planning and grounding for theintended task. e.g. the top-right image in the Fig-ure 1 shows the adaptively inferred world modelpertaining to the instruction “pick up the leftmostblue gear” which is different than the one inferredfor the instruction “pick up the largest red object”.

2 BACKGROUND

The algorithms and models presented in this paperspan the topics that include robot perception andnatural language understanding for human-robotinteraction. Perception is a central problem inthe the field of situated robotics. Consequently, aplenty of research has focused on developing rep-resentations that can faciliate planning and reason-ing for highly specific situated tasks. These repre-sentations vary significantly depending on the ap-plication, from two-dimensional costmaps (Elfes,1987), volumetric 3D voxel representations (Hor-nung et al., 2013, 2010), primitive shape basedobject approximations (Miller et al., 2003; Hueb-ner and Kragic, 2008) to more rich representationsthat model high level semantic properties (Galindoet al., 2005; Pronobis and Jensfelt, 2012), 6 DOFpose of the objects of interest (Hudson et al., 2012)or affordances between objects (Daniele et al.,2017). Since inferring exhaustively detailed worldmodels is impractical, one solution is to designperception pipelines that infer task relevant worldmodels (Eppner et al., 2016; Fallon et al., 2014).Inferring efficient models that can support reason-

ing and planning for a wide distribution of tasksremains an open research question.

Natural language interfaces provides intutiveand multi-resolution means to interact with thecollaborative robots. Contemporary models(Tellex et al., 2011; Howard et al., 2014; Boular-ias et al., 2015; Matuszek et al., 2013) frame theproblem of language understanding as a symbolgrounding problem (Harnad, 1990). Specifically,of inferring correspondences between the linguis-tic constituents of the instruction and the symbolsthat represent perceived entities in the robot’s en-vironment such as objects and regions or desiredactions that the robot can take. (Howard et al.,2014) frames this problem as one of inference in aprobabilistic graphical model called a DistributedCorrespondence Graph (DCG). This model lever-ages the hierarchical structure of the syntacticallyparsed instruction and conditional independenceassumptions across constituents of a discrete sym-bol space to improve the run-time of probabilis-tic inference. Other variations include the Hier-archical DCG (Propp et al., 2015) and AdaptiveDCG (Paul et al., 2016) to further improve therun-time performance in cluttered environmentswith known environment models. Recently, thesemodels have been used to augment perception andrepresentations. (Daniele et al., 2017) uses DCGfor supplementing perception with linguistic in-formation for efficiently inferring kinematic mod-els of articulated objects. (Duvallet et al., 2014;

153

Hemachandra et al., 2015) use DCG to augmentthe representations by exploiting information inlanguage instruction to build priors over the un-known parts of the world. A limitation of cur-rent applications of probabilistic graphical modelsfor natural language symbol grounding is that theydo not consider how to efficiently convert obser-vations or measurements into sufficiently detailedrepresentation suitable for inference. We proposeto use DCG for the problem of adapting the per-ception pipelines for inferring task optimal repre-sentations.

Our work is most closely related to that of(Matuszek et al., 2013). Their work presents anapproach for jointly learning the language andperception models for grounded attribute learn-ing. Their model infers the subset of objectsbased on color and shape which satisfy the at-tributes described in the natural language descrip-tion. Similarly, (Hu et al., 2016) proposes deeplearning based approach to directly segment ob-jects in RGB images that are described by the in-struction. We differentiate our approach by ex-panding the diversity and complexity of percep-tual classifiers, enabling verbs to modify objectrepresentations, and presenting an end-to-end ap-proach to representation adaptation and symbolgrounding using computationally efficient proba-bilistic graphical models. In the following sectionswe introduce our approach to adapting perceptionpipelines, define our experiments, and present re-sults against a suitable baseline.

3 TECHNICAL APPROACH

We describe the problem of understanding naturallanguage instructions as one of probabilistic in-ference where we infer a distribution of symbolsthat express the intent of the utterance. The mean-ing of the instruction is taken in the context of asymbolic representation (Γ), observations (zt) anda representation of the language used to describethe instruction (Λ). A probabilistic inference us-ing a symbolic representation that is described bythe space of trajectories X (t) that the robot maytake takes the form of equation:

x(t)∗ = arg maxx(t)∈X(t)

p(x(t)|Λ, zt) (1)

Solving this inference problem is computation-ally intractable when the space of possible trajec-tories is large. Contemporary approaches (Tellex

et al., 2011; Howard et al., 2014) frame this prob-lem as a symbol grounding problem, i.e. inferringthe most likely set of groundings (Γs∗) given asyntactically parsed instruction Λ = {λ1, ..., λm}and the world model Υ.

Γs∗ = arg maxγ1...γn∈Γs

p(γ1...γn|Λ,Υ) (2)

Here, the world model Υ is a function of theconstructs of the robot’s perception pipeline (P ),and the raw observations zt.

Υ ≈ f(P, zt) (3)

The groundings Γs are symbols that representobjects, their semantic properties, regions derivedfrom the world model, and robot actions and goalssuch as grasping the object of interest or navigat-ing to a specific region in the environment. The setof all groundings Γs = {γ1, γ2, ..., γn} is called asthe symbol space. Thus the symbol space formsa finite space of interpretations in which the in-struction will be grounded. The DCG is a prob-abilistic graphical model of the form described inequation 2. The model relates the linguistic com-ponents λi ∈ Λ to the groundings γj ∈ Γs throughthe binary correspondence variables φij ∈ Φ.DCG facilitates inferring the groundings at a par-ent phrase in the context of the groundings at itschild phrases Φci. Formally, DCG searches forthe most likely correspondence variables Φ∗ in thecontext of the groundings γij , phrases λi, childcorrespondences Φci and the world model Υ bymaximizing the product of individual factors.

Φ∗ = arg maxφij∈Φ

|Λ|∏i=1

|Γs|∏j=1

p(φij |γij , λi,Φci,Υ) (4)

Inferred correspondence variables Φ∗ representthe expression of the most likely groundings Γs∗.The factors in the equation 4 are approximated bylog-linear models Ψ:

Φ∗ = arg maxφij∈Φ

|Λ|∏i=1

|Γs|∏j=1

Ψ(φij , γij , λi,Φci,Υ)

(5)Model training involves learning the log-linear

factors from the labeled data relating phraseswith true groundings. Inference process involvessearching for the set of correspondence variablesthat satisfy the above equation. The run-time per-formance of probabilistic inference with the DCG

154

is positively correlated with the complexity ofthe world model Υ. This is because the size ofthe symbolic representation Γs increases with thenumber of objects in the environment representa-tion. Recognizing that some objects (and the sym-bols based on those objects) are inconsequential tothe meaning of the instruction, we consider the op-timal representation of the environment Υ∗ as onewhich is necessary and sufficient to solve equa-tion 5. Thus we hypothesize that the time to solveequation 6 will be less than that for the equation 5.

Φ∗ = arg maxφij∈Φ

|Λ|∏i=1

|Γs|∏j=1

Ψ(φij , γij , λi,Φci,Υ∗)

(6)Typically the environment model Υ is com-

puted by a perception module P from a set ofobservations z1:t = {z1 . . . zt}. In cluttered en-vironments we assume that inferring an exhaus-tively detailed representation of the world that sat-isfies all possible instructions is impractical forreal-time human-robot interactions. We proposeusing language as mean to guide the generationof these necessary and sufficient environment rep-resentations Υ∗ in turn making it a task adaptiveprocess. Thus we define Υ∗ inferred from a singleobservation as:

Υ∗ ≈ f(P, zt,Λ) (7)

where P denotes the perception pipeline of therobotic intelligence architecture. We adapt DCGto model the above function by creating a novelclass of symbols called as perceptual symbols ΓP .Perceptual symbols are tied to their correspondingelements in the perception pipeline. i.e. to the vi-sion algorithms. Since this grounding space is in-dependent of the world model Υ, the random vari-able used to represent the environment is removedfrom equation 5. We add a subscript p to denotethat we are reasoning in the perceptual groundingspace.

Φ∗ = arg maxφij∈Φ

|Λ|∏i=1

|ΓP |∏j=1

Ψ(φij , γij , λi,Φci) (8)

Equation 8 represents the proposed modelwhich we refer to as the language-perceptionmodel (LPM). It infers the symbols that informthe perception pipeline configurations given a nat-ural language instruction describing the task. The

space of symbols ΓP describe all possible configu-rations of the perception pipeline. For example, asshown in the Figure 1, for the instruction “pick upthe leftmost blue gear”, we may need elements inour pipeline that can detect blue objects and gears.Detecting green objects, spherical shapes, or six-dimensional pose of the chips can object would notbe necessary to generate the symbols necessary forthe robot to perform the instruction.

We assume that the perception pipeline (P ) ispopulated with a set of elements E = {E1, ..., En}such that each subset Ei ∈ E represents a setof algorithms that are responsible for inferring aspecific property of an object. e.g. a red color-detection algorithm would be a member of thecolor detector family responsible for inferring thesemantic property “color” of the object. While asix degree-of-freedom (DOF) pose detection al-gorithm would be a member of the pose detec-tor family. More generally, E can be defined as:E = {e1, e2, ..., em}. With these assumptions, wedefine our independent perceptual symbols as:

ΓIDP = {γei |ei ∈ E} (9)

We can imagine that these symbols would beuseful to ground simple phrases such as ”the redobject” or ”the ball” etc. where the phrases re-fer to a single property of the object. In the morecomplicated phrases such as ”the red ball” or ”theblue box” we have a joint expression of proper-ties. i.e. we are looking for objects which maxi-mize the joint likelihood p(red, sphere|o). Sincethese properties are independent we can infer themseparately for every object ok ∈ O. However, wecan represent the above joint likelihood expressionas p(red, sphere) = p(red)p(sphere|red). Inthis case, it allows conditioning the evaluation ofsphere detection on only a subset of objects whichwere classified as being red by the red detector. Toadd this degree of freedom in the construction ofthe perception pipeline, we define additional set ofsymbols which we refer to as conditionally depen-dent perceptual symbols:

ΓCDP = {γei,ej |ei, ej ∈ E ; i 6= j} (10)

The expression of the symbol γei,ej refersto running the element ei from the perceptionpipeline on the subset of objects which were clas-sified positive by the element ej . Finally the com-plete perceptual symbol space is:

ΓP = {ΓIDP ∪ ΓCD

P } (11)

155

4 EXPERIMENTAL DESIGN

Herein with our experiments we demonstrate theutility of our language perception model for thetask of grounded language understanding of themanipulation instructions. As shown in Figure 3the process involves two distinct inferences: Infer-ring the perceptual groundings given a languageinstruction ( eq. 8 ), and inferring high level mo-tion planning constraints given the language andthe generated world model ( eq. 5 and eq. 6 ). Inthis section we describe our assumptions, and de-fine the distinct symbolic representations used inour experiments for each of the above tasks. Wethen discuss our instruction corpus and the detailsof the individual experiments.

Robot and the EnvironmentFor our experiments a Rethink Robotics BaxterResearch Robot is placed behind a table. Therobot is assumed to perceive the environment us-ing a head-mounted RGB-D sensor. Robot’s workspace is populated using objects from the stan-dard YCB dataset (Berk Calli, 2017), custom 3Dprinted ABS plastic objects, and multicolored rub-ber blocks. We define the world complexity interms of the number of objects present on the tablein the robot’s field of view. The world complexityranges from 15 to 20 in our experiments.

Symbolic RepresentationThe symbolic representation defines the space ofsymbols or meanings in which the natural lan-guage instruction will be grounded or understood.As mentioned before we define two distinct setsof symbols in our experiments. ΓP defines the setof perceptual symbols which are used by the lan-guage perception model, and ΓS defines the set ofsymbols which are used by the symbol groundingmodel.

ΓP is a function of the elements E of the per-ception pipeline. The elements ei ∈ E in ourperception pipeline are selected such that they canmodel the robot’s environment with a spectrum ofsemantic and metric properties which will be nec-essary towards performing symbol grounding andplanning for all of the instructions in our corpus.In our experiment we define E as:

E = {C ∪ G ∪ L ∪ B ∪ R ∪ P} (12)

Here, C is a set of color detectors, G is a set ofgeometry detectors, L is a set of object label de-tectors, B is a set of bounding box detectors, R

is a set of region detectors, and P is a set of posedetectors.

C = { cdi | i ∈ color}G = { gdi | i ∈ geometry}L = { ldi | i ∈ label}B = { bdi | i ∈ bbox}R = { rdi | i ∈ region}P = { pdi | i ∈ pose}

(13)

where color = {red, green, blue, white, yellow, or-ange}, geometry = {sphere, cylinder, cuboid}, la-bel = {crackers box, chips can, pudding box, mas-ter chef can, bleach cleanser, soccer ball, mustardsauce bottle, sugar packet}, bbox = {non-oriented,oriented }, region = {left, right, center}, pose = {3 DOF, 6 DOF }. Given the perception elementsdefined in the equation 13, we define the indepen-dent perceptual groundings ( ΓID

P ) previously de-fined in equation 9 as follows:

ΓC = {γcdi | cdi ∈ C}ΓG = {γgdi | gdi ∈ B}ΓL = {γldi | ldi ∈ L}ΓB = {γbdi | bdi ∈ B}ΓR = {γrdi | rdi ∈ R}ΓP = {γpdi | pdi ∈ P}

(14)

ΓIDP = { ΓC ∪ ΓG ∪ ΓL ∪ ΓB ∪ ΓR ∪ ΓP} (15)

We define the conditionally dependent percep-tual groundings ( ΓCD

P ) previously defined in equa-tion 10 as following:

ΓGC = {γ(gdi,cdj) | gdi ∈ G, cdj ∈ C}ΓLC = {γ(ldi,cdj) | ldi ∈ L, cdj ∈ C}ΓPC = {γ(pdi,cdj) | pdi ∈ P, cdj ∈ C}ΓPG = {γ(pdi,gdj) | pdi ∈ P, gdj ∈ G}ΓPL = {γ(pdi,ldj) | pdi ∈ P, ldj ∈ L}

(16)

ΓCDP = { ΓGC ∪ ΓLC ∪ ΓPC ∪ ΓPG ∪ ΓPL} (17)

These symbols provide us the ability to selec-tively infer desired properties in the world. Abovepresented independent and conditionally depen-dent symbols together cover the complete spaceof perceptual symbols used by the LPM:

ΓP = {ΓIDP ∪ ΓCD

P } (18)

156

Algorithmic details of the percepion elementsare as follows : A single RGB point cloud is fedin as a raw sensor observation to the pipeline. ARANSAC (Fischler and Bolles, 1981) based 3Dplane detection technique is used for segmentingthe table-top and the objects. HSV colorspace isused for detecting colors. RANSAC based modelfitting algorithms form the core of the geometrydetectors. A 4 layer ( 256 - 128 - 64 - 32 ) feed for-ward neural network is trained to infer the seman-tic labels of the objects. It takes in a 32 x 32 RGBimage and infers a distribution over 8 unique YCBobject classes. A PCA based oriented boundingbox estimation algorithm is used to approximatethe 6 DOF pose for the individual objects. Algo-rithms are implemented using OpenCV and PCLlibrary (Rusu and Cousins, 2011).

The space of symbols for the symbol ground-ing model is similar to the representation definedin (Paul et al., 2016). This space uses symbolsto represent objects in the world model (ΓO), se-mantic object labels (ΓL), object color(ΓC), objectgeometry(ΓG) regions in the world(ΓR), spatial re-lationships (ΓSR) and finally high level planningconstraints that define the end goal (ΓPC). Theinferred constraints forms an input to a planningalgorithm that can then generate trajectories to ac-complish the desired task. Thus the complete sym-bol space for the symbol grounding model is:

ΓS = { ΓO∪ΓL∪,ΓC∪ΓG∪ΓR∪ΓSR∪ΓPC} (19)

CorpusFor training and testing the performance of thesystem we generate an instruction corpus usingthe linguistic patterns similar to that described in(Paul et al., 2016). The corpus used in our experi-ments consists of 100 unique natural language in-structions. Details of the grammar extracted fromthis corpus is described in the appendix. Eachinstruction describes a manipulation command tothe robot while referring to the objects of inter-est using their semantic or metric properties. e.g.“pick up the green cup” or “pick up the biggestblue object”. If multiple instances of the sameobjects are present in the robot’s work space thenthe reference resolution is achieved by using spa-tial relationships to describe the object of interest.e.g.“the leftmost blue cube” or “rightmost red ob-ject” etc.

As shown in Figure 2, the instructions in thecorpus are in the form of syntactically parsed trees.

Figure 2: Syntactically parsed tree for the instruc-tion ”pick up the leftmost red block”.

Each instruction is generated in the context of aspecific table-top object arrangement. Thus eachinstruction is associated with a pair of RGB-D im-age. A total of 10 unique table-top arrangementsare used to generate the set of 100 instructions.

One copy of the corpora is annotated for train-ing LPM using (ΓP ) while another for trainingthe symbol grounding model using (ΓS). The an-notations for LPM corpus are selected such thatthat the perception pipelines configured using theannotated groundings would generate the optimalworld representations that are necessary and suf-ficient to support grounding and planning for thegiven tasks.

We have instructions with varying complexityin our corpus. The instruction complexity fromthe perception point of view is quantified in termsof the total number of perceptual groundings ex-pressed at the root level. e.g. “pick up the ball”is relatively a simple instruction with only sin-gle grounding expressed at the root level, while“pick up the blue cube and put the blue cube nearthe crackers box” is a more complicated instruc-tion having seven groundings expressed at the rootlevel. This number was found to vary in the rangeof one to seven in our corpus.

Experiments and Metrics

We structure our experiments to validate twoclaims. The first claim is that adaptively infer-ring the task optimal representations reduce theperception run-time by avoiding exhaustively de-tailed uniform modeling of the world. The sec-ond claim is that reasoning in the context of theseoptimal representations also reduces the inferencerun-time of the symbol grounding model. An out-line of our experiments is illustrated in Figure 3.In the first experiment, we study the root-level in-ference accuracy of LPM ( groundings expressedat the root level of the phrase ) as a function of thegradual increase in the training fraction. For each

157

Figure 3: Comparative Experiments: Boxes in the bottom half denote the baseline framework whereasthe boxes in the top half represent the proposed framework. Filled boxes enclose the variables that arecompared in the experiments.

value of training fraction in the range [ 0.2 , 0.9 ]increasing with a step of 0.1, we perform 15 vali-dation experiments. The training data is sampledrandomly for every individual experiment. Addi-tionally, we perform a leave-one-out cross valida-tion experiment. We use the inferences generatedby the leave-one-out cross validation experimentsas inputs to drive the adaptive perception for eachinstruction.

In the second experiment, we compare the cu-mulative run-time of LPM inference ( eq. 8 ) andadaptive perception ( T1+T2 ) against the run-timefor complete perception ( T4 ) - our baseline, forincreasingly complex worlds.

In the third experiment, we compare the infer-ence time of the symbol grounding model reason-ing in the context of the adaptively generated op-timal world models ( T3, eq. 6 ) against the infer-ence time of the same model but when reasoningin the context of the complete world models ( T5,eq. 5 ). We also check whether the planning con-straints inferred in both cases match the groundtruth or not. Experiments are performed on a sys-tem running a 2.2 GHz Intel Core i7 CPU with 16GB RAM.

5 RESULTS

This section presents the results obtained for theabove mentioned three experiments. Specifically,the learning characteristics of LPM, the impactof LPM on the perception run-time, and the im-pact the adaptive representations on the symbolgrounding run-time.

Leftmost graph in the Figure 4 shows the resultsof the first experiment. We can see that the infer-ence accuracy grows as a function of a gradual in-crease in the training data. A growing trend is anindicator of the language diversity in the corpus.

Mean inference accuracy starts at 39.25%±5 for k= 0.2 and it reaches 84% for leave-one-out crossvalidation experiment ( k = 0.99 ).

Middle graph in the Figure 4 shows the resultof the second experiment. We can clearly see thatthe run-time for complete perception grows withthe world complexity while the run-time of adap-tive perception stays nearly flat and is significantlylower in all cases. Since the adaptive perceptionrun-time varies according to the task, we see big-ger error bars. The drop in the complete percep-tion run-time for world complexity of 20 is jus-tifiable as the run-time of our geometry detectionalgorithm was proportional to the size of the indi-vidual objects, and all of the objects for that exam-ple world were smaller than other examples.

World T4 ( sec ) T1 + T2 ( sec )Complexity baseline proposed15 4.40 ± 0.05 0.96 ± 0.0716 4.99 ± 0.02 1.33 ± 0.3417 5.40 ± 0.06 1.11 ± 0.1118 5.82 ± 0.18 1.51 ± 0.2620 4.17 ± 0.05 1.11 ± 0.25Mean 5.03 ± 0.07 1.20 ± 0.21

Table 1: Adaptive perception run-time comparedagainst complete perception run-time. Deviationmeasures are 95% confidence interval values.

Rightmost graph in the Figure 4 shows the resultof the third experiment. It shows that the symbolgrounding run-time when reasoning in the contextof detailed world models( Υ ) grows as a func-tion of the world complexity. However, it is sig-nificantly lower when reasoning in the context ofadaptively generated world models ( Υ∗ ) and isindependent of the world complexity.

The achieved run-time gains are meaningful

158

Figure 4: Graph on the left shows the LPM inference accuracy as a function of gradual increase in thetraining fraction. In the middle is the bar chart comparing the run-time for complete perception ( T4 )against the cumulative run-time of LPM inference and adaptive perception ( T1 + T2 ). Finally, on theright is a bar chart comparing the run-time of symbol grounding when reasoning in the context of theadaptively generated optimal representations ( T3 ) against when reasoning in the context of exhaustivelydetailed world models ( T5 ). The error bars indicate 95% confidence intervals.

World T5 ( ms ) T3 ( ms )complexity baseline proposed15 167 ± 12 21 ± 216 191 ± 16 23 ± 517 222 ± 12 22 ± 318 245 ± 12 27 ± 420 325 ± 10 20 ± 5Mean 214 ± 12 23 ± 4

Table 2: Per phrase symbol grounding run-time inms ( rounded to the nearest integer ) using adaptiverepresentations compared against the same whenusing complete representations. Deviation mea-sures are 95% confidence interval values.

only if we do not incur a loss in the symbolgrounding accuracy. Table 3 shows the impact ofLPM on SG accuracy and summarizes the gains.

Perception Avg. Avg. SGType TP ( sec ) TSG ( ms ) Acc.Complete 5.03 ± 0.07 214 ± 12 63%Adaptive 1.20 ± 0.21 23 ± 4 66%Ratio 4.19 9.30

Table 3: Impact of LPM on average percep-tion run-time per instruction (TP ), average symbolgrounding run-time per instruction (TSG), and thesymbol grounding accuracy.

6 CONCLUSIONS

Real-time human-robot interaction is critical forthe utility of the collaborative robotic manipula-

tors in shared tasks. In scenarios where inferringexhaustively detailed models of all the objects isprohibitive, perception represents a bottleneck thatinhibits real-time interactions with collaborativerobots. Language provides an intuitive and multi-resolution interface to interact with these robots.While recent probabilistic frameworks have ad-vanced our ability to interpret the meaning of com-plex instructions in cluttered environments, theproblem of how language can channel the interpre-tation of the raw observations to construct worldmodels which are necessary and sufficient for thesymbol grounding task is not extensively stud-ied. Our proposed DCG based Language Percep-tion Model, demonstrates that we can guide per-ception using language to construct world mod-els which are suitable for efficiently interpretingthe instruction. This provides run-time gains interms of both perception and symbol grounding,thereby improving the speed with which collabo-rative robots can understand and act upon humaninstructions. In ongoing and future work we areexploring how language can aid efficient construc-tion of global maps for robot navigation and ma-nipulation by intelligently sampling relevant ob-servations from a set of observations.

7 ACKNOWLEDGMENTS

This work was supported in part by the NationalScience Foundation under grant IIS-1637813 andthe New York State Center of Excellence in DataScience at the University of Rochester.

159

References

James Bruce Aaron Walsman Kurt Konolige Sid-dhartha Srinivasa Pieter Abbeel Aaron M DollarBerk Calli, Arjun Singh. 2017. Yale-cmu-berkeleydataset for robotic manipulation research. vol-ume 36, page 261 268.

Abdeslam Boularias, Felix Duvallet, Jean Hyaejin Oh,and Anthony (Tony) Stentz. 2015. Grounding spa-tial relations for outdoor robot navigation. In 2015IEEE International Conference on Robotics and Au-tomation (ICRA).

Andrea F. Daniele, Thomas M. Howard, andMatthew R. Walter. 2017. A multiview approachto learning articulated motion models. In Proceed-ings of the International Symposium of Robotics Re-search (ISRR).

F. Duvallet, M.R. Walter, T.M. Howard, S. Hemachan-dra, J. Oh, S. Teller, N. Roy, and A. Stentz. 2014. Aprobabilistic framework for inferring maps and be-haviors from natural language. In Proceedings ofthe 14th International Symposium on ExperimentalRobotics.

A Elfes. 1987. Sonar-based real-world mapping andnavigation. IEEE Journal of Robotics and Automa-tion, 3(3).

Clemens Eppner, Sebastian Hofer, Rico Jonschkowski,Roberto Martın-Martın, Arne Sieverling, VincentWall, and Oliver Brock. 2016. Lessons from theamazon picking challenge: Four aspects of buildingrobotic systems. In Proceedings of Robotics: Sci-ence and Systems, AnnArbor, Michigan.

Maurice Fallon, Scott Kuindersma, Sisir Karumanchi,Matthew Antone, Toby Schneider, Hongkai Dai,Claudia Prez D’Arpino, Robin Deits, Matt DiCicco,Dehann Fourie, Twan Koolen, Pat Marion, MichaelPosa, Andrs Valenzuela, KuanTing Yu, Julie Shah,Karl Iagnemma, Russ Tedrake, and Seth Teller.2014. An architecture for online affordancebasedperception and wholebody planning. Journal ofField Robotics, 32(2):229–254.

Martin A. Fischler and Robert C. Bolles. 1981. Ran-dom sample consensus: A paradigm for model fit-ting with applications to image analysis and auto-mated cartography. Commun. ACM, 24(6):381–395.

C. Galindo, A. Saffiotti, S. Coradeschi, P. Buschka,J. A. Fernandez-Madrigal, and J. Gonzalez. 2005.Multi-hierarchical semantic maps for mobilerobotics. In 2005 IEEE/RSJ International Con-ference on Intelligent Robots and Systems, pages2278–2283.

Stevan Harnad. 1990. The symbol grounding problem.In Physica D: Nonlinear Phenomena, volume 42,pages 335–346.

S. Hemachandra, F. Duvallet, T.M. Howard, N. Roy,A. Stentz, and M.R. Walter. 2015. Learning mod-els for following natural language directions in un-known environments. In Proceedings of the IEEEInternational Conference on Robotics and Automa-tion. IEEE.

A. Hornung, K. M. Wurm, and M. Bennewitz. 2010.Humanoid robot localization in complex indoor en-vironments. In 2010 IEEE/RSJ International Con-ference on Intelligent Robots and Systems, pages1690–1695.

Armin Hornung, Kai M. Wurm, Maren Bennewitz,Cyrill Stachniss, and Wolfram Burgard. 2013. Oc-tomap: an efficient probabilistic 3d mapping frame-work based on octrees. Autonomous Robots,34(3):189–206.

T.M Howard, S. Tellex, and N. Roy. 2014. A natu-ral language planner interface for mobile manipula-tors. In Proceedings of the IEEE International Con-ference on Robotics and Automation, pages 6652–6659. IEEE.

Ronghang Hu, Marcus Rohrbach, and Trevor Darrell.2016. Segmentation from natural language expres-sions. In European Conference on Computer Vision,pages 108–124. Springer.

N. Hudson, T.M. Howard, J. Ma, A. Jain, M. Ba-jracharya, S. Myint, L. Matthies, P. Backes,P Hebert, T. Fuchs, and J. Burdick. 2012. End-to-end dexterous manipulation with deliberative inter-active estimation. In Proceedings of the 2012 IEEEInternational Conference on Robotics and Automa-tion.

K. Huebner and D. Kragic. 2008. Selection of robotpre-grasps using box-based shape approximation. In2008 IEEE/RSJ International Conference on Intelli-gent Robots and Systems, pages 1765–1770.

Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer,and Dieter Fox. 2013. Learning to Parse NaturalLanguage Commands to a Robot Control System.Springer International Publishing, Heidelberg.

Andrew T. Miller, Steffen Knoop, Henrik I. Chris-tensen, and Peter K. Allen. 2003. Automatic graspplanning using shape primitives. In ICRA.

R. Paul, J. Arkin, N. Roy, and T.M. Howard. 2016. Ef-ficient grounding of abstract spatial concepts for nat-ural language interaction with robot manipulators.In Proceedings of the 2016 Robotics: Science andSystems Conference.

A. Pronobis and P. Jensfelt. 2012. Large-scale se-mantic mapping and reasoning with heterogeneousmodalities. In 2012 IEEE International Conferenceon Robotics and Automation, pages 3515–3522.

O. Propp, I. Chung, M.R. Walter, and T.M. Howard.2015. On the performance of hierarchical dis-tributed correspondence graphs for efficiency sym-bol grounding of robot instructions. In Proceedings

160

of the 2015 IEEE/RSJ International Conference onIntelligent Robots and Systems.

Radu Bogdan Rusu and Steve Cousins. 2011. 3D ishere: Point Cloud Library (PCL). In IEEE Inter-national Conference on Robotics and Automation(ICRA), Shanghai, China.

Stefanie Tellex, Thomas Kollar, Steven Dickerson,Matthew Walter, Ashis Banerjee, Seth Teller, andNicholas Roy. 2011. Understanding natural lan-guage commands for robotic navigation and mobilemanipulation.

A Grammar and Lexicon of the Corpus

We list the grammar rules and the lexicon for ourcorpus to demonstrate the diversity of the instruc-tions. Following table lists the words scraped fromthe instructions in our corpus. We have a total of56 unique words.

VB → {pick|put}RP → {up}DT → {the|all}CC → {and}

VBZ → {is}WDT → {that}

VB → {near|in|on}PRP → {your}

NN →

cup|pudding|box|cube|object|ball|master|chef|can|soccer|gear|mustard|sauce|bottle|sugar|packet|block|cleanser|middle|left|right|crackers|cheezit|cleanser|packet|block

NNS →

{cups|chips|cubes|objects|balls

}

JJ →

{blue|green|yellow|red|white

}

JJS →

nearest|rightmost|leftmost|farthest|biggest|smallest|largest|closest

Table 4: The words scraped from the corpus ofannotated examples

Following table lists the grammar rules scrapedfrom the instructions in our corpus. We have atotal of 23 unique grammar rules.

SBAR → WHNP S

S → VP

VP → VB PRT NP

VP → CC VP VP

VP → VB NP PP

VP → VBZ PP

WHNP → WDT

PRT → RP

PP → IN NP

NP → DT JJ NN

NP → DT NN NN

NP → DT JJS JJ NN

NP → NP PP

NP → DT

NP → DT JJ NNS

NP → DT NN

NP → DT NN NN NN

NP → DT JJ NN NN

NP → DT JJS NN

NP → DT NNS NN

NP → PRP NN

NP → NP SBAR

NP → DT NNS

Table 5: The grammar rules scraped from the cor-pus of annotated examples