the svs spatial/visual reasoning system · 2017. 6. 14. · motivation much work has been done in...
TRANSCRIPT
![Page 1: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/1.jpg)
The SVS Spatial/Visual Reasoning System
5/5/2008 1
Reasoning SystemSamuel WintermuteUniversity of Michigan
Soar Workshop 28, Ann Arbor, MI
![Page 2: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/2.jpg)
Outline
� Motivation� Design� SVS Scene Graphs
25/5/2008
� SVS Scene Graphs� Spatial Generation and Extraction� Visual Generation and Extraction� Example Problem� Nuggets and Coal
![Page 3: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/3.jpg)
Motivation
� Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning
35/5/2008
spatial and visual reasoning� Two systems, SVI and SRS, have been
developed� SVS is the unification of SVI and SRS,
with additional features
![Page 4: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/4.jpg)
Perception
45/5/2008
The agent approaches an enemy castle, and hides in the distance.
![Page 5: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/5.jpg)
Having been here before, the agent is able to retrieve a long-term memory of the castle, and imagines it in detail.
Imagery
55/5/2008
Knowing the castle is well-defended, the agent contemplates a stealthy approach…
![Page 6: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/6.jpg)
Imagery
Perhaps a good place to hide is behind the right leg of the gateway arch? The agent visualizes itself in that position (here represented by a green ‘A’).
65/5/2008
![Page 7: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/7.jpg)
Imagery
The agent knows that enemy guards patrol the large turret of the castle. It imagines an enemy (here, represented as a red ‘E’), in position.
75/5/2008
Can the guard see the hiding spot?
![Page 8: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/8.jpg)
Imagery
The agent imagines the perspective of the guard, and checks if the ‘A’ is visible. It is, partially. It concludes that the hiding spot will not be effective.
85/5/2008
![Page 9: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/9.jpg)
High-Level DesignSoar
Qualitative / Symbolic
Quantitative / Spatial
95/5/2008
Spatial Scene
Perceptual LTMViewpoint
Visual Buffer
Environment
Visual / Depictive
![Page 10: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/10.jpg)
SVS Scene Graphs
scene
� SVS now makes scene graphs (in Spatial STM and LTM) accessible to Soar
� Soar can expand and contract nodes
transformation
object
texture
105/5/2008
housetree
toptrunk roofframe
siding door
![Page 11: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/11.jpg)
High-Level DesignSoar
115/5/2008
Spatial Scene
Perceptual LTMViewpoint
Visual Buffer
Environment
![Page 12: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/12.jpg)
Perceptual LTM
� Stores prototypical objects (shapes), textures, and
125/5/2008
Perceptual LTM
textures, and transformations
� Hierarchically organized in scene graphs
![Page 13: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/13.jpg)
High-Level DesignSoar
135/5/2008
Spatial Scene
Perceptual LTM
Visual Buffer
Viewpoint
Environment
![Page 14: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/14.jpg)
Spatial Scene� 3D spatial short-term
memory� Scene graph
But grounded in
145/5/2008
Spatial Scene
�But grounded in coordinates
![Page 15: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/15.jpg)
High-Level DesignSoar
155/5/2008
Spatial Scene
Perceptual LTMViewpoint
Visual Buffer
Environment
![Page 16: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/16.jpg)
Visual Buffer� Short-term memory� Contains a set of 2D,
depictive representations
165/5/2008
representations� Derived from the
scene, at a given viewpoint� Theoretically, the
opposite is trueVisual Buffer
![Page 17: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/17.jpg)
High-Level DesignSoar
175/5/2008
Spatial Scene
Perceptual LTMViewpoint
Visual Buffer
Environment
![Page 18: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/18.jpg)
Soar Interface
� Interfaces are (mostly) qualitative: all quantitative information stays within SVS
� SVS provides contents of all memories to Soar� Scene graphs are expandable/collapsable
185/5/2008
� Soar can add new information to SVS� Generate new objects in the scene, and new depictions in the visual
buffer
� Soar can retrieve information from SVS� Extract predicates from the scene and visual buffer through queries
![Page 19: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/19.jpg)
Outline
� Motivation� Design� SVS Scene Graphs
195/5/2008
� SVS Scene Graphs� Spatial Generation and Extraction� Visual Generation and Extraction� Example Problem� Nuggets and Coal
![Page 20: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/20.jpg)
Scene Graph Example: Objects
transformation
housetree
frame
sceneobject
texture
205/5/2008
toptrunk roofframe
siding door
![Page 21: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/21.jpg)
Scene Graph Example: Transformations
transformation
housetree
frame
sceneobject
texture
215/5/2008
toptrunk roofframe
siding door
origin
![Page 22: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/22.jpg)
Scene Graph Example: Textures
transformation
housetree
frame
sceneobject
texture
225/5/2008
toptrunk roofframe
siding door
![Page 23: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/23.jpg)
Outline
� Motivation� Design� SVS Scene Graphs
235/5/2008
� SVS Scene Graphs� Spatial Generation and Extraction� Visual Generation and Extraction� Example Problem� Nuggets and Coal
![Page 24: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/24.jpg)
Spatial Generation
� SVS supports imagery where Soar directly provides a scene graph� Scene graphs are composable, any valid combination of nodes yields a scene� These operations are called “graph generations”
� SVS also supports imagery where objects and/or transformations are qualitatively described
245/5/2008
qualitatively described� Scene graph is implicit in the description� These operations are called “projections”
� Motion models allow SVS to represent movement (my other talk)
� All images (spatial and visual) exist as long as the command is on the output-link� Images can be manipulated by changing the command in place
![Page 25: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/25.jpg)
Accessing Long-Term Memory
� Items in LTM have a ^class-id� e.g. car, car37, car-transform37, car-texture37
� Items in the scene have a ^class-id and an ^instance-id� e.g. ^class-id car, ^instance-id car:i23
255/5/2008
� A retrieval from LTM happens when a ^class-id appears in a generate command� Objects are copied all the way down to the leaves� Graph generations can access LTM nodes identically to nodes in the
scene
![Page 26: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/26.jpg)
Spatial Graph Generation Example
^generate-spatial^transformation ^object^parent scene ^source tree ^source house-trans ^new-id tree2^new-id tree-trans2^child-new-id tree2
265/5/2008
house tree
toptrunk
scene
^child-new-id tree2
top2trunk2
tree2
![Page 27: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/27.jpg)
Spatial Graph Generation Example
^generate-spatial^transformation ^object^parent scene ^source-class birch-tree ^source house-trans ^new-id tree2^new-id tree-trans2^child-new-id tree2
275/5/2008
house tree
toptrunk
scene
^child-new-id tree2
top2trunk2
tree2
![Page 28: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/28.jpg)
Spatial Projection Example
^generate-spatial^projection^hull^instance-id tree^instance-id house
^transform-parent scene
285/5/2008
house tree
toptrunk
scene
^transform-parent ^texture-source.class-id red
hull-1
scene
![Page 29: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/29.jpg)
Spatial Extraction Example^extract-spatial ^relationship adjacent^primary-object frame^reference-object ?^value true
^extract-spatial-result ^retrieved *5
^relationship adjacent^primary-object frame^reference-object {door, siding,
roof, house, scene}^value true
295/5/2008
tree
toptrunk
scene
house
roofframe
siding door
![Page 30: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/30.jpg)
Outline
� Motivation� Design� SVS Scene Graphs
305/5/2008
� SVS Scene Graphs� Spatial Generation and Extraction� Visual Generation and Extraction� Example Problem� Nuggets and Coal
![Page 31: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/31.jpg)
Visual Generation
� Visual generation commands add depictions to the visual buffer
� The scene generator renders all or part of the spatial scene to the VB, from a given camera
315/5/2008
spatial scene to the VB, from a given camera position
� Other visual generators make new depictions without accessing the spatial scene� Depictive manipulation (Scott Lathrop’s talk) is a
subset of this
![Page 32: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/32.jpg)
Visual Scene Generationfrom the perspective of chandelier01 looking towards table04, generate the scene
from the perspective
325/5/2008
chandelier01
table04
from the perspective of chandelier01 looking towards table04, generate table04
from the perspective of chandelier01 looking towards table04, generate chair14, ignoring table04
![Page 33: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/33.jpg)
Visual Generation and Extraction
generate a depiction outlining the enclosed space in chair14:depiction
335/5/2008
query: are any pixels in chair14:depiction:enc filled in?
result: yes
![Page 34: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/34.jpg)
Outline
� Motivation� Design� SVS Scene Graphs
345/5/2008
� SVS Scene Graphs� Spatial Generation and Extraction� Visual Generation and Extraction� Example Problem� Conclusion, Nuggets and Coal
![Page 35: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/35.jpg)
Perception
355/5/2008
The agent approaches an enemy castle, and hides in the distance.
![Page 36: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/36.jpg)
Having been here before, the agent is able to retrieve a long-term memory of the castle, and imagines it in detail.
Imagery
365/5/2008
Knowing the castle is well-defended, the agent contemplates a stealthy approach…
![Page 37: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/37.jpg)
Imagery
Perhaps a good place to hide is behind the right leg of the gateway arch? The agent visualizes itself in that position (here represented by a green ‘A’).
375/5/2008
![Page 38: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/38.jpg)
Imagery
The agent knows that enemy guards patrol the large turret of the castle. It imagines an enemy (here, represented as a red ‘E’), in position.
385/5/2008
Can the guard see the hiding spot?
![Page 39: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/39.jpg)
Imagery
The agent imagines the perspective of the guard, and checks if the ‘A’ is visible. It is, partially. It concludes that the hiding spot will not be effective.
395/5/2008
![Page 40: The SVS Spatial/Visual Reasoning System · 2017. 6. 14. · Motivation Much work has been done in the last few years to enhance Soar with lower-level spatial and visual reasoning](https://reader034.vdocuments.net/reader034/viewer/2022051904/5ff5f90892f24e72db6227c8/html5/thumbnails/40.jpg)
Nuggets and Coal
� Nuggets� Architecture seems to be becoming simpler, not more complex� Soar now has direct access to the scene graph� Episodic memory integration is close to being possible
405/5/2008
� Coal� Requires the environment (or low-level perceptual system) to
provide a scene graph� Integration with environments still requires lots of custom code� Scene graph representation in SVS is dictated by graphical
modelers and their tools� Implementation isn’t done yet