learning object affordances based on structural object representation
TRANSCRIPT
![Page 1: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/1.jpg)
Learning object affordances based on structural object representation
Kadir F. UyanikAsil Kaan Bozcuoglu
EE 583 Pattern RecognitionJan 4, 2011
![Page 2: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/2.jpg)
Content• Goal• Inspirations• Potential Difficulties• Problem Definition• Proposed Method• References• Appendix
![Page 3: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/3.jpg)
Goal
![Page 4: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/4.jpg)
Goal
![Page 5: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/5.jpg)
Goal
![Page 6: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/6.jpg)
Goal
![Page 7: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/7.jpg)
InspirationsEcological Psychologist James Jerome Gibson
1904 -1979
Cognitive PsychologistIrving Biederman
1939 -
![Page 8: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/8.jpg)
Inspirations:Affordances[1]
[1] J. J. Gibson (1977), The Theory of Affordances. In Perceiving, Acting, and Knowing, Eds. Robert Shaw and John Bransford, ISBN 0-470-99014-7.[2] E. Sahin, M. Cakmak, M.R.Dogar, E. Ugur , G. Ucoluk, To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control, Adaptive Behavior , 2007 pp: 447-472
“… an affordance is neither an objective property nor a subjective property; or both if you like. An affordance cuts across the dichotomy of subjective-objective and helps us to understand its inadequacy. It is equally a fact of the environment and a fact of behavior. It is both physical and psychical, yetneither. An affordance points both ways, to the environment and to the observer.”
![Page 9: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/9.jpg)
Inspirations:Affordances[1]
[1] J. J. Gibson (1977), The Theory of Affordances. In Perceiving, Acting, and Knowing, Eds. Robert Shaw and John Bransford, ISBN 0-470-99014-7.[2] E. Sahin, M. Cakmak, M.R.Dogar, E. Ugur , G. Ucoluk, To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control, Adaptive Behavior , 2007 pp: 447-472
“… an affordance is neither an objective property nor a subjective property; or both if you like. An affordance cuts across the dichotomy of subjective-objective and helps us to understand its inadequacy. It is equally a fact of the environment and a fact of behavior. It is both physical and psychical, yetneither. An affordance points both ways, to the environment and to the observer.”
![Page 10: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/10.jpg)
Inspirations:Affordances[1]
[1] J. J. Gibson (1977), The Theory of Affordances. In Perceiving, Acting, and Knowing, Eds. Robert Shaw and John Bransford, ISBN 0-470-99014-7.
<entity> <behavior>
<effect>
environment agent
(<effect>, <(entity, behavior)>)Revised Definition: An affordance is an acquired relation between a <(entity, behavior)> tuple of an agent such that the application of the <behavior> on the <entity> generates a certain <effect>[2].
[2] E. Sahin, M. Cakmak, M.R.Dogar, E. Ugur , G. Ucoluk, To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control, Adaptive Behavior , 2007 pp: 447-472
“… an affordance is neither an objective property nor a subjective property; or both if you like. An affordance cuts across the dichotomy of subjective-objective and helps us to understand its inadequacy. It is equally a fact of the environment and a fact of behavior. It is both physical and psychical, yetneither. An affordance points both ways, to the environment and to the observer.”
![Page 11: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/11.jpg)
[3] Recognition-by-components: A theory of Human Image Understanding, Psychological Review, Vol. 94 (1987), pp. 115-148
“There are small number of geometric components that constitute the primitive elements of the object recognition system (like letters to form words)”
Inspirations:Human Image Understanding[3]
![Page 12: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/12.jpg)
[3] Recognition-by-components: A theory of Human Image Understanding, Psychological Review, Vol. 94 (1987), pp. 115-148
“There are small number of geometric components that constitute the primitive elements of the object recognition system (like letters to form words)”
Inspirations:Human Image Understanding[3]
![Page 13: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/13.jpg)
Potential Difficulties[4]
• Structural description not enough, also need metric info
[4] M. A. Arbib CS564 – Brain Theory and Artificial Intelligence, USC, Fall 2001, Lecture 7: Object Recognition
![Page 14: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/14.jpg)
Potential Difficulties[4]
• Structural description not enough, also need metric info
• Difficult to extract geons from real images
[4] M. A. Arbib CS564 – Brain Theory and Artificial Intelligence, USC, Fall 2001, Lecture 7: Object Recognition
![Page 15: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/15.jpg)
Potential Difficulties[4]
• Structural description not enough, also need metric info
• Difficult to extract geons from real images
• Ambiguity in the structural description: most often we have several candidates
[4] M. A. Arbib CS564 – Brain Theory and Artificial Intelligence, USC, Fall 2001, Lecture 7: Object Recognition
![Page 16: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/16.jpg)
Potential Difficulties[4]
• Structural description not enough, also need metric info
• Difficult to extract geons from real images
• Ambiguity in the structural description: most often we have several candidates
• For some objects, deriving a structural representation can be difficult
[4] M. A. Arbib CS564 – Brain Theory and Artificial Intelligence, USC, Fall 2001, Lecture 7: Object Recognition
![Page 17: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/17.jpg)
Problem Definition
HOW TO• decompose objects into parts/components ?• find relations between components ?• find a generic graph representation of an
<action-entity-effect> three tuple ?
![Page 18: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/18.jpg)
Object Decomposition
Proposed Algorithm
![Page 19: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/19.jpg)
Object Decomposition
Proposed Algorithm
![Page 20: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/20.jpg)
Object Decomposition
Proposed Algorithm
![Page 21: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/21.jpg)
Object Decomposition
Proposed Algorithm
![Page 22: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/22.jpg)
Object Decomposition
Proposed Algorithm
![Page 23: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/23.jpg)
Object Decomposition
Proposed Algorithm
![Page 24: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/24.jpg)
Object Decomposition
Proposed Algorithm
![Page 25: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/25.jpg)
Object Decomposition
Proposed Algorithm
![Page 26: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/26.jpg)
Object Decomposition
Proposed Algorithm
![Page 27: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/27.jpg)
Object Decomposition
Proposed Algorithm
![Page 28: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/28.jpg)
Object Decomposition
Proposed Algorithm
![Page 29: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/29.jpg)
Object DecompositionWhat is missing?
•Use/try different clustering algorithms
•Triangulate 3D surfaces, Delaunay
• Compute gaussian curvature on each vertex
• Detect region boundaries, curvature thresholding
•Perform iterative region growing, flood fill
![Page 30: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/30.jpg)
Graphical Representation
• We represent each objects in non-directed graphs as follows:– Each node has the info of geometric
shape of the part– Each edge has the information of
direction of edge for three axises, i.e from node1 to node2, x axis increases.
![Page 31: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/31.jpg)
Graphical Representation
Similarity Checking
[isIsomorphic, label_list]= check_Isomorphism(G1, G2)If isIsomorphic
Check geometric shapes of same labeled nodes in two graphsCheck direction of equivalent edges in both graphsIf both are matched, return trueElse return false
Else return false
![Page 32: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/32.jpg)
Isomorphism check: Two candidates: - n1 = n6, n2 = n4, n3 = n5 (Attributes matched!) - n1 = n4, n2 = n6, n3 = n5 (Attributes isn’t matched)
Graphical Representation
Similarity Checking
![Page 33: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/33.jpg)
![Page 34: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/34.jpg)
Current System• 80% is successful • Assumes no occlusion.
– For the cup case, handles should always be visible
• Needs metric info to distinguish bigger objects from small ones
![Page 35: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/35.jpg)
One way to go…• Learning a generic graph for each affordance type.• Checking the maximal- cliques of the match graph while comparing graph
of an object and a generic graph.• Mahalanobis distance metric for generic graphs and use MLE
![Page 36: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/36.jpg)
Tools
![Page 37: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/37.jpg)
References
[1] J. J. Gibson (1977), The Theory of Affordances. In Perceiving, Acting, and Knowing, Eds. Robert Shaw and John Bransford, ISBN 0-470-99014-7.[2] E. Sahin, M. Cakmak, M.R.Dogar, E. Ugur , G. Ucoluk, To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control, Adaptive Behavior , 2007 pp: 447-472[3] Recognition-by-components: A theory of Human Image Understanding, Psychological Review, Vol. 94 (1987), pp. 115-148[4] M. A. Arbib CS564 – Brain Theory and Artificial Intelligence, USC, Fall 2001, Lecture 7: Object Recognition
![Page 38: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/38.jpg)
Thanks for listening
![Page 39: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/39.jpg)
Appendix
![Page 40: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/40.jpg)
Human Image Understanding• Hypothesis: small number of geometric components that constitute
the primitive elements of the object recognition system (like letters to form words)
• Geons are directly recognized from edges, based on their nonaccidental properties (i.e., 3D features that are usually preserved by the projective imaging process).
– edges are straight or curved– pairs of edges are parallel or non-parallel– vertices will always appear to be vertices
• Non-accidental properties allows geons to be recognized from any perspective.
• The information in the geons are redundant so that they can be recognized even when partially occluded.
![Page 41: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/41.jpg)
AppendixThe Importance of spatial arrangement
![Page 42: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/42.jpg)
AppendixThe Principal of non-accidentalness
Examples:
• Colinearity
• Smoothness
• Symmetry
• Parallelism
• Cotermination
![Page 43: Learning Object Affordances Based on Structural Object Representation](https://reader033.vdocuments.net/reader033/viewer/2022061123/5473e637b4af9f06698b46eb/html5/thumbnails/43.jpg)
AppendixSome non-accidental differences