machine vision for service robots & surveillance
TRANSCRIPT
![Page 1: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/1.jpg)
DelftUniversity ofTechnology
Machine Vision for Service Robots &
Surveillance
Prof.dr.ir. Pieter Jonker
EMVA Conference 2015
![Page 2: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/2.jpg)
2 | 16
Cognitive Robotics (text intro)
This presentation addresses the issue that machine vision and learning
systems will dominate our lives and work in future, notably through
surveillance systems and service robots. Pieter Jonker’s specialism is robot
vision; the perception and cognition of service robots, more specifically
autonomous surveillance robots and butler robots. With robot vision
comes the artificial intelligence; if you perceive and understand, than
taking sensible actions becomes relative easy. As robots – or autonomous
cars, or … - can move around in the world, they encounter various
situations to which they have to adapt. Remembering in which situation
you adapted to what is learning. Learning comes in three flavors:
cognitive learning through Pattern Recognition (association), skills learning
through Reinforcement Learning (conditioning) and the combination
(visual servoing); such as a robot learning from observing a human how
to poor in a glass of beer. But as always in life this learning comes with a
price; bad teachers / bad examples.
![Page 3: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/3.jpg)
DelftUniversity ofTechnology
• Professor of (Bio) Mechanical EngineeringVision based Robotics group, TU-Delft Robotics Institute
• Chairman Foundation Living Labs for Care Innovation
• CEO LEROVIS BV, CEO QdepQ BV, CTO Robot Care Systems
Prof.dr.ir. Pieter Jonker
EMVA Conference
Athens 12 June 2015
Machine Vision for Service Robots &
Surveillance
![Page 4: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/4.jpg)
4 | 16
Content
• What is cognitive robotics + examples
• Cognitive learning and pattern recognitionSkills learning and reinforcement learningCognitive skills learning (visual servoing)
• Impact of robots / learning systems
![Page 5: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/5.jpg)
5 | 16
Content
• What is cognitive robotics + examples
• Cognitive learning and pattern recognitionSkills learning and reinforcement learningCognitive skills learning (visual servoing)
• Impact of robots / learning systems
![Page 6: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/6.jpg)
6 | 16
Cognitive Robotics
• Machine vision with machine learning
• Study the human and robot body & brain
• Robot Vision, Artificial Intelligence, Embedded Systems, Robot Platforms
• Learning to recognize: saliency, objects, persons, faces, places, affordances, actions, activities
• Learn to perform: actions, activities, …
• With robot vision in the control loop
![Page 7: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/7.jpg)
7 | 16
For the purpose of (1):
• Human and machine tightly coupled cooperating on the control level with proprioceptic / haptic feedback: e.g. exo-skeleton Mindwalker
![Page 8: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/8.jpg)
8 | 16
Tight human-machine control loops
• Mindwalkerexoskeleton:• Align joints
• Measure Forces
• Control Forces
• Control Stifness
![Page 9: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/9.jpg)
9 | 16
For the purpose of (2):
• Human and machine less tightly coupled cooperating on the control and protocol level using proprioceptic and visual feedback in the control loop: autonomous car driving, robot football, …
![Page 10: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/10.jpg)
10 | 16
Less tight human machine control loop
• EZ10 people mover• Measure road users:road, road-signs, cars, bicycles, pedestrians;intentions, paths, possible collisions
• Control collision free auto-motion
• Control alternative paths while moving
• Fusion of camera, radar, gps, maps data for (auto) pose and ADAS
![Page 11: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/11.jpg)
11 | 16
Robot Football
![Page 12: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/12.jpg)
12 | 16
Legged robots are still too difficult
![Page 13: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/13.jpg)
13 | 16
For the purpose of (3):
• Human and machine less loosely coupled cooperating on a protocol level using vision only in the loop: turn taking in cooperative assembly, automatic surveillance robots
![Page 14: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/14.jpg)
14 | 16
Bin picking with monocular vision
• Scale Invariant Feature tracking
![Page 15: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/15.jpg)
15 | 16
Less tight human machine control loop
![Page 16: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/16.jpg)
16 | 16
SAM Warehouse Surveillance
• (Large Maps) Visual Slam, RF Beacons
![Page 17: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/17.jpg)
17 | 16
SAM Outdoor Surveillance
• (Large Maps) Visual Slam, RF Beacons, Radar, GPS, IR
![Page 18: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/18.jpg)
18 | 16
Less tight human machine control loop
• Factory in a day• Quickly set-up a production
• Simple protocols; e.g. collaboration by turn-taking
• Intention projection;
• Autonomous vehicles
• Robot arms
![Page 19: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/19.jpg)
19 | 16
Content
• What is cognitive robotics + examples
• Cognitive learning and pattern recognitionSkills learning and reinforcement learningCognitive skills learning (visual servoing)
• Impact of robots / learning systems
![Page 20: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/20.jpg)
20 | 16
1. Perceive (pattern recognition) & understand
2. Interact with users
3. Memorize
4. Physically help with simple task (pick up objects)
5. Provide feedback and motivation
6. Signal and alarm
The most advanced is research onCognitive Butler Robots
![Page 21: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/21.jpg)
21 | 16
Butler robots; where are we now
![Page 22: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/22.jpg)
22 | 16
Pointing at Objects
![Page 23: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/23.jpg)
23 | 16
Cognitive Robotics(percieve and understand)
• Cognitive learning: to recognize objects, persons, faces, actions, places, …
• Skills learning: to move, walk, grasp, …
• Combining: Visual servoing for grasping, balance while acting, human / robot collaboration
• Target: a baby of 6 months
![Page 24: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/24.jpg)
24 | 16
Cognitive Robotics
![Page 25: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/25.jpg)
25 | 16
Robot Cognition
• Learn to recognize unknown objects
• Find the proper viewpoint
• Find the proper point to grasp
• Grasp this by visual servoing
• Remember this with reinforcement learning
![Page 26: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/26.jpg)
26 | 16
1. sensors or/and cameras with image processing
2. Measure features: x, y, …
3. Label / name the item
4. Label = ax + by + cz + …
5. Learn a, b, c by trial and error
6. Learn over groups and individuals
7. Use the system
Cognitive learning by pattern recognition
![Page 27: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/27.jpg)
27 | 16
Skills learning by reinforcement learning
![Page 28: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/28.jpg)
28 | 16
learning visual servoing tasks by example
![Page 29: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/29.jpg)
29 | 16
Towards learning of visual reflexes
![Page 30: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/30.jpg)
30 | 16
Robot head with its visual reflexes
Analogy with the processing in the cerebellum?
![Page 31: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/31.jpg)
31 | 16
Using a fusion of depth Cues• Depth from vergence• Depth from stereo• Depth from luminance
(EMD + DFL)
PCT/EP2013/051255ADAPTIVE MULTI-DIMENSIONAL
DATA DECOMPOSITION
Download the QdepQ 3D viewer from the Google playstore!
![Page 32: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/32.jpg)
32 | 16
1. Human Machine Loop:• tightly coupled control over propriocepsis / hapics• loosely coupled control over propriocepsis + vision• loosely coupled protocol over vision
2. The most advanced research is on cognitive butler robots• cognitive learning• skills learning
3. Learning visual reflexes• combined (cognitive + skills learning)
Conclusions
![Page 33: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/33.jpg)
33 | 16
Content
• What is cognitive robotics + examples
• Cognitive learning and pattern recognitionSkills learning and reinforcement learningCognitive skills learning (visual servoing)
• Impact of robots / learning systems
![Page 34: Machine Vision for Service Robots & Surveillance](https://reader030.vdocuments.net/reader030/viewer/2022012700/61a31b4dc6be8e769f26c5b5/html5/thumbnails/34.jpg)
34 | 16
1. Cognitive Robotics will penetrate out lives
2. New is the cognitive skills (visual servoing) learning capability, beware!
3. This brings the troubles along of bad teachers, bad youth, personality disorders, …
Propositions