sean m. ficht. problem definition previous work methods & theory results
TRANSCRIPT
![Page 1: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/1.jpg)
Sean M. Ficht
![Page 2: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/2.jpg)
Problem Definition
Previous Work
Methods & Theory
Results
![Page 3: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/3.jpg)
Track and follow specific person with a mobile robot
Cluttered environments
Brief occlusion
Long occlusion
Cooperative user
![Page 4: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/4.jpg)
Helper robot
Carry items for a person• Example: Hospital situation
![Page 5: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/5.jpg)
Problem Definition
Previous Work
Methods & Theory
Results
![Page 6: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/6.jpg)
Person following with a mobile robot
• Appearance based
• Optical flow based
• Stereo vision based
![Page 7: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/7.jpg)
Segmentation of image
Classification
Detection
Limitations
Sidenbladh, Kragic, and Christensen; ICRA; 1999Tarokh and Ferrari; Journal of Robotic Systems; 2003Schlegel, Illman, Jaberg, Schuster, and Worz; British Machine Vision Conference; 2005
![Page 8: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/8.jpg)
Calculate optical flow
Use to segment image
Limitations
Chivilo, Mezzaro, Sgorbissa, and Zaccaria; IROS; 2004Piaggio, Fornaro, Piombo, Sanna, and Zaccaria; IEEE ISIC/CIRA/ISAS joint conference; 1998
![Page 9: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/9.jpg)
Find features
Segment from background
Use to track
Limitations
Zhichao and Birchfield; IROS; 2007
![Page 10: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/10.jpg)
Problem Definition
Previous Work
Methods & Theory
Results
![Page 11: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/11.jpg)
Kinect
Provides a depth image
Provides a RGB color image
Packaged solution
![Page 12: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/12.jpg)
Detection and Tracking• Generic detector
• Specific appearance model
• Integrating particle filter
Robot Control
![Page 13: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/13.jpg)
HOG person detector (OpenCV)
• HOG descriptoro Cells -> Block -> Windowo 4 cells in a blocko 105 blocks in a windowo 64x128 window
• Training
Dalal and Triggs, CVPR, 2005
![Page 14: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/14.jpg)
Gradient of the Image
Binning of pixels in cells
Grouping of cells into blocks
Normalization
![Page 15: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/15.jpg)
Kernel convolution
• Magnitude = (gx2 + gy
2)
• Angle = arctan(gy/gx)
Directional change in intensity
![Page 16: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/16.jpg)
Bins apply to each cell
Nine separate bins
Gradient magnitude added to bin
![Page 17: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/17.jpg)
Cells grouped into blocks
4 cells per block
Blocks overlap one another
![Page 18: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/18.jpg)
![Page 19: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/19.jpg)
HOG person detector
• HOG descriptoro Cells -> Block -> Windowo 4 cells in a blocko 105 blocks in a windowo 64x128 window
• Training
![Page 20: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/20.jpg)
Support Vector Machine (SVM) classifier
Binary classifier
Trained on images
![Page 21: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/21.jpg)
Detection and Tracking• Generic detector
• Specific appearance model
• Integrating particle filter
Robot Control
![Page 22: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/22.jpg)
Color Histogram
Segmentation by depth to create template
![Page 23: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/23.jpg)
Represents distribution of colors
10 bins for each color channel• 1000 element color histogram
Pixel classification
2 bin example• Bin 1: 0-127• Bin 2: 128-255
![Page 24: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/24.jpg)
Average depth
Threshold (0.3 meters)
Template used to make color histogram
![Page 25: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/25.jpg)
Detection and Tracking• Generic detector
• Specific appearance model
• Integrating particle filter
Robot Control
![Page 26: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/26.jpg)
System State
Motion model
Observation model
Expected state
Resample
![Page 27: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/27.jpg)
Hybrid state space
X and Y in image coordinates• Scaled according to depth
Z in depth coordinates
![Page 28: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/28.jpg)
![Page 29: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/29.jpg)
![Page 30: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/30.jpg)
![Page 31: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/31.jpg)
![Page 32: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/32.jpg)
![Page 33: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/33.jpg)
![Page 34: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/34.jpg)
![Page 35: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/35.jpg)
Detection and Tracking• Generic detector
• Specific appearance model
• Integrating particle filter
Robot Control
![Page 36: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/36.jpg)
Input: tracking information from tracking algorithm
Uses tracking information to make movement decisions
Executes movement and returns to tracking algorithm
![Page 37: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/37.jpg)
![Page 38: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/38.jpg)
Problem Definition
Previous Work
Methods & Theory
Results
![Page 39: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/39.jpg)
No occlusion
• Other people present (different depth)• Other people present (similar depth)• Pose change
Brief occlusion
Long occlusion
![Page 40: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/40.jpg)
![Page 41: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/41.jpg)
![Page 42: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/42.jpg)
Initial template
![Page 43: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/43.jpg)
![Page 44: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/44.jpg)
![Page 45: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/45.jpg)
![Page 46: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/46.jpg)
Initial template Non-occluded target Occluded target
![Page 47: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/47.jpg)
Average between 73% and 74%
![Page 48: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/48.jpg)
![Page 49: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/49.jpg)
![Page 50: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/50.jpg)
Problem• Follow a person in different scenarios
System• RGB-D sensor• Generic detector• Specific appearance model• Particle filter• Robot control architecture
Performance• Performed in three separate test scenarios• Rapid side to side target motion trade-off• Large target scale changes
![Page 51: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/51.jpg)
Train a new HOG detector to handle scale issues
Using more particles
KLT features for trajectory histories
Adaptive appearance model
![Page 52: Sean M. Ficht. Problem Definition Previous Work Methods & Theory Results](https://reader035.vdocuments.net/reader035/viewer/2022062809/5697bf7a1a28abf838c82cfa/html5/thumbnails/52.jpg)