![Page 1: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/1.jpg)
Computer Vision CMSC 426
Fall 2019
![Page 2: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/2.jpg)
Logistics
• Four projects, and three homework assignments (programming + discussion): in groups of three
• Midterm exam: in class • All information available from the Website • Grading
– Projects 50 % – Homework 25% – MidTerm 25%
![Page 3: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/3.jpg)
Programming
• Python • Primer on Python?
![Page 4: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/4.jpg)
Administration • Webpage
• Get homework and projects • Syllabus • Other documents
• Piazza • Ask questions
• do not post solutions • do not ask if your answer or approach is correct
• Discuss issues • Public versus private
• ELMS • Hand in homework and projects • See grades
![Page 5: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/5.jpg)
Recommended TextsComputer Vision: Algorithms and Application, Richard Szeliski Online: http://szeliski.org/Book/
Computer Vision: A Modern Approach David Forsyth and Jean Ponce Online: http://www.csd.uwo.ca/~olga/Courses/Winter2010/CS4442_9542b/CVbook.pdf Digital Image Processing, Prentice Hall, Rafael Gonzalez, and Richard Woods, 2008. Online: http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/Digital_Image_Processing_2ndEd.pdf
Multiple View Geometry in Computer Vision Richard Hartley and Andrew Zisserman University Press, 2004, selected chapters available online: http://www.robots.ox.ac.uk/~vgg/hzbook/
![Page 6: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/6.jpg)
What is Computer Vision
Sensor Interpreting device
Interpretations
water, grass, mountain, lamp post etc.
Image/video
![Page 7: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/7.jpg)
What is Computer Vision
Sensor Interpreting device
Interpretations
water, grass, mountain, lamp post etc.
Image/video
![Page 8: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/8.jpg)
What is Computer Vision
Sensor Interpreting device
Interpretations
water, grass, mountain, lamp post etc.
Image/video
![Page 9: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/9.jpg)
Goals of Computer Vision• Build machines and develop algorithms which can automatically
replicate some functionalities of biological visual system
- Systems which navigate in cluttered environments - Systems which can recognize objects, activities - Systems which can interact with humans/world
• Synergies with other disciplines and various applications Artificial Intelligence - robotics, natural language understanding
• Vision as a sensor - medical imaging, Geospatial Imaging, robotics, visual surveillance, inspection
![Page 10: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/10.jpg)
- image appearance depends on structure of the scene - material and reflectance properties of the objects - position and strength of light sources
![Page 11: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/11.jpg)
[Felleman & Van Essen, 1991]
This is the part of your brain that processes visual information
Visual Information Processing
![Page 12: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/12.jpg)
• About 40% of our brain is devoted to vision • We see immediately and can form and understand
images instantly
• Applications and examples
![Page 13: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/13.jpg)
Connections to other disciplines
Computer Vision
Image Processing
Machine Learning
Artificial Intelligence
Robotics
Cognitive science Neuroscience
Computer Graphics
![Page 14: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/14.jpg)
Goal of Computer Vision
230 25 34 123
45 0 10 52
65 11 210 42
78 87 56 90
23 18 29 61
what we see what computers see
![Page 15: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/15.jpg)
This is how a computer represents it
![Page 16: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/16.jpg)
And so is this …
![Page 17: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/17.jpg)
And so are these!
![Page 18: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/18.jpg)
Why study computer vision?• Vision is useful: Images and video are everywhere!
![Page 19: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/19.jpg)
Vision as measurement device
over
![Page 20: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/20.jpg)
Special effects: shape and motion capture
Source: S. Seitz
![Page 21: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/21.jpg)
3D Modeling
http://www.photogrammetry.ethz.ch/research/cause/3dreconstruction3.html
![Page 22: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/22.jpg)
Face recognition: Apple iPhoto software
http://www.apple.com/ilife/iphoto/
![Page 23: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/23.jpg)
Biometrics
How the Afghan Girl was Identified by Her Iris Patterns
Source: S. Seitz
![Page 24: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/24.jpg)
Biometrics
Fingerprint scanners on many new laptops, other devices
Face recognition systems now beginning to appear more widely http://www.sensiblevision.com/
Source: S. Seitz
![Page 25: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/25.jpg)
Optical character recognition (OCR)
Technology to convert scanned docs to text • If you have a scanner, it probably came with OCR
software
Digit recognition, AT&T labs
License plate readers http://en.wikipedia.org/wiki/Automatic_number_plate_recognition
Source: S. Seitz
![Page 26: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/26.jpg)
Mobile visual search: Google Goggles
![Page 27: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/27.jpg)
Google Art Museum Project
Navigate museums of the world
![Page 28: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/28.jpg)
![Page 29: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/29.jpg)
Automotive safety
• Mobileye: Vision systems in high-end BMW, GM, Volvo models – “In mid 2010 Mobileye will launch a world's first application of full
emergency braking for collision mitigation for pedestrians where vision is the key technology for detecting pedestrians.”
Source: A. Shashua, S. Seitz
![Page 30: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/30.jpg)
Vision in supermarkets
LaneHawk by EvolutionRobotics “A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket,and with LaneHawk,you are assured to get paid for it… “ Source: S. Seitz
![Page 31: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/31.jpg)
Vision-based interaction (and games)
Nintendo Wii has camera-based IR tracking built in. See Lee’s work at CMU on clever tricks on using it to create a multi-touch display!
Source: S. Seitz
Assistive technologies
Sony EyeToy
Xbox and Kinect sensor
![Page 32: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/32.jpg)
Classification
![Page 33: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/33.jpg)
Vision as a source of semantic information
slide credit: Fei-Fei, Fergus & Torralba
![Page 34: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/34.jpg)
Object categorization
sky
building
flag
wallbanner
bus bus
face
street lamp
caslridse credit: Fei-Fei, Fergus & Torralba
![Page 35: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/35.jpg)
Challenges: viewpoint variation
Michelangelo 1475-1564 slide credit: Fei-Fei, Fergus & Torralba
![Page 36: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/36.jpg)
Challenges: illumination
image credit: J. Koenderink
![Page 37: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/37.jpg)
Challenges: scale
slide credit: Fei-Fei, Fergus & Torralba
![Page 38: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/38.jpg)
Challenges: deformation
Xu, Beihong 1943
slide credit: Fei-Fei, Fergus & Torralba
![Page 39: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/39.jpg)
Challenges: occlusion
Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba
![Page 40: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/40.jpg)
Challenges: background clutter
![Page 41: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/41.jpg)
Challenges: Motion
![Page 42: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/42.jpg)
Challenges: object intra-class variation
slide credit: Fei-Fei, Fergus & Torralba
![Page 43: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/43.jpg)
Challenges: local ambiguity
slide credit: Fei-Fei, Fergus & Torralba
![Page 44: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/44.jpg)
Levels of complexity
• Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)
• Midlevel vision – assembly of local information (segmentation, contour completions, grouping)
• Scene analysis – recognition of objects, scenes • Active vision – how to control and use the
resources to adjust the sensor to gather additional information
• Goal directed vision – control behaviors based on visual information
![Page 45: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/45.jpg)
Contents of the ClassImage Processing, Low-level and Mid-level Vision :
• Image sensing, lenses • Non-traditional sensors & perceptual coordinate systems • Photometry and Color • Filtering, correlation, convolution, noise • Fourier transform • Edge detection, Boundary detection • Hough transforms • Features, Corners, SIFT features • Image and Motion • Segmentation • Texture Analysis Multiple view Geometry for Robotics: • Geometric transforms • Projective geometry • Camera Calibration • Epipolar geometry • Stereopsis • Optical flow • Tracking
Image Recognition • Recognition of specific objects • Recognition using Machine Learning, SVM, HOG features • Recognition using Neural Networks • Applications of Recognition
![Page 46: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/46.jpg)
Short description of Projects
![Page 47: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/47.jpg)
Homework 1
• Review of estimation • LS estimation TLS estimation ,
LS with Regularization and RANSAC, applied to the problem of line fitting
![Page 48: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/48.jpg)
Project 1: Color Segmentation with GMM
Detect the ball in images “seen” by Nao.
You will learn about Color imaging and about Clustering approaches (K-mean and GMMs)
![Page 49: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/49.jpg)
Homework 2: Image Features and Warping
The project involves: corner detection and geometric transformations between image planes
![Page 50: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/50.jpg)
Project 2: Panorama Stitching
Use the image features to derive the transformation between images and blend them.
![Page 51: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/51.jpg)
Project 3: Segmentation with Graphcuts
Segment foreground from background in log-polar coordinates using Edge, Color, Texture, and Motion information.
![Page 52: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/52.jpg)
Homework 3:Image classification using HOGs and Bag of Words
![Page 53: Computer Vision CMSC 426Levels of complexity • Early vision – local operations, compute maps, or statistics of individual pixels (edges, motion fields, depth maps)](https://reader035.vdocuments.net/reader035/viewer/2022070114/6070f4863357ae5164654521/html5/thumbnails/53.jpg)
Project 4:Image classification using CNNs
Write a CNN, then train it as a classifier