Computational Photography - ?· Computational photography • More than digital photography • Arbitrary…

Download Computational Photography - ?· Computational photography • More than digital photography • Arbitrary…

Post on 25-Aug-2018

215 views

Category:

Documents

3 download

Embed Size (px)

TRANSCRIPT

  • Computational Photography

    Matthias Zwicker University of Bern

    Fall 2012

  • Today Course organization

    Course overview

    Image formation

  • Course organization Instructor

    Matthias Zwicker (zwicker@iam.unibe.ch)

    Teaching Assistant

    Daniel Donatsch (donatsch@iam.unibe.ch)

    mailto:zwicker@iam.unibe.chmailto:knaus@iam.unibe.ch

  • Course organization Lecture

    Mondays, 14:00-16:00

    Engehaldenstrasse 8, Room 3

    Exercises

    Mondays, 16:00-17:00

    Engehaldenstrasse 8, Room 3

  • Class web page Class overview

    http://www.cgg.unibe.ch/teaching/computational-photography

    http://www.cgg.unibe.ch/teaching/computational-photography

  • ILIAS Use your campus account to log in

    Join course Magazin Weitere Institutionen; Weiterbildungen und Studiengnge BeNeFri Joint Master in Computer Science HS2012 2012HS: 31051 Computational Photography

    Lecture slides

    Exercise description & material

    Additional reading material

    Forum

    Any questions and discussions related to class material and exercises

    https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=1https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=711https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447636https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447636https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447639https://ilias.unibe.ch/repository.php?cmd=frameset&ref_id=447650

  • Exercises 6 assignments

    Programming projects

    Matlab Available in ExWi pool

    Exercises on paper

  • Exercises Final grade: 40% exercises, 60% final exam To qualify for final exam: need 70% of

    exercise score Late penalty

    50% of original score Exceptions for military service, illness

    Collaboration Discussion among students is encouraged Each student must write up and turn in his/her

    own solution If we detect copied material, you will need to talk

    to us and explain your material in person; if we are not satisfied, you will not get credit

  • Final exam Written, 105 minutes

    Bring two A4 sheets (4 pages) of hand written notes

    Relevant material: slides and exercises

    Wikipedia links not part of class material, but may be useful to better understand concepts discussed in class

    Date: February 2013

  • Prerequisites Familiarity with

    Linear algebra (matrix calculations, linear systems of equations, least squares problems)

    Programming experience

  • Today Course organization

    Course overview

    Image formation

  • Computational photography Topics of this class

    Role of computation, algorithms in digital photography today

    Algorithms to extend and improve capabilities of digital photography in the future

  • Photography Traditionally

    Measuring light

    Optics focuses light on sensor

    Sensor records image

    Sensors

    Digital Film

    http://en.wikipedia.org/wiki/Single-lens_reflex_camera http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera

    http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camerahttp://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera

  • Computational photography More than digital photography

    Arbitrary computation between light measurement and final image Light measured on sensor is not final image Computation enhances and extends capabilities of

    digital photography Two types of computation

    1. Post-process after traditional imaging 2. Design of new camera devices that require

    computation to form an image Overview of recent research

    http://en.wikipedia.org/wiki/Computational_photography

    http://en.wikipedia.org/wiki/Computational_photography

  • Removing imaging artifacts Denoising & deblurring

    http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf

    Blurry + Output Noisy Algorithm

    http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf

  • Removing imaging artifacts High dynamic range images & tone mapping

  • Image manipulation Panoramas

    http://en.wikipedia.org/wiki/Image_stitching

    http://en.wikipedia.org/wiki/Image_stitching

  • Computational optics

    Coded aperture

    Captured image, slightly blurry everywhere

  • Computational optics

    Recovered depth

    Refocused image Sharp foreground, blurry background

  • Focus of class Fun with digital photography and

    computer programming

    Algorithms and computational techniques with potential applications in the consumer domain

    Mostly software, less hardware

    Recent research

  • What you will learn Basic understanding of photography, light,

    and color Practical experience with implementation of

    algorithms for image processing & computational photography

    Cool and creative applications of mathematical tools Fourier transforms Linear and non linear filtering Optimization techniques (least squares, iteratively

    re-weighted least squares, graph cuts) Probabilistic models

    Many applications beyond processing images!

  • Related areas, not covered Image processing for scientific applications

    Physics, biology, etc.

    Optics, lens design

    Photosensors, sensor design

    Computational imaging

    Tomography, radar, microscopy

    3D imaging

    Using photo processing tools, e.g. Photoshop

    Artistical aspects of photography

  • Syllabus 1. Introduction, image formation 2. Color & color processing 3. Dynamic range & contrast 4. Sampling, reconstruction, & the frequency domain 5. Image restoration: denoising & deblurring 6. Image manipulation using optimization 7. Gradient domain image manipulation 8. Warping & morphing 9. Panoramas 10. Automatic alignment 11. Probabilistic image models 12. Light fields 13. Capturing light transport

    http://www.cgg.unibe.ch/teaching/computational-photography

    http://www.cgg.unibe.ch/teaching/computational-photography

  • Cameras, image artifacts

    Image formation

  • Color Color perception, color spaces, color

    measurement, color processing

  • Dynamic range & contrast HDR imaging

    http://en.wikipedia.org/wiki/High_dynamic_range_imaging http://en.wikipedia.org/wiki/Tone_mapping

    http://en.wikipedia.org/wiki/High_dynamic_range_imaginghttp://en.wikipedia.org/wiki/Tone_mapping

  • Sampling, reconstruction Sampling artifacts

    Frequency domain analysis

    Spatial Domain Frequency Domain

  • Image restoration Denoising & deblurring

    Blurry input Deblurred output

    Estimated blur kernel (scaled) http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/

    http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/

  • Image manipulation using optimization Photomontage, matting, colorization

    http://grail.cs.washington.edu/projects/photomontage/

    http://www.cs.huji.ac.il/~yweiss/Colorization/

    http://grail.cs.washington.edu/projects/digital-matting/image-matting/

    http://grail.cs.washington.edu/projects/photomontage/http://www.cs.huji.ac.il/~yweiss/Colorization/http://grail.cs.washington.edu/projects/digital-matting/image-matting/

  • Gradient domain manipulation Poisson equation

    http://portal.acm.org/citation.cfm?id=882269

    http://portal.acm.org/citation.cfm?id=882269

  • Warping & morphing

  • Panoramas Automatic alignment, stitching

    http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/

    http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/

  • Probabilistic models Faces, textures

    http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf

    http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf

  • Beyond 2D images

    Light fields

    http://www-graphics.stanford.edu/papers/fourierphoto/

    http://www-graphics.stanford.edu/papers/fourierphoto/http://www-graphics.stanford.edu/papers/fourierphoto/http://www-graphics.stanford.edu/papers/fourierphoto/

  • Capturing light transport Dual photography

    http://www-graphics.stanford.edu/papers/dual_photography/

    http://www-graphics.stanford.edu/papers/dual_photography/http://www-graphics.stanford.edu/papers/dual_photography/http://www-graphics.stanford.edu/papers/dual_photography/

  • Today Course organization

    Course overview

    Image formation

  • Models of light

  • Question Why is there no image on a white piece of

    paper?

  • Question Why is there no image on a white piece of

    paper?

    Receives all light rays

    Images from all viewpoints

    Need to select light rays for specifice image, viewpoint

    How?

  • Invented by Alhazen, 10th century http://en.wikipedia.org/wiki/Pinhole_camera

    Pinhole camera

    http://en.wikipedia.org/wiki/Pinhole_camera

  • Limitations Small pinhole: sharper image, longer exposure

    Larger pinhole: blurrier image, shorter exposure

  • Camera model Thin lens, aperture, shutter, film

  • Lenses Gather more light

    Proportional to area of lens aperture

    Sharp image if focused

    Use refraction http://en.wikipedia.org/wiki/Refraction

    Lens

    http://en.wikipedia.org/wiki/Lens_(optics)

    Scene point (emits or

    reflects light) Image of

    scene point

    http://en.wikipedia.org/wiki/Refractionhttp://en.wikipedia.org/wiki/Lens_(optics)

  • Lenses Pinhole Lens

    6 sec. exposure 0.01 sec exposure

  • Thin lens model http://en.wikipedia.org/wiki/Thin_lens

    Theoretical model for well-behaved lenses

    Properties

    1. All parallel rays converge at focal length

    2. Rays through the center are not deflected

    Same perspective image as pinhole placed at center of lens

    http://en.wikipedia.org/wiki/Thin_lens

  • Thin lens model How are arbitrary rays deflected when

    passing through a thin lens?

    1. Parallel rays converge at focal length f

  • Thin lens model 2. Rays through center are not deflected

  • Thin lens model Similar triangles

  • Thin lens model More similar triangles

  • Thin lens model Thin lens formula

    All rays passing through a single point y on a plane at distance in front of the lens will pass through a single point y at distance behind the lens

  • Thin lens model Focus at infinity:

    Film at distance f

    Closest focusing distance:

    Film at infinity

    Film plane

    Object

  • Thin lens model Out of focus film plane results in spherical

    blur

    Out of focus film planes

    Spherical blur

  • Properties of real lenses Mostly undesired!

    Aberrations

    Spherical aberration Chromatic aberration

    Distortion

    Barrel distortion Pincushion distortion

    Etc.

    Barrel & pincushion distortion

  • Question Whats the advantage of a lens with a

    short focal length? In what situation would this be useful?

    Whats the advantage of a lens with a long focal length? In what situation would this be useful?

  • Camera model Thin lens, aperture, shutter, film

  • Aperture Amount of light captured at sensor

    proportional to area of aperture

    Aperture

  • Aperture Blurriness of out of focus objects depends

    on aperture size

    Aperture size determines depth of field: depth range that is sharp in image

    Aperture

  • Depth of field

  • Circle of confusion Also called blur circle

    Calculation of radius c Lens focused at S1 Object at S2 Aperture A Focal length f sensor

    http://en.wikipedia.org/wiki/Circle_of_confusion

    Proportional to A

    http://en.wikipedia.org/wiki/Circle_of_confusion

  • f-number http://en.wikipedia.org/wiki/F-number

    Definition: (focal length) / (diameter of aperture)

    Large aperture means small f-number

    Practice: f-stops increase by factors of

    f/2.0, f/2.8, f/4, f/5.6, f/8 Aperture area gets halved in each step

    http://en.wikipedia.org/wiki/F-number

  • f-number Smaller f-number

    Larger aperture Capture more light Small (shallow) depth of field

    Larger f-number

    Smaller aperture Capture less light Large depth of field

  • Camera model Thin lens, aperture, shutter, film

  • Shutter speed http://en.wikipedia.org/wiki/Shutter_speed

    Determines time the film is exposed to light

    Amount of light captured is proportional to exposure time

    Long exposure leads to motion blur

    http://en.wikipedia.org/wiki/Shutter_speed

  • Reciprocity Amount of light captured stays

    same if exposure is doubled and aperture area is halved (or vice versa)

  • Reciprocity Which exposure/aperture combination?

  • Camera model Thin lens, aperture, shutter, film

  • Film Film/sensor responds roughly linearly to light

    Double the amount of light leads to double the recorded value

    Film speed: sensitivity of film to light

    Digital photography analog: sensor gain (scaling or amplification factor)

    Measured using ISO scale

    Linear: sensitivity is proportional to ISO value Double ISO value, halve the exposure time, get

    same recorded value

  • Film Trade-off: higher gain, more noise

    ISO 100 ISO 3200

  • Film Underexposure

    Not enough light, image too dark Overexposure

    Film or sensor is saturated Clipping of highlight details

    Good exposure Overexposure Underexposure

  • Conclusions Simple camera model

    Thin lens, aperture, shutter, film

    Photographs often have undesired artifacts

    Distortions, color artifacts, blur, noise, under/overexposure

    Goal

    Develop algorithms to remove artifacts after image is captured

  • References Photography, by London, Upton, Stone

  • Next time Color, color processing

    Computational PhotographyTodayCourse organizationCourse organizationClass web pageILIASExercisesExercisesFinal examPrerequisitesTodayComputational photographyPhotographyComputational photographyRemoving imaging artifactsRemoving imaging artifactsImage manipulationComputational opticsComputational opticsFocus of classWhat you will learnRelated areas, not coveredSyllabusImage formationColorDynamic range & contrastSampling, reconstructionImage restorationImage manipulation using optimizationGradient domain manipulationWarping & morphingPanoramasProbabilistic modelsLight fieldsCapturing light transportTodayModels of lightQuestionQuestionPinhole cameraLimitationsCamera modelLensesLensesThin lens modelThin lens modelThin lens modelThin lens modelThin lens modelThin lens modelThin lens modelThin lens modelProperties of real lensesQuestionCamera modelApertureApertureDepth of fieldCircle of confusionf-numberf-numberCamera modelShutter speedReciprocityReciprocityCamera modelFilmFilmFilmConclusionsReferencesNext time

Recommended

View more >