advanced topics in computer vision” - york universitymbrown/eecs6323/... · “advanced topics in...
TRANSCRIPT
1
“Advanced Topics in Computer Vision”
Computational Photography
Prof. Michael S. BrownEECS – Lassonde School of Engineering
Introduction
Lecturer
• Dr. Michael S. Brown• Professor
EECS Department
Lassonde School of Engineering
• Office Location
– Lassonde 3022
• Office Hours
– Please arrange by email: [email protected]
(Subject: EECS 6323)
Brown 2
3
Welcome
• You probably have questions:
– What is computational photography?
– What was wrong with regular photography?
– What am I going to learn?
– How much work is this course?
– Will I get an A?
• Hope to answer these questions in today’s lecture
4
Tentative - Assessment
• 4-5 Assignments– Most assignments will have multiple subproblems from which you can select
• Tentative Assignments– Assignment 1: basic image processing routines
• Thresholding, edge detection, deblurring, etc. . .
– Assignment 2: Interactive computer vision/Computational Processing• Interactive Image Snapping
• Interactive Image Segmentation
• Gradient-domain Cut-and-pasting
– Assignment 3: Computational Illumination• HDR imaging + tone mapping
• Flash/No-Flash low-light photography
• Multi-flash camera (gradient camera)
– Assignment 4: Computational Optics• Coded exposure camera
• Focal Stacking (everywhere in-focus-image)
• Image pre-conditioning for projector blur
– Assignment 5: Misc/Vision Related• Seam Carving
• Texture Synthesis
Assignments will be marked
via individual one-on-one
sessions with me. I’ll likely
mark two assignments at a
time to minimize meetings.
What is Computational
Photography?
• Definition 1: the use of photographic imagery to create
graphics content
• Definition 2: The use of computational techniques to
overcome limitations of conventional photography
Definition from: A. Efros, Berkeley
Originally called Image-Based Rendering
(IBR), but calling it Computational
Photography is much cooler .
We will focus primarily on
topics/research related to
definition 2.
5
What is Computational
Photography?
• Definition 1: the use of photographic imagery to
create graphics content
• Definition 2: The use of computational techniques to
overcome limitations of conventional photography
6
Definition from: A. Efros, Berkeley
The Realism Spectrum
+ easy to create new worlds
+ easy to manipulate objects/ viewpoint
- very hard to look realistic
+ instantly realistic
+ easy to acquire
- very hard to manipulate
objects/viewpoint
Computer Graphics PhotographyComputationalPhotography
RealismManipulationEase of capture
Slide credit: A. Efros
Image-based rendering . . . Exploit
images to help improve the realism of
graphics. . .
Where have you seen IBR?
8
Texture maps Environment Maps
Mosaics
What is Computational
Photography?
• Definition 1: the use of photographic imagery to create
graphics content
• Definition 2: The use of computational techniques to
overcome limitations of conventional photography
9
Definition from: A. Efros, Berkeley
• Blur, camera shake, noise, damage
Limitations of Conventional Photography
10
• Limited resolution
Limitations of Conventional Photography
11
• Bad color / no color
Limitations of Conventional Photography
12
• Unwanted objects
Limitations of Conventional Photography
13
• Limited dynamic range
Limitations of Conventional Photography
14
• Single viewpoint, static 2D picture
Limitations of Conventional Photography
15
• Single depth of focus
Limitations of Conventional Photography
16
Where can we make changes?
17Shree Nayar’s (U. Columbia) vision for “Computational Cameras”...
What else can we do?
18
Put the “human in the loop” when editing photographs.
Three main topics covered
• Computational Processing
– Interactive Computer Vision
– Process an image, with the human in the loop
• Examples: Image segmentation, Image Colorization, Compositing . . .
• Computational Optics
– Modification of the optics
– Assume image is processed with prior knowledge of the
modification
• Examples: Coded exposure, coded aperture, translating imagery, hybrid
cameras . . .
• Computational Illumination
– Modification/control of the illumination and or exposure
– Assume image will be processed using the prior knowledge of
the illumination manipulation
• Examples: HDR imaging, dual photography, flash/no flash imaging . . .
19
20
Another opinion . . .
What is computational
photography?
Following notes (with some modifications) are from the SIGGRAPH’2007 course on Computational Photography, by Prof. Jack Tumblinfrom Northwestern University, USA.
Focus, Click, Print:
‘Film-Like Photography’
2D Image:
‘Instantaneous’
Intensity Map
Light + 3D Scene:
Illumination, shape, movement, surface BRDF,…
Ang
le(
,)
Positio
n(x
,y)
‘Center of
Projection’
(P3 or P2
Origin)
Ray Bundles
Ray Bundles
Display on a monitor(or print it).
Film-Like Photography
Thought Experiment:
Side-by-side digital camera & film camera.
• COMPARE:
– Digital Camera result.
– Film result.
Can we See more, Do more, Feel more?
Has photography really changed yet ?
scene
display
Scene
Light
Intensities
Display
Light
Intensities
‘Pixel values’(scene intensity? display intensity?
perceived intensity? ‘blackness/whiteness’ ?)
display
Digitally Perfected Photography?
‘Film-Like’ Photography
Film Camera design assumptions:
– ‘Instantaneous’ light measurement…
– Of focal plane image behind a lens.
– Reproduce those amounts of light.
Implied:
“What we see is
focal-plane intensities.”
well, no…we see much more!
(seeing is deeply cognitive)
Jack’s Definitions
• ‘Film-like’ Photography:
– Static ‘instantaneous’ record of
the 2D image formed by a lens
Display image sensor image
• ‘Computational’ Photography:
– displayed image sensor image
– A more expressive, controllable displayed result,
transformed, merged, decoded sensor data
What is Photography?
Safe answer:
A wholly new,expressive medium (ca. 1830s)
• Manipulated display of what we think, feel, want, …– Capture a memory, a visual experience in tangible form
– ‘painting with light’; express the subject’s visual essence
– “Exactitude is not the truth.” –Henri Matisse
• A ‘bucket’ word: a neat container for messy notions(e.g. aviation, music, comprehension)
• A record of what we see,or would like to see,in tangible form.
• Does ‘film’ photography
always capture it? No!
• What do we see?What is missing?
What is Photography?
Harold ‘Doc’ Edgerton 1936
DisplayRGB(x,y,tn)
ImageI(x,y,λ,t)
Light &
Optics3D Scenelight sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepointposition,
movement,
projection,
…
PHYSICAL PERCEIVED
What is Photography?
Exposure
Control,
tone mapScenelight sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepointposition,
movement,
projection,
…
Vis
ion
Photo: A Tangible Record
Editable, storable as
Film or Pixels
3D Scene?light sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepoint?position,
movement,
projection,
…
Meaning…
Visual
Stimulus
3D Scenelight sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepointposition,
movement,
projection,
…
PHYSICALPERCEIVED or UNDERSTOOD
Ultimate Photographic Goals
Vis
ion
Sen
sor(
s)
Com
pu
tin
g
Light &
Optics
Photo: A Tangible Record
Millions of delicate, fascinating treasures:
– < 1% of Smithsonian collection ever exhibited
– sparse $, displays; conservation limits access
A Driving Problem: Museum Artifacts
Current Archives:
Not rich enough
• Fixed, static viewpoint
• Fixed, static lighting
• Custom light: impractical
• Conflates shapes,
materials, shadows,
texture, highlights, …
Can you understand
this shape?
Current Archives:
Not rich enough
• Fixed, static viewpoint
• Fixed, static lighting
• Custom light: impractical
• Conflates shapes,
materials, shadows,
texture, highlights, …
Can you understand
this shape?
What ‘photo archive’
can best match in-hand,
direct examination ?
What is missing?
Missing:
Reliable Visual Boundaries5 ray sets explicit geometric occlusion boundaries
Ramesh Raskar, MERL, 2004
Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz
http://research.famsi.org/kerrmaya.html
Missing: Occlusion Removal
BOTH capture visual appearance;
BOTH should be easy to make!
Missing:
Viewpoint Freedom “Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98
Missing:
Interaction…Adjust everything: lighting, pose, viewpoint, focus, FOV,…
Winnemoller EG 2005: after Malzbender, SIGG2001
Can I moved the light source?
Missing:
Expressive Time Manipulations
What other ways
better reveal
appearance to
human viewers?
(Without direct shape
measurement? )
Time for space wiggle. Gasparini, 1998.
Can you understand
this shape better?
Photographic Signal: Pixels Rays
• Core ideas are ancient, simple, seem obvious:
– Lighting: ray sources
– Optics: ray bending/folding devices
– Sensor: measure light
– Processing: assess it
– Display: reproduce it
• Ancient Greeks:
‘eye rays’ wipe the world
to feel its contents…
http://www.mlahanas.de/Greeks/Optics.htm
The Photographic Signal Path
Claim: Computing can improve every step
Light Sources Sensors Data Types,
Processing
DisplayRays
OpticsOptics
Scene
Rays
Eyes
Review: How many Rays in a 3-D Scene?
A 4-D set of infinitesimal members.
Imagine:
– Convex Enclosure of a 3D scene
– Inward-facing ray camera at every surface point
– Pick the rays you need for ANY camera outside.
2D surface of cameras,
2D ray set for each camera,
4D set of rays.
(Levoy et al. SIGG’96) (Gortler et al. ‘96)
+
The 4D Light Field
If you’re a bit confused, don’t worry, we will talk about this more. . .
4-D Light Field / Lumigraph
Measure all the outgoing light rays of the
scene.
And assumesa fixed illumination of the 3D scene.
4-D Illumination Field
Same Idea: Measure all the incoming light rays
coming into the scene.
Now thinkabout theilluminationof a 3D scene.It is also a 4Dbundle of rays.
4D x 4D = 8-D Reflectance Field
Ratio: Rij = (outgoing rayi) / (incoming rayj)
45[Debevec et al. 2002]
[Debevec et al. 2000] [Masselus et al. 2002]
[Masselus et al. 2003] [Malzbender et al. 2002]
[Matusik et al. 2002]
Is a 4-D Light Source Required?
Is A 4D Camera Required? e.g. MIT Dynamic Light Field Camera 2002
• Multiple dynamic Virtual Viewpoints
• Efficient Bandwidth usage:‘send only what you see’
• 64 tightly packed commodity CMOS webcams
• 30 Hz, Scaleable, Real-time:
or is it just “more film-like cameras, but now with computers!” ?
Is this the whole answer?
Or do Ray Changes Convey Appearance?
5 ray sets explicit geometric occlusion boundaries
Ramesh Raskar, MERL, 2004
Or do Ray Changes Convey Appearance?
• These rays + all these rays give me…
• MANY more useful
details I can examine…
Mild Viewing & Lighting Changes;
Are these Enough?
Convicing visual appearance:
Is Accurate Depth really necessary?
a few good 2-D images may be enough…
“Image jets, Level Sets,
and Silhouettes“
Lance Williams,
talk at Stanford, 1998.
‘The Ideal Photographic Signal’
I (Jack) CLAIM IT IS:
All Rays? Some Rays? Changes in Some Rays
Photographic ray space is vast and redundant>8 dimensions: 4D view, 4D light, time, ,
? Gather only ‘visually significant’ ray changes ?
? What rays should we measure ?
? How should we combine them ?
? How should we display them ?
Future PhotographyNovel Illuminators
Novel Cameras
Scene: 8D Ray Modulator
Generalized
Sensors
Generalized
Processing4D Ray
Sampler
Ray Reconstructor
General Optics:4D Ray Benders
Recreated 4D Light field
Lights
Modulators
4D Incident Lighting
Ge
ne
ral O
ptics:
4D
Ray B
en
ders
Generalized Display
Novel Displays
Beyond ‘Film-Like’ Photography
Call it ‘Computational Photography’:
To make ‘meaningful ray changes’ tangible,
• Optics can do more…
• Sensors can do more…
• Light Sources can do more…
• Processing can do more…
by applying low-cost storage,
computation, and control.
Introduction Summary
• Photography is changing
– Already, most images are “touched up” via software
– Cameras have built-in algorithms that modify the
picture directly (sometimes without you knowing)
• Camera hardware is changing too
– Since we plan to process the image, we can now
explore ways to modify the optics and illumination
with post-processing
– This may change the way lens, flashes are used in
the future
53