course 1: room 6de computational photography a.2: concepts tumblin

of 34/34
Course 1: Room 6DE Computational Photography A.2: Concepts A.2: Concepts Tumblin Tumblin

Post on 14-Jan-2016




2 download

Embed Size (px)


  • Course 1: Room 6DE Computational PhotographyA.2: Concepts Tumblin

  • Jack TumblinNorthwestern University A2: Core Concepts(30 minutes)

  • Focus, Click, Print: Film-Like Photography Light + 3D Scene:Illumination, shape, movement, surface BRDF, Angle(,)Position(x,y)2D Image:InstantaneousIntensity Map Center of Projection(P3 or P2 Origin)Ray BundlesRay Bundles

  • Film-Like PhotographyThought Experiment:Side-by-side digital camera & film camera.

    COMPARE:Digital Camera result. Digitized (Scanned) Film result.

    ? Can we See more, Do more, Feel more? ? Has photography really changed yet ?

  • scene displayScene LightIntensitiesDisplayLight IntensitiesPixel values (scene intensity? display intensity? perceived intensity? blackness/whiteness ?)displayDigitally Perfected Photography?

  • Film-Like PhotographyFilm Camera design assumptions:Instantaneous light measurementOf focal plane image behind a lens.Reproduce those amounts of light.

    Implied: What we see is focal-plane intensities. well, nowe see much more! (seeing is deeply cognitive)

  • Our Definitions Film-like Photography:Static instantaneous record of the 2D image formed by a lensDisplay image sensor image

    Computational Photography:displayed image sensor imageA more expressive, controllable displayed result, transformed, merged, decoded sensor data

  • What is Photography?

    Safe answer:

    A wholly new, expressive medium (ca. 1830s)

    Manipulated display of what we think, feel, want, Capture a memory, a visual experience in tangible formpainting with light; express the subjects visual essenceExactitude is not the truth. Henri Matisse

  • What is Photography?A bucket word: a neat container for messy notions (e.g. aviation, music, comprehension)

    A record of what we see, or would like to see, in tangible form.Does film photography always capture it? No!

    What do we see? What is missing?

    Harold Doc Edgerton 1936

  • DisplayRGB(x,y,tn)ImageI(x,y,,t)Light & Optics3D Scenelight sources, BRDFs,shapes,positions,movements,Eyepoint position, movement,projection,PHYSICALPERCEIVEDWhat is Photography?Exposure Control, tone mapScenelight sources,BRDFs,shapes,positions,movements,Eyepointposition, movement,projection,VisionPhoto: A Tangible RecordEditable, storable as Film or Pixels

  • 3D Scene?light sources, BRDFs,shapes,positions,movements,Eyepoint?position, movement,projection,MeaningVisualStimulus3D Scenelight sources, BRDFs,shapes,positions,movements,Eyepoint position, movement,projection,PHYSICALPERCEIVED or UNDERSTOODUltimate Photographic GoalsVisionSensor(s)ComputingLight & OpticsPhoto: A Tangible Record

  • A Driving Problem: Museum ArtifactsMillions of delicate, fascinating treasures:< 1% of Smithsonian collection ever exhibitedsparse $, displays; conservation limits access

  • Current Archives: Not rich enoughFixed, static viewpointFixed, static lighting Custom light: impracticalConflates shapes, materials, shadows, texture, highlights,

    Can you understand this shape?

  • Current Archives: Not rich enoughFixed, static viewpointFixed, static lighting Custom light: impracticalConflates shapes, materials, shadows, texture, highlights,

    Can you understand this shape?

    What digital archive can best match in-hand, direct examination ?

    What is missing?

  • Missing: Reliable Visual Boundaries5 ray sets explicit geometric occlusion boundariesRamesh Raskar, MERL, 2004

  • Missing: Occlusion Removal Rollout Photographs Justin Kerr: Slide idea: Steve Seitz BOTH capture visual appearance; BOTH should be easy to make!

  • Missing: Viewpoint Freedom Multiple-Center-of-Projection Images Rademacher, P, Bishop, G., SIGGRAPH '98

  • Missing: InteractionAdjust everything: lighting, pose, viewpoint, focus, FOV,Winnemoller EG 2005: after Malzbender, SIGG2001

  • Missing: Expressive Time ManipulationsWhat other ways better reveal appearance to human viewers?(Without direct shape measurement? )Time for space wiggle. Gasparini, 1998.Can you understand this shape better?

  • Photographic Signal: Pixels RaysCore ideas are ancient, simple, seem obvious:Lighting: ray sourcesOptics: ray bending/folding devicesSensor: measure lightProcessing: assess itDisplay: reproduce it

    Ancient Greeks: eye rays wipe the world to feel its contents

  • The Photographic Signal PathClaim: Computing can improve every stepLight SourcesSensorsData Types, ProcessingDisplayRaysOpticsOpticsSceneRaysEyes

  • Review: How many Rays in a 3-D Scene?A 4-D set of infinitesimal members. Imagine:Convex Enclosure of a 3D scene Inward-facing ray camera at every surface pointPick the rays you need for ANY camera outside. 2D surface of cameras, 2D ray set for each camera, 4D set of rays.(Levoy et al. SIGG96)(Gortler et al. 96) +

  • 4-D Light Field / LumigraphMeasure all the outgoing light rays.

  • 4-D Illumination FieldSame Idea: Measure all the incoming light rays

  • 4D x 4D = 8-D Reflectance FieldRatio: Rij = (outgoing rayi) / (incoming rayj)

  • Is a 4-D Light Source Required?

  • Is A 4D Camera Required? e.g. MIT Dynamic Light Field Camera 2002Multiple dynamic Virtual ViewpointsEfficient Bandwidth usage: send only what you seeYang, et al 200264 tightly packed commodity CMOS webcams30 Hz, Scaleable, Real-time:

    or is it just more film-like cameras, but now with computers! ? Is this the whole answer?

  • Or do Ray Changes Convey Appearance?5 ray sets explicit geometric occlusion boundariesRamesh Raskar, MERL, 2004

  • These rays + all these rays give me

    MANY more useful details I can examineOr do Ray Changes Convey Appearance?

  • Mild Viewing & Lighting Changes; Are these Enough?Convicing visual appearance:Is Accurate Depth really necessary? a few good 2-D images may be enoughImage jets, Level Sets, and SilhouettesLance Williams, talk at Stanford, 1998.

  • The Ideal Photographic SignalI CLAIM IT IS: All Rays? Some Rays? Changes in Some Rays Photographic ray space is vast and redundant >8 dimensions: 4D view, 4D light, time, ,? Gather only visually significant ray changes ?

    ? What rays should we measure ? ? How should we combine them ?? How should we display them ?

  • Future PhotographyNovel IlluminatorsNovel CamerasScene: 8D Ray ModulatorGeneralized SensorsGeneralized Processing4D Ray SamplerRay ReconstructorGeneral Optics: 4D Ray BendersRecreated 4D Light fieldLightsModulators4D Incident LightingViewed 4D Light FieldGeneral Optics: 4D Ray Benders Generalized Display Novel Displays

  • Beyond Film-Like PhotographyCall it Computational Photography: To make meaningful ray changes tangible,

    Optics can do moreSensors can do more Light Sources can do moreProcessing can do more by applying low-cost storage, computation, and control.

  • We still hang on to the mistaken notion that were copying the image formed by the lens to the image formed by the display, an intrinsically 2D process to approximate the appearance of a 3D scene.

    Weve confused the PROCESS of photography with its PURPOSE and GOALS.At first, it was a wonder we could do it at all:Now its a wonder how easily we take (bad) photos,how many choices and adjustments we can make to our cameras to make them better, but even more importantly, how many OTHER CHOICES we have besides a lens and a box holding a sensitized plate. We have many other choices for image formation (tomography, coded image methods, structured lighting, coded aperture, etc. etc.) for lighting (projectors, movable sources, multispectral sources, tuneable lasers, flash, strobe, reflectors, Schlieren retro-reflectors), and for display (interactive devices; light-sensitive displays, HDR, etc.)

    . Yet look at how much of high-quality photography is dominated by overcoming device limitations, artful choices of lighting, and adjusting the myriad settings our cameras and digital darkrooms offer to us.But let me pose a question: Do our digital cameras give us ANYTHING NEW? I argue that digital photography is EXACTLY THE SAME as film photography,except we replaced film with a grid of light meters.We cant DO any more, we cant SEE anymore, we cant FEEL any more than we could with film.

    Digital Photography is almost entirely a matter of copying---just like film!The underlying assumption is that we copy a 2D scene to a 2D display, and if we do it accurately, were done.

    A common misconception:ItsUm, er. This isnt

    Humans see basic, partial information about boundaries, shape, occlusion, lighting, shadows and texture, with few discernible difficulties with high dynamic range, resolution, or noise, lighting, or exposure. This basic data is usually difficult or impossible to reliably extract from pixels. But why require extraction? Instead, we should encode this information as part of the image itself. Towards this goal, Bixels offer a straightforward way to represent intensity and gradient discontinuities within images with subpixel precision, at a fixed cost an additional 8 bits per pixel.

    BLACKEST OF BLACK BOXESWhat we would like is something that more directly describes the visual experience, --something that, with some computing, would allow a computer-equipped display to construct a display image, one that, based on the viewing conditions, has the best chance of evoking the desired perceptions of the original scene.

    Occlusion often hides visually important features that help us understand what we see. GREEKS: Photog. SEEMS obvious because what we gather can be described by ray geometryif we think of our retina as a sensory organ, we WIPE it across the scene, as if light let our retina reach out and touch what is around us. So lets look further into that:; lets consider light as a way of exploring our surroundings without contact, a magical way of transporting the the perceivable properties of our surroundings into our brain. EVEN THE GREEKS knew this idea wellthey used RAYS in exploration of vision, and described how rays going through a small aperture mapped angle to position

    We tend to think of photography as capturing light, not visual impressions. BUT VISUAL IMPRESSIONS DEPEND ON EVERY STAGE OF The Photographic Signal Path

    If we REPLACE 2D PIXELS WITH NOTIONS OF MEANINGFUL CHANGES IN SETS OF RAYS, then (JT: MISSING HERE: LIGHT IS LINEAR)There are various light-stage like approaches to capture the information needed for image based rendering/lighting. However, most approaches areExpensiveNon-portableLimited size

    This is because these systems attempt to capture exact representations of the reflectance field, and need to be extensively calibrated. This may be seen as an overkill for the task of relighting where we only want something that looks good instead of something that is 100% physically accurate.Other groups have built hardware for these purposes, and heres one example from MITthe Dynamic Light Field Camera. This array of commodity webcams was used for real-time light field viewing. Their system dealt with the large video data rates by transmitting only the pixel information necessary to create one or some small number of light field views. So although they could move the viewpoint around, they discarded most of the video data. This meant, for example, that they could not record the light field video to view later.

    Notes: they dont have CMUs 9 second limitation, but at a penalty. They overcame the bandwidth issues of 64 cameras by reducing both the resolution and quality of their cameras, and they dont record the LF. You cant go back later and analyze or even replay it. Also, their commodity webcams did not have synchronization inputs. Auto white balance (which doesnt work) and manual tweaking8 cameras per PCRays are an infinitesimal discrete, computed abstractionthey match what we perceive (an infinitely sharp world of disjoint objects), and they also escape a great deal inconvenient physics that entangles photography in practical difficulties They ignore rarely-perceived effects (diffraction, noise, fluorescence) that are computationally MUCH more difficult.ASIDE: Rays largely abandoned in modern optics & lens designreplaced by `Fourier Optics methods that properly account for diffraction effects, coherent (laser) light and nearly all wave propagation effects (see the classic textbook by Goodman, 1968). WHY USE Rays? They are ENOUGH

    Up until the time of machine-assisted image making, none of these efx of physics were a problemhuman perception guided image making instead. (FREDO DURAND has a WONDERFUL DISCUSSION and SURVEY of thisTowards Computer Depiction/ CITE).

    THERE ARE AT LEAST 4 blocks that we can generalize and improve: lighting, optics, sensors, processing, (display: light sensitive display)

    (DONT EXPLAIN THE INTERIORS! Side-tracks your talk too much, audience gets lost, you lose momentum to carry them along, they stop considering your larger thesis statement)