Depth and Image from a Single Image

1Fabricating BRDFs at High Spatial Resolution Using Wave OpticsAnat Levin, Daniel Glasner, Ying Xiong, Fredo Durand, Bill Freeman, Wojciech Matusik, Todd Zickler.

Weizmann Institute, Harvard University, MIT12Appearance fabricationGoal: Fabricating surfaces with user defined appearance Applications: - Architecture Product designSecurity markers visible under certain illumination conditions Camouflage- Photometric stereo (Johnson&Adelson 09)

Reflectance AcquisitionFabrication

2Different types of materials reflect light in different ways.Many efforts have been invested in the acquisition of reflectance properties of real material. But today we focus at the inverse problem how can we fabricate surfaces with user designed appearance? Or desired BRDF?This is important for many applications including architecture and product design, security markers visible under certain conditions. Paints with desired appearance properties can be useful for camouflaging, and for computer vision algorithms such as photometric stereo.

3BRDF (Bidirectional Reflectance Distribution Function)

zDot (pixel) unit on surface?x3Since this talk is about BRDF lets redefine it quickly.BRDF is a function describing the amount of energy from illumination direction l, reflected toward viewing direction v, from the area of a certain surface dot that we highlight here in orange.

4

ReflectanceDiffuseShiny

Fabricating spatially varying BRDFHere is a preview of a surface we fabricated. What you are seeing is a silicon wafer etched using a photo lithography process. Whats special about it is that we have designed it such that every small dot on it reflects light in a pre-specified way.For example here are two dots and their BRDFs. For simplicity we plot it as a function of viewing directions assuming a constant illumination.In this case the reflectance from the first point is wider, or more diffuse and the second one is narrower, or more shiny.

45Controlling reflectance via surface micro-structure

ReflectanceDiffuseShinySurface micro structure

What surface micro-structure produces certain reflectances?To achieve these reflectances we cover the area of each dot with a properly designed micro structure.To design BRDF we first need to understand what in the surface micro structure make it reflect light in a certain way.

56SurfaceReflectancePrevious work: BRDF fabrication using micro-facets theory (Weyrich et al. 09)3cmSurface: oriented planner facetsLimited spatial resolution Dot size ~ 3cm x 3cm

6Previous attempts to fabricate surfaces with user defined BRDF are mostly based on the micro-facet theory. Given a target BRDF they construct a surface as a collection of small planar mirrors. The mirrors orientation is selected such that they reflect light in desired directions. This approach produces impressive results but its main limitation is resolution.The size of the smallest dot unit they control is not so small, It is actually 3cm wide.It will look like a BRDF only if you look at it from a few dozen meters away, If you get closer you will start seeing the individual mirror facets.Unfortunately there are inherent reasons why we cannot scale down such a design.

7Micro-facet model: limitations

3cm0.3cm0.03cm0.003cmSurface scaleReflectance

Wave effects at small scales => Substantial deviation from geometric optics prediction7For example, lets try to scale down the same surface, and simulate the reflectance it will produce. If we scale further the reflectance is actually very different than the model prediction. The reason is that the micro facet model which was used to design this surface uses geometric optics theory. However at small scales reflectance is dominated by wave effects such as interference and diffraction. And their output can be very different than the geometric optics prediction.8Previous work: BRDF designWeyrich et al. (2009); Fabricating microgeometry for custom surface reflectance.

Matusik et al. (2009); Printing spatially-varying reflectance

Finckh et al. (2010); Geometry construction from caustic images Dong et al. (2010); Fabricating spatially-varying subsurface scattering. Papas et al (2011); Goal-based caustics.

Malzbender et al. (2012); Printing reflectance functions

Lan et al. (2013); Bi-Scale Appearance FabricationGeometric Optics8All recent attempts to fabricate BRDFs are based on geometric optics considerations and therefore they inherently suffer from the same resolution problems.

9Previous work: Wave scatteringWave models for BRDF: He et al. 91; Nayar et al. 91; Stam 99; Cuypers et al. 12

Holographye.g. Yaroslavsky 2004; Benton and Bove 2008No practical surface constructionSpecific illumination conditions (often coherent), not general BRDF9We build on previous work that derive BRDF models from wave optics principles. However these papers focus on rendering and they dont address practical construction issues such as resolution and machine restrictions. There is a lot of work on wave based scattering in the context of holography but holography relies heavily on coherent monochromatic illumination while we wish to view our surfaces under natural white illumination.

1010Contributions:Extra high resolution fabrication

Analyze wave effects under natural illumination

Analyze spatial-angular resolution tradeoffs

Practical surface design algorithm compatible with existing micro-fabrication technology

3cm0.1mmOur goal is to fabricate surfaces with spatially varying BRDF at a very high resolution, Instead of the 3 cm dots of previous approaches, we achieve 0.1 mm dots.For the first time the resolution that we produce is high enough to look smooth to the human eye.To achieve that we have to analyze wave optics effects under natural illumination, not only coherent lasers. We will also have to analyze the tradeoffs between the spatial and angular resolution we can achieve, and our BRDFs have high spatial resolution but their angular resolution is lower than previous approaches. Another central issue is to match our design to the capabilities of existing micro- fabrication technology.

1011Surface should be stepwise constant with a small number of different depth values

xzPrototype: Binary depth values Restricts achievable BRDFs

11Photolithography and its limitationsGeometric optics predicts: surface is a mirrorWave optics: variety of reflectance effects

11The main existing technology for generating micron sized structures is photo lithography. However this cannot generate continuous slanted surfaces. It is only able to generate piecewise constant surfaces. Also to simplify the construction the number of different depth values in the design should be kept as small as possible, since each depth requires a separate etching pass. In practice our prototype used only binary heights, but in the next slides we will demonstrate the more general case with multiple heights.

12Preview: reflectance = Fourier transform

ReflectanceDiffuseShinySurface micro-structure

AnisotropicWideNarrow

Narrow

WideBefore we go into the details I want to start by summarizing the main result.Which is that reflectance is the Fourier power spectrum of the surface height.Therefore there is an inverse relation between the size of surface features to the width of the reflectance lobe.Small features elements lead to a wide diffused reflectance, and wider features leads to narrow shiny reflectance. In the same way we can use rectangular features and get an anisotropic reflectance, narrow at one direction and wide at the other.

1213Background: understanding light scattering1. Coherent illumination: laser in physics lab

2. Incoherent illumination: natural world

13We need to start with a brief summary of wave optics principles. We will start by discussing coherent illumination, such as lasers, which is easier to analyze. In order to handle natural illumination, we will then have to understand how incoherence changes what we observe.

14Wave effects on light scatteringzx14Assume an incoming illumination from direction l, which hits the area of a certain dot on the surfaceThe incident energy scatters over the space. You can see here a plot of energy at different points. It usually have a wiggly structure due to wave interference.

15Surface scattering Fourier transform2

Fourier transformSee also:He et al. 91Stam 99zx15Luckily, it can be shown that when the light source and viewing point are sufficiently far there is a simple formula for the reflectanceWe take a signal whose phase is proportional to the surface height and the reflectance is its Fourier transform at a spatial frequency which corresponds to the x projection of the lighting and viewing directions.

16Inverse width relationship2Wide surfacefeaturesNarrow (shiny) reflectancex16The fact that the reflectance is the fourier transform implies that there is an inverse relation between feature size and reflectance size. For example here the surface contains relatively wide features and the energy is concentrated at a small range of angles, corresponding to a shiny material

17Inverse width relationship2Wide (diffuse) reflectancexNarrow surfacefeatures17If we use narrower features the energy is spread over a wider range of directions, corresponding to a diffuse material.

18Inverse width relationship2impulse (mirror) reflectancexFlat surface18If the surface is fully planar the reflectance is a narrow beam. A short calculation shows that this beam is obtained in the mirror direction, in agreement with the geometric optics model.

19Reflectance design with coherent illumination:Fourier power spectrum of surface height to produce reflectance

Challenges: Complex non-linear optimization May not have a solution with stepwise constant heightsInexact solutions: speckles

19Having understood that reflectance is the Fourier transform, designing a BRDF reduces to finding a surface height whose Fourier transform produces the desired reflectance. This is a complicated non-linear optimization problem. If we also want to constrain the height surface to be piecewise constant there may not be a feasible solution at all. A common artifact of an imperfect solution is speckles

20SpecklesNoisy reflectance from an inexact surfacex20By speckles I mean that the reflectance from most surfaces is noisy, these are not the common smooth BRDF functions we are used to see in computer graphics.

21Reflectance design with coherent illumination:Fourier power spectrum of surface height to produce reflectance

Challenges: Complex non-linear optimization May not have a solution with stepwise constant heightsInexact solutions: speckles

Our approach:Bypass problems utilizing natural illuminationPseudo random surface replaces optimizationNeed to model partial coherence

21All the problems we mentioned are characteristics of coherent illumination. However, coherent illumination you can only meet with lasers in a physics lab, while we wish to view our surfaces under natural illumination.Utilizing properties of natural illumination will help us bypass all these problems. And in fact, instead of solving a complex optimization problem we will be able to use a pseudo random surface.Natural illumination is considered incoherent but in such small scales it is not perfectly incoherent and some partial coherence remain. Analyzing the exact amount of partial coherence is central to our design.

22Incoherent illumination: Point source=> Area sourceArea source =collection of independent coherent point sources x22So the main difference between coherent and incoherent illumination is the size of the source. Coherent illumination is generated when light is emitted from an infitisimally small pinhole source. On the other hand, in the real world we meet area light sources, which can be viewed as a collection of independent coherent point sources.So for each surface dot we need to consider the subtended angle of directions

23Incoherent reflectance: blurring coherent reflectance by source angle*xAngular ConvolutionIllumination angleCoherent reflectance23Mathematically the incoherent reflectance is an angular convolution of the coherent intensity with a kernel representing the subtended angle of directions in the source.

24Reflectance averaged over illumination angleis smoothx24Incoherent reflectance: blurring coherent reflectance by source angle24and this blurring process can remove high frequencies and produce the smooth reflectances we are used to see.

25Challenge: avoiding specklesAngular v.s. spatial resolution tradeoffs.

Partial coherence.Our analysis: 25So, we need to understand when can we actually eliminate speckles? In the paper we analyse tradeoffs between the angular and spatial resolution we can hope to achieve. A main factor in the analysis is defining the amount of partial coherence. I will try to give a quick overview of these results, you can find the details in the paper.

26Angular resolution => Spatial coherence resolutionx26We denote the subtended angle of the illumination by delta_a. This is effectively the illumination resolution.It leads to a spatial coherent size we denote by delta_c. This is the size of a spatial unit over which the incoming illumination wave can be treated as coherent. We show in the paper that the coherent area is inversely proportional to the illumination angle.

27Angular resolution => spatial coherence resolutionxCoherent areaPhase changeCoherent:Incoherent:Partial coherent:27We denote the subtended angle of the illumination by delta_a. This is effectively the illumination resolution.It leads to a spatial coherent size we denote by delta_c. This is the spatial unit size over which the illumination wave is coherent. Although I havent described this some of you know that the illumination is actually a wave. One can show that when the illumination is coherent its intersection with a plane is a single frequency sinusoid. A fully incoherent illumination is a random wave.While a partially coherent wave contains coherent sinusoids over short intervals delta_c with random phase changes between them.

28Angular resolution -> spatial coherence resolutionxCoherent areaCoherent:Incoherent:Partial coherent:28The coherent area is inversely proportional to the illumination angle, for example reducing the angle increases the coherent area.

29Angular resolution => Spatial coherence resolutionxEach coherent region emits a coherent field with speckles29Over each coherent area we can use the rules of coherent illumination, and it scatters a coherent filed with speckles

30Angular resolution => Spatial coherence resolutionxEach coherent region emits a coherent field with speckles30

31Angular resolution => Spatial coherence resolutionxEach coherent region emits a coherent field with speckles31

32Angular resolution => Spatial coherence resolutionAveraging different noisy reflectances from multiple coherent regions => smooth reflectance. x32In practice we observe the averaged intensity of all these fields, And if we average enough of them the reflectance can be smooth.

33Angular resolution => Spatial coherence resolutionxDot sizeCoherent size33We can show that we can expect a smooth reflectance only if the coherent area is sufficiently smaller than the target spatial resolution, that is, smaller that the minimal area of a spatial dot unit we wish to resolve. That is, if we zoom over the area of our dot, we see that it includes multiple coherent segments..

34Angular resolution => Spatial coherence resolutionxCoherent sizeDot size34If we attempt to increase angular resolution, and reduce the illumination angle, the coherent size will increase. Now the dot area does average enough independent coherent regions and we are likely to see speckles.

35Angular resolution => Spatial coherence resolutionxDot sizeCoherent size Human eye resolution + typical angle of natural sources.=> Smooth reflectance (see paper) 35In the paper we make a calculation showing that given the minimal resolution the human eye can resolve and the typical angle of natural sources the sufficient resolution conditions hold and we get smooth reflectances.

36Recap:

Coherent BRDF = Fourier power spectrum of surface height.

Incoherent BRDF = Fourier power spectrum of surface height, blurred by illumination angle.

36Lets recap. We saw that under coherent illumination the reflectance is the Fourier transform.Under natural incoherent illumination, in which the source occupies a non-zero angle, we see a blurred version of the Fourier power spectrum.

Next: Design surface height to produce desired BRDF.

Coherent design: Fourier power spectrum to produce BRDF - Complex non linear optimization

Incoherent design: Blurred Fourier power spectrum to produce BRDF - Pseudo randomness is sufficient 37Next we show how the analysis helps us design a surface with a desired BRDF. Under coherent illumination finding a surface height which produces the exact Fourier transform is hard.In contrast, under natural illumination, only the blurred power spectrum needs to produce the desired reflectance, which means that multiple surfaces produce the same reflectance and we have much more freedom in the design.And in practice we can simply use pseudo random surface.

38Surface tiling algorithmxxzz38In order to tile the surface we just sample steps independently. We sample a step width from a distribution p_a And we select the step height z independently from a uniform distribution p_z.

39Surface tiling algorithmxCoherent illumination=> noisy reflectance39Under coherent illumination such a random surface will lead to a very noisy surface.

40Surface tiling algorithmx40Under incoherent illumination, when the sufficient resolution conditions apply we can get a smooth reflectance.This happens when the coherent size is sufficiently smaller than the dot size, so that each dot averages enough independent samples from the same model, so effectively we see the expected power spectrum of the stochastic process not a single noisy sample. In the paper we derive a formula which tells you how to choose the distribution p_a on step widths to achieve a desired output BRDF. Note that the step size distribution is the only thing we need to change to get different BRDFs.

Step size distribution41Surface samplingSampled surface micro-structure

ReflectanceDiffuseGlossy

ShinyHere are a few step size distributions, the surface we sampled and the resulting BRDF

4142BRDFs produced by our approachAnisotropicAnisotropic anti-mirrors IsotropicAnti-mirror

42So here are some of the reflectance functions we can produceFirst we have isotropic lobs at a variety of widths.And also various anisotropic lobsIn the paper we discuss another type of reflectance function which we call the anti- mirror. It reflects light at all directions except the mirror direction, you can see that it has a black hole in the center. It can therefore be used for applications such as camouflaging objects.A number of anisotropic extensions of this design generate a variety of reflectance functions, which could be used as reflective derivative or laplacian filters.

43Fabrication results

Electron microscope scanning of fabricated surface

20mmSo finally we get to some results.Here is our wafer and an electron microscope scanning of the micro structure we etched on it. We illuminated a point on our wafer with a laser beam and imaged the reflectances.And we repeat that for a few other regions.

43

44Imaging reflectance from fabricated surface

Specular spike, artifact of binary depth prototype, can be removed with more etching passes (see paper) 44Here are some of the reflectances, overall they match nicely with the design.In the second row you can see in the center a spot which is a specular spike in the reflectance. This spot is not part of the original design and it is an artifact of the fact that our simplified prototype used only binary depth. It can be avoided with more etching passes.

Imaging under white illumination at varying directionswafercameraMoving light Next, we imaged the pattern under varying illumination directions, using a white light source.and lets see some interesting reflectance effects we can generate.

45

Vertical illuminationHorizontal illuminationNegative image

Anisotropic BRDFs at opposite orientationsWe start by illuminating the pattern from the vertical direction, and here is what we see. Now lets switch horizontal illumination. We see the negative image, the face is dark and the hair is bright. To see how we achieve this, here is the map of one of our pattern.Each dot in the red area was fabricated with an anisotropic BRDF with one orientation, and in the green regions the opposite anisotropic orientation. Therefore these areas are bright under one orientation of the illumination direction, and dark from the other one.

46

VerticalHorizontalNegative imageHere is another example with the siggraph logo. Under vertical illumination the text is bright and the pattern is dark, and under horizontal illumination we see the inverse effect.

47

Narrow IsotropicAnti-mirror

large incident angle:Anti-mirror kids: brightBackground: darkSmall incident angle:Anti-mirror kids: darkBackground: bright

In this pattern the background is made with a narrow isotropic BRDF and the kids area with anti-mirror BRDF.When illuminated from a large incident angle the background is dark and the kids are bright. As we reduce the illumination angle things change and from a small angle the shiny background is bright, but the anti-mirror kids are dark.

4849Limitations49Before warping up I would like to point out some limitations.First our approach allows control over reflectance but not over albedo or color. Another issue is that our binary height prototype restricts the BRDFs we could achieve. First we cannot control the specular spike and second our BRDFs must be symmetric. In the paper we show that these problems can be removed if we use as little as 4 different depth values, but this requires more etching passes. Finally the spatial resolution of our BRDFs is high, but the angular resolution is more limited than in previous approaches.

50Limitations50Before warping up I would like to point out some limitations.First our approach allows control over reflectance but not over albedo or color. Another issue is that with our simple binary depth prototype we couldnt control the specular spike. In the paper we show that this spike can be removed if we use as little as 4 different depth values, but this requires more etching passes. Finally the spatial resolution of our BRDFs is high, but the angular resolution is more limited than in previous approaches.

51SummarySpatially varying BRDF at high spatial resolution (220 dpi).

Analyze wave effects under natural illumination.

Account for photolithography limitations.

Pseudo randomness replaces sophisticated surface design.

51So to summarize we shown how to Fabricate spatially varying BRDF with very high spatial resolution, we can control the appearance of 220 dots per inch.To achieve this we had to analyze wave effects and resolution tradeoffs under natural illumination, not only coherent lasers.

We also had to adjust the design to the capabilities of existing photolithography technology.In practice the BRDF construction algorithm is very simple. A pseudo random surface can bypass the need for sophisticated optimization in the design.

Thank you!52

20mmWafer available after session