Diffuse Reflection Imaging:Earthshine and other Faint Signals

Sam HasinoffMIT CSAIL, TTIC, Google[x]

Samuel W. Hasinoff, Anat Levin, Philip R. Goode, and William T. Freeman, Diffuse Reflectance Imaging with Astronomical Applications,Proc. 13th IEEE International Conference on Computer Vision, ICCV 2011http://people.csail.mit.edu/hasinoff/diffuse/

2

Bill FreemanMIT CSAIL

Joint work with:

Anat Levin Weizmann Institute

Philip GoodeBig Bear Solar Observatory

Special thanks to: Bernhard Schölkopf, Frédo Durand, Livia Illie, David Chen

3

Indirect Imaging with Reflective Objects

specular reflectance

virtual camera

glossyreflectance

bad camera?

diffuse reflectance

really bad camera?

-180° 0° +180°0

1

4

Resolution of a Diffuse Reflector

Diffuse surfaces blur together incident lighting from many directions

How much detail can we resolve?

Lambertian reflectance

5

Single-Bounce Light Transport

reflectedradiance

surfacealbedo

incomingradiance

BRDF clipped cosinevisibility

reflectance = (albedo lighting) BRDF for convex objects, distant illumination*.

-180° 0° +180°0

1

6

Linear Light Transport

Usual matrix formulation

observedpixels

transfermatrix

distantlighting

noise

transfer matrix encodes geometry, BRDF, surface albedo

7

Resolution of a Diffuse Reflector

about 9 pixels?images of a convex Lambertian objects

with distant lighting lie in a 9D subspace (<2% average error)

[Basri and Jacobs, 2001][Ramamoorthi and Hanrahan, 2001]

e.g. lighting-insensitive recognition with 9 basis images

…

8

Key: Occlusion Geometry Codes Illumination

convexabout 9 pixels?

self-occlusionpotentially much better

9

Key: Occlusion Geometry Codes Illumination

convexabout 9 pixels?

self-occlusionpotentially much better

What 3D shape lets us best resolve the lighting?

10

Sculpture Design for Reflectance Imaging

double-pinholes coded reflectance natural complexity

• self-occlusion preserves high frequencies• extreme: isolate rays by building pinhole cameras

using the scene• arbitrary fidelity (in theory), limited only by diffraction

and geometric calibration

Standard MAP estimation• Gaussian prior [Wiener 1949]

• Sparse derivative prior [Levin et al. 2007]

11

Bayesian Reconstruction Method

For time-varying lighting, reconstruct stable part• marginalize out temporal variations

12

Reconstructing a Changing Target

temporal binning withGaussian prior

temporal variations as correlated noise

Approach #1: Rim Reflectance Imaging13

Exploit visibility changes near occlusion boundary• works for compact lighting, e.g.

object lit head on• multiple orientations = tomography

+ single shot enough- need high resolution- need accurate geometry

not previously used in astronomyor computer vision

Approach #2: Time Varying Imaging14

Exploit variation over time in the occlusion geometry• classical astronomy (“light curves”)

+ single pixel enough- need natural variation- need many shots- target might change

e.g. surface of Pluto from Charon’s 1985-1990 transits [Young et al. 1999]

Earth from Space from Earth - Feasibility Study

Earth from Apollo 17, 1972 (NASA)

15

Low-Resolution Expectations16

NASA’s “Blue Planet”

reconstruction we’d be happy with

Moon as Earth Reflector (“Earthshine”)

Leonardo Da Vinci’s Codex Leicester (ca. 1510)

17

18

sunlight

Earth north pole

(Exaggerated) Geometry of Earthshine

Moon north pole Moon from observer

observer

19

sunlight

Earth north pole

Moon north pole Moon from observer

observer

Earthshine

Moonshine

(Exaggerated) Geometry of Earthshine

Integrating over Earth from the Moon20

Sun and

Moon tovisible EarthEarth ),,Sun(BRDF)ˆˆ()ˆunS( dRIRIRRII aaR

albedo of “unknown” Earth patch

Earth phase (≈ 180° - Moon phase)

cos(incoming)cos(outgoing)

Aug 8, 2010, 5am Feb 24, 2010, 1am

Earth integrand, from point on the Moon (simulation)

Earthshine as Linear Transfer Matrix21

y = T x + n∙observations of

earthshine Earthimage

Earth-Moontransfer matrix

noise

22

Recipe for Earthshine Imaging

feasible observing times: • moon >2° above horizon (air

mass)• moon <85% full

(moonshine)• sun >8° below horizon

(twilight)• clear skies

track moon - multi-minute exposures> -2.5 app. mag. (V)

block moonshine glare with occluder

BBSO [Qiu et al. 2003]

Approach #1: Visibility at Moon Rim

Earthrise, as seen from Apollo 8 (NASA)

23

• From rim of Moon, different Earth regions visible

• Special challenges:• need very high resolution, SNR• need detailed moon topography• BRDF at grazing angles?

24

Best Case Earth-Bound Moon Imaging

8m telescopeESO Very Large Telescope (Chile)0.07” resolution26,000 pixels across lunar disk

by comparison, HST is 0.05” (i.e. near diffraction limit for it 2.4m mirror)

http://optics.org/article/9654/

25

ground truth earth model (75x30 pixels)

relative contribution,Lambertian Earth

moon simulation, 2475 samples (55 per radial line)

Sampling the Earth and Moon Surface

26

Simulated Reconstruction for Single Shot sp

arse

prio

r rec

onst

ructi

on

std. optics (0.4”) adaptive optics (0.07”)

80 d

B

60

dB

40 d

B

Approach #2: Scanning over Time27

• Over time, Earth region visible to Moon & Sun changes• Single pixel observations is fine

• Special challenges:• 1D set of viewpoints per day• time varying cloud cover, snow, etc.

region of Earth visible from Moon’s disk, i.e. contributing to earthshine

1am, Feb 24, 2010 5am, Feb 24, 2010 2am, Aug 14, 2010

28

Simulated Reconstruction over Time

1 nightAug 12-13, 2010

107 obs. (@3 min)

1 monthAug, 2010

412 obs. (@10 min)

1 yearJan-Dec, 2010

963 obs. (@60 min)

reco

n.co

ntrib

.gr

d. tr

uth

29

Simulated Reconstruction over Time

1 nightAug 12-13, 2010

107 obs. (@3 min)

1 monthAug, 2010

412 obs. (@10 min)

1 yearJan-Dec, 2010

963 obs. (@60 min)

reco

n.co

ntrib

.gr

d. tr

uth

30

Time-Varying Clouds

• 50-75% of Earth covered by clouds• large-scale systems last about 3 days

const. mean cloudreconstruction

earth with ISCCP clouds next day clouds

simulated reconstruction (1 year @3min, 60 dB)

mean clouds

naïve time-varyingreconstruction

covariantreconstruction

31

Reconstructing Mars from Light Curves

• direct observation• outer planet, so less phase change (>85% disk lit)• much less active weather than earth• axial tilt of 25° (2D directions)• more Lambertian?

Martian surface

32

Martian Reconstruction from Historical Spectra

ground truth + visibility relative contribution

234 spectra (single-pixel images) observedat 2 sites, from 1963-1965 [Irivine et al. 1968ab]

recon. from simulation (scaled) recon. from real data (hVB), 47 dB

33

Simulation: Limits of Single-Pixel Mars Reconstruction

200 samples 400 samples 800 samples

1600 samples 3200 samples 23731 samples

1 Martian year (= 2 years)single-pixel images, 60 dB, no temporal variation

10 Years of Moon Photos34

earthshine

moonshine filter ∙

Ia

Ib

• 10 years• 40k+ photos• photos every 5 min• Big Bear Solar Observatory (BBSO), near LA• single CCD (grayscale)

Moon from BBSO [Qiu et al. 2003]

The Earthshine-Moonshine Ratio35

earthshine

moonshine filter ∙

Ia

Ib

)obs,,Sun(BRDF)obs,,Earth(BRDF

Moon

Moon

Sun

Earth

b

a

b

a

II

II

II

irradiance from “unknown” Earth < 1°

lunar phase 0° = full, 180° = new

Earth

atmosphere

eII 0

moonshine reference: factors out viewing effects

36

• imaging tradeoffs, e.g. blur vs. noise• sensor characteristics

2.8 dB

14.6 dB

Coda: Optimizing Capture Strategies

vs.

http://people.csail.mit.edu/hasinoff/