what the spider’s eyes don’t tell the spider’s...

What the spider’s eyes don’t tell the spider’s brain

Depth Perception from Image Defocus in a Jumping Spider

(*) “Depth Perception from Image Defocus in a Jumping Spider” Nagata, Koyanagi, Tsukamoto, Saeki, Isono, Shichida, Tolunaga, Kinoshita, Arikawa, and Terakita.

How to judge distance to prey?

• World is 3-D — images are 2-D

Can’t determine distance monocularly

Scale factor ambiguity





• Many “depth cues”

Ratio image size to object size

Ratio image motion to object motion

...








...

• Lens accomodation








...


• Binocular stereo








...


• Binocular stereo

• Defocus blur

Accomodation? (1/f = 1/a+ 1/b)

Binocular stereo?

(*) “Jumping Spider Vision”, David Hill, Wikipedia

Defocus blurring?

Defocus blurring?

PSF

R

Defocus blurring? 2J1(Rρ)/(Rρ)

PSF

R

MTF

3.8317 / R

Multi-layer retina


Depth from two image planes


What’s wrong with that model?

What is wrong with that model?

• Assumes image on back layer (L1) is always in focus

But this would require accomodation;

then there is no need for anything else!





• Assumes blur on front layer (L2) depends on distance

If back is in focus then the blur in front is fixed;

blur in front merely reflects inter image layer spacing!





• Assumes blur on front layer (L2) depends on distance

If back is in focus then the blur in front is fixed;

blur in front merely reflects inter image layer spacing!

• Assumes amount of blur can be ascertained from image

Problem is ill posed; for example:

Blurry image of sharp texture same assharp image of blurry texture!

Some possible approaches

• Transport of Intensity Equation (TIE)

∇xy ·(I(x, y, z)

∇xyφ(x, y, z)

k

)= −∂I(x, y, z)

∂z



∇xy ·(I(x, y, z)

∇xyφ(x, y, z)

k

)= −∂I(x, y, z)

∂z

• Light-field propagation



∇xy ·(I(x, y, z)

∇xyφ(x, y, z)

k

)= −∂I(x, y, z)

∂z

• Light-field propagation

• Deconvolution

• . . .

System model

E(x,y)

⊗ b1(x,y)

⊗ b2(x,y)

E1(x,y)

E2(x,y)

Solution based on this

E(x,y)

⊗ b1(x,y)

⊗ b2(x,y)

E1(x,y)

E2(x,y)

⊗ b2(x,y)

⊗ b1(x,y)

−

Solution based on this

E(x,y)

⊗ b1(x,y)

⊗ b2(x,y)

E1(x,y)

E2(x,y)

⊗ b2(x,y)

⊗ b1(x,y)

− b1 ⊗ b2 = b2 ⊗ b1

Doing it in parallelE(x,y)

⊗ b(z) ⊗ b(z+d)

E1(x,y) E2(x,y)

⊗ b(1+d) ⊗ b(1)−

mag

⊗ b(2+d) ⊗ b(2)−

mag

⊗ b(3+d) ⊗ b(3)−

mag

⊗ b(4+d) ⊗ b(4)−

mag

argmin

0.00 mm

0.15 mm

0.30 mm

0.45 mm

0.60 mm

0.75 mm

0.90 mm

1.05 mm

1.20 mm

Recovery of in-focus distance

0 1 2 3 4 5 6

0

1

2

3

4

Recovering the “in focus” image

• Ill-posed problem from single defocused image:

P1(u) = P(u)M1(u)

• can’t recover frequency components where M1(u) = 0.

Recovering the “in focus” image

• Ill-posed problem from single defocused image:

P1(u) = P(u)M1(u)

• can’t recover frequency components where M1(u) = 0.

• But with two images — defocused to different degrees:

P2(u) = P(u)M2(u)

P1(u)M∗1 (u)+ P2(u)M∗

2 (u) = P(u)(‖M1(u)‖2 + ‖M2(u)‖2

)• works as long as, for any u, either M1(u) �= 0 or M2(u) �= 0.

• (actually, use Wiener filtering)

What the spider’s eyes don’t tell the spider’s brain

Pillbox convolved with pillbox is not a pillbox

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0

Calibration of lens motion

60

61

62

63

0 10 20

f (mm)

steps60

61

62

63

0 10 20

f (mm)

steps

Lens motion from estimates of zeros in DFT

1208 1209 1210 1211 1212 1213 1214 1215 12160.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

1.2

1.3

mm

frame

what the spider’s eyes don’t tell the spider’s...

Documents