3d vision the university of oxford andrew glennerster physiology andrew fitzgibbon engineering

3D Vision

The University of Oxford

Andrew Glennerster Physiology

Andrew Fitzgibbon Engineering

Vision

A fundamental biological system

An attractive engineering problem

How are these related?

A 3D Visual Task

TOC

• Computer vision state of the art• Biology state of the art

– an implausible model?

• Open questions

Artificial 3D Vision

Measurements

3D Structure

Application: 3D Video Editing

Application: 3D Video Editing

“Computer Vision” Summary

• Single representation suits all– pointing– navigation

• Built on– algebra– geometry– statistics

• Really works on real data

Biological systems can perform 3D tasksusing visual information

• Pointing to remembered objects• Ants homing• Spiders turning to view prey

Computer vision is an unlikely model for animal vision

• A “likely” model needs topredict experimental databe neurally plausible

• We don’t know how to multiply matrices in neurons

• CV is open loop– animate systems characterized by feedback

• 3D not necessary for many tasks

What is the incumbent model?

Posterior parietalcortex

Eye-centredcoordinates

Head-centredcoordinates

Body-centredcoordinates

World-centredcoordinates

Eyeposition

Audition

Neck proprioception

Vestibularinformation

Visualinformation

Adapted from figure 3 from Andersen et al (1997)

An Alternative

• In simpler animals, the requirement for a map is questioned.

• Many tasks can be performed by local, low-level, reactive rules

(Franz and Mallot, 2000)

“Biomimetic” Navigation

(Franz and Mallot, 2000)

“Biomimetic” Navigation

“Biomimetic” Navigation

• Neurally plausible

• Real robot implementations

• Cannot model 3D pointing yet

• No complete model of ant, for example

“computer vision hitting a computational wall”

• Need infinite-time, infinite-space operation

• Computers are getting faster...

• When will O(2t) beat O(n3)?– n is number of features– t is years from now

Dynamic Scenes

Open Problems

• No complete model of ant navigation– Photons to pointing

• No plausible model of 3D vision in higher animals– Coordinate system: Is there one?– One representation or many?

• Computer vision’s computational wall– One solution will be better engineering– Another may be “parallel slow neurons”

• Dynamic scenes

Thanks To

Andrew GlennersterAndrew ParkerAndrew Blake

Andrew ZissermanAndrew DavisonAndrew Stoddart

...and a few non-andrews

Questions

• How do biological systems perform such tasks?– We want an explanation “from

photons to pointing”

• How might artificial systems perform such tasks?– Artificial systems are “easy” to analyse

Status

• We have models for some of the biology

• We have artificial systems which work in useful but limited domains