low-latency event-based visual odometry · ‣dvs: interesting new sensor for agile robots! ‣...

1

Andrea Censi 1

Low-Latency Event-Based Visual OdometryDavide Scaramuzza 2

Acknowledgments: Jonas Strubel, Christian Brandli, Tobi Delbruck, Matia Pizzoli, Christian Forster.

1 Laboratory for Information and Decision Systems Massachusetts Institute of Technology

2 Department of Informatics University of Zurich

slides available at censi.mit.edu/slides

http://censi.mit.edu/slides

2

Low-Latency Event-Based Visual Odometry

3

recovering camera motion

Page 4: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

4

brightness increased

brightness decreased

using a “neuromorphic” sensor

Page 5: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

4

brightness increased

brightness decreased

using a “neuromorphic” sensor

Page 6: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

5

‣ What is a “neuromorphic” sensor?!

‣ Why would you want to use in robotics?

Come to the “Non-traditional Cameras” workshop on Thursday for technical details!

Page 7: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

6

Page 8: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

7

Slide by Christian Brandli

Page 9: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ Traditional camera: frames at fixed intervals.

8

timeframe

Page 10: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ Traditional camera: frames at fixed intervals.

8

timeframe next frame

Page 11: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

‣ “Neuromorphic” sensor: events generated any time a single pixel sees a change in brightness.

brightness

time

Page 12: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 13: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 14: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 15: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 16: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 17: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 18: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 19: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 20: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 21: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 22: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 23: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

9

brightness

time

Page 24: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

10

time

events stream

Page 25: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

10

time

events stream

k-th event:

Page 26: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

10

time

events stream

timestamp (1 µs resolution)

k-th event:

Page 27: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

10

time

events stream

pixel coordinates

k-th event:

Page 28: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

10

time

events stream

pixel coordinates

sign (+1 or -1)

k-th event:

Page 29: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ “Neuromorphic” sensors:!- Event-based !- Asynchronous and distributed

per-pixel processing!- Analog computation

11

Liu, Delbruck. Neuromorphic Sensory Systems. 2010 DOI:10.1016/j.conb.2010.03.00

http://dx.doi.org/10.1016/j.conb.2010.03.00

Page 30: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ “Neuromorphic” sensors:!- Event-based !- Asynchronous and distributed

per-pixel processing!- Analog computation

11

Liu, Delbruck. Neuromorphic Sensory Systems. 2010 DOI:10.1016/j.conb.2010.03.00

http://dx.doi.org/10.1016/j.conb.2010.03.00

Page 31: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ “Dynamic Vision Sensor” (DVS): a neuromorphic sensor developed by Delbruck’s group (ETH).!- Commercially available ($2.5k).

12

Lichtsteiner, Posch, Delbruck. A 128x128 120 dB 15µs Latency Asynchronous Temporal Contrast Vision Sensor. 2008 DOI:10.1109/JSSC.2007.914337

http://dx.doi.org/10.1109/JSSC.2007.914337

Page 32: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

13

‣ Earliest “silicon retina” developed by Carver Mead at Caltech.

Mead. Neuromorphic electronic systems. IEEE Proceedings, 1990!

Page 33: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ The DVS is fast!

14

Page 34: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

14

Mueggler et al., IROS 2014

Page 35: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

14

CMOS

Page 36: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

14

CMOS DVS

Page 37: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

14

CMOS DVS

‣ DVS visualization: events histogram in a time slice.

Page 38: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

15

‣ The DVS is really fast!

Page 39: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

15

Page 40: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

16

1 video frame = 33 ms

1 video frame = 0.05 ms

Page 41: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

16

1 video frame = 0.05 ms

Page 42: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

17

“fast” sensorslow-latency control loop

commands

‣ Goal: low-latency control architecture.

17

commands

“slow” sensors

Page 44: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

17

commands

“slow”, “cognitive” level

“context”

“slow” sensors

Page 45: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

18

DVS

CMOS camera

Page 46: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

19

low-latency control loop

“context”

DVS

CMOS camera

Page 47: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

19

low-latency control loop

“context”

DVS

CMOS camera

20

Event-Based Visual Odometry

‣ Goal: estimate the camera motion from the events.

‣ Large solution space!- use additional sensors? !

this work: yes, use additional CMOS sensor!- use “features”? !

this work: no, use raw events

20

Event-Based Visual Odometry

‣ Goal: estimate the camera motion from the events.

21

(calibration details in the Thursday workshop)

DVS events timeCMOS frames

Page 51: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

21

Page 52: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

21

Page 53: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

22

Page 54: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

22

- one event doesn’t tell much!- low resolution (128 × 128)!

Page 55: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

22

- one event doesn’t tell much!- low resolution (128 × 128)!

- you see incremental motion!- you can decouple the motion along the axes

Page 56: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

23

Page 57: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

23

‣ Idea: reduce the problem to “localization” with respect to the previous CMOS frame.

Page 58: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

The 1D case — pure rotation

time

pixel

Page 59: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

Page 60: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

Page 61: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

Page 62: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 63: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 64: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 65: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 66: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 67: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 68: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 69: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 70: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

24

time

pixel

estimated velocity

Page 71: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

25

estimated velocity

Page 72: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

25

estimated velocity

Page 73: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

‣ Very good accuracy for rotation !drift comparable to visual odometry

26

Page 74: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

26

1.0°-8.9°

2.1°

90°

odometry estimate

truth0°

Page 75: Low-Latency Event-Based Visual Odometry · ‣DVS: Interesting new sensor for agile robots! ‣ Goal: low-latency control architectures! ‣ So far: worked on inference problems!-blinking

26

‣ Poor tracking of translation!very low SNR due to small apparent motion!(also: needs to know the image’s depth map)

1.0°-8.9°

2.1°

90°

odometry estimate

truth0°

‣ DVS: Interesting new sensor for agile robots!

‣ Goal: low-latency control architectures!

‣ So far: worked on inference problems!- blinking LEDs tracking (IROS 2013)!- visual odometry w/CMOS fusion (this work)!- feature-based visual odometry (Mueggler et al., IROS 2014)

27

Summary

‣ DVS: Interesting new sensor for agile robots!

‣ Goal: low-latency control architectures!

‣ So far: worked on inference problems!- blinking LEDs tracking (IROS 2013)!- visual odometry w/CMOS fusion (this work)!- feature-based visual odometry (Mueggler et al., IROS 2014)

27

Summary

low-latency event-based visual odometry · ‣dvs: interesting new sensor for agile robots! ‣...

Documents