a new rank constraint on multi -view fundamental matrices, ieee...

IEEE 2017 Conference on Computer Vision and Pattern

Recognition A New Rank Constraint on Multi-view Fundamental Matrices,

and its Application to Camera Location RecoveryS. Sengupta, T. Amir, M. Galun, A. Singer, T. Goldstein, D. Jacobs, R. Basri

Our Contribution

Ø A novel rank constraint on Fundamental matrices in multi-view settings

Ø Constrained low rank optimization to de-noise and recover missing fundamental matrices

Ø Improvement over state-of-the-art in structure from motion

Our Result

2

64

0 F̂12 F̂13 � �F̂21 0 F̂23 F̂24 �F̂31 F̂32 0 F̂34 F̂35

� F̂42 F̂43 0 F̂45

� � F̂53 F̂54 0

3

75

| {z }

⇡

2

40 �12 �13 �14 �15

�21 0 �23 �24 �25

�31 �32 0 �34 �35

�41 �42 �43 0 �45

�51 �52 �53 �54 0

3

5

| {z }

�

2

40 F12 F13 F14 F15

F21 0 F23 F24 F25

F31 F32 0 F34 F35

F41 F42 F43 0 F45

F51 F52 F53 F54 0

3

5

| {z }F̂ ⇤ F

with F = A+AT and rank(A) = 3.

F̂13

F̂23F̂24

F̂34

F̂35

F̂45

For n not all collinear cameras, the multi-view fundamental matrix F of size 3𝑛×3𝑛 satisfies :

Fii = 0,

F = A+AT ,

rank(A) = 3,

rank(F ) = 6.

Number of Images50 100 150

Rel

ativ

e Im

prov

emen

t (in

%)

0

5

10

15

20Median Essential Matrix Error

Our vs LUDOur vs ShapeKick

Number of Images50 100 150

Per

cent

of I

mpr

oved

Tria

ls

50

60

70

80

90

100Median Essential Matrix Error

Our vs LUDOur vs ShapeKick

Experimental ResultsØ 14 scenes from Photo-tourism datasetØ For each scene, 5 different sub-samples of 50, 100 and 150 imagesØ Measure error on Essential matrix and camera location recoveryØ Performance Measures:

o Relative Improvement (in %).o Number of Improved Trials.

Essential Matrix : Improves over LUD :

- by 18% on average- In 86% of all trials

Number of Images50 100 150

Rel

ativ

e Im

prov

emen

t (in

%)

0

10

20

30

40

50

60Median Translation Error

Our vs LUDOur vs ShapeKickOur vs 1DSfM

Number of Images50 100 150

Per

cent

of I

mpr

oved

Tria

ls

50

60

70

80

90

100Median Translation Error

Our vs LUDOur vs ShapeKickOur vs 1DSfM

Camera Location : Improves over LUD :

- by 20% on average- In 84% of all trials

References1.“Robust Global Translation with 1DSfM” K. Wilson et.al. ECCV 2014.(1DSfM)2.“ Robust camera location estimation by convex programming” O. Ozyesil et.al. CVPR 2015. (LUD)3. “ Shapefit and shapekick for robust, scalable structure from motion” T. Goldstein et.al. ECCV 2016. (ShapeKick)

Rank constrained Optimization

Robust L1 cost function

Solve using IRLS - ADMM

ConstructionFor simplicity, we show our construction for Essential matrices

𝑇& = 𝑡& ×: Camera location, 𝑅&: Camera orientation

Equality is proved in the paper

Result applies also to fundamental matrices since 𝐹&+ = 𝐾-.𝐸&+𝐾-0

median T error mean T error median E error mean E error

Impro

vem

ent over

LU

D (

%)

0

5

10

15

20

Our - fundamental recoveryOur - essential recovery

Our method can also optimize directly over Fundamental matrices by jointly optimizing camera rotation, location and calibration.

Code

www.umiacs.umd.edu/~sengupta/SfM_CVPR2017_code.zip

Our Rank Constraint