illumination independent marker tracking using cross-ratio invariance

Illumination Independent Marker Tracking using

Cross-Ratio Invariance

Vincent Agnus Stéphane Nicolau Luc Soler

Institut de Recherche Contre les Cancers de l'Appareil Digestif

ACM Symposium on Virtual Reality Software and Technology(VRST2014)

November 11, 2014

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 1 / 18

Outline

Causes & E�ects of light condition on corner detection

How cross-ratio can be used to correct corner shifting

Method & validation

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 2 / 18

Motivation

Corner Detection : Illumination ConditionsExample

Light conditions in�uence the estimation of corner position

Outcomes : Errors

Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

Motivation

Corner Detection : Illumination ConditionsExample

Light conditions in�uence the estimation of corner position

Outcomes : Errors

Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

Motivation

Corner Detection : Illumination ConditionsExample

Light conditions in�uence the estimation of corner position

Outcomes : Errors

Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

Motivation

Corner Detection : Illumination ConditionsExample

Light conditions in�uence the estimation of corner position

Outcomes : Errors

Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

Motivation

Corner Detection : Illumination ConditionsExample

Light conditions in�uence the estimation of corner position

Outcomes : Errors

Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

Motivation

Medical applications using markers

A low cost and accurate guidance system for

laparoscopic surgery : Validation on an

abdominal phantom [Nicolau VRST2005]

A low cost simulator to practice ultrasound

image interpretation and probe manipulation :

Design and �rst evaluation [Nicolau ISVRI2011]

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18

Motivation

Medical applications using markers

A low cost and accurate guidance system for

laparoscopic surgery : Validation on an

abdominal phantom [Nicolau VRST2005]

A low cost simulator to practice ultrasound

image interpretation and probe manipulation :

Design and �rst evaluation [Nicolau ISVRI2011]

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18

Error on tag position

⇒ Larger position errors for :

Needle tip

UltraSound virtual plane

Motivation

Corner DetectionShift under light conditions

Shifting Causes : Mitsunaga and Nayar 1999

non-linear mapping between scene radiance and measured brightness

measured brightness rasterisation (spatial and intensity) on CDD

White

Black

Scene Radiance

state of the art

Studies : marker detection , shifting ⇒ Biased images

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

Motivation

Corner DetectionShift under light conditions

Shifting Causes : Mitsunaga and Nayar 1999

non-linear mapping between scene radiance and measured brightness

measured brightness rasterisation (spatial and intensity) on CDD

White

Black

Scene Radiance Measured Brightness

RadiometricResponse

state of the art

Studies : marker detection , shifting ⇒ Biased images

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

Motivation

Corner DetectionShift under light conditions

Shifting Causes : Mitsunaga and Nayar 1999

non-linear mapping between scene radiance and measured brightness

measured brightness rasterisation (spatial and intensity) on CDD

White

Black

Scene Radiance Measured Brightness

RadiometricResponse

state of the art

Studies : marker detection , shifting ⇒ Biased images

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

Motivation

Corner DetectionShift under light conditions

Shifting Causes : Mitsunaga and Nayar 1999

non-linear mapping between scene radiance and measured brightness

measured brightness rasterisation (spatial and intensity) on CDD

White

Black

Scene Radiance Measured Brightness

Rasterisation Spatial & IntensityRadiometric

Response

state of the art

Studies : marker detection , shifting ⇒ Biased images

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

Proposed Method

Key Idea

Light Condition

underexposure : black region ↗, white region ↘overexposure : black region ↘, white region ↗

⇒ error on estimated position of the marker

How we can detect and correct this phenomena ?

Our method

use the marker geometry : inner & outer corners

check that white/black region proportion are preserved

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 6 / 18

Proposed Method

Corner Detection ShiftDetection & Correction

Cross Ratio (CR)

A projective invariant feature de�nedfor collinear points sets

A B C D

A′ B′ C ′ D′

CR =(AB/AC)

(DB/DC)=

(A′B′/A′C ′)

(D′B′/D′C ′)

Cross Ratio on markers

ABC

D

A BC D

CR should be constant :position & light

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18

Proposed Method

Corner Detection ShiftDetection & Correction

Cross Ratio (CR)

A projective invariant feature de�nedfor collinear points sets

A B C D

A′ B′ C ′ D′

CR =(AB/AC)

(DB/DC)=

(A′B′/A′C ′)

(D′B′/D′C ′)

Cross Ratio on markers

ABC

D

A BC D

CR should be constant :position & light

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18

Proposed Method

Theoretical Cross Ratio vs Measured Cross Ratio

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

Proposed Method

Theoretical Cross Ratio vs Measured Cross Ratio

CRmodel

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

Proposed Method

Theoretical Cross Ratio vs Measured Cross Ratio

CRmodel

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

Proposed Method

Theoretical Cross Ratio vs Measured Cross Ratio

CRmodel

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

Proposed Method

Cross Ratio : selected points

Cross Ratio : Our Study

A BC D

4+4+2 lines for CR

Cross Ratio : inner corners of themarker (future)

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18

Proposed Method

Cross Ratio : selected points

Cross Ratio : Our Study

A BC D

4+4+2 lines for CR

Cross Ratio : inner corners of themarker (future)

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18

Proposed Method

Cross Ratio : selected points

Cross Ratio : Our Study

A BC D

4+4+2 lines for CR

Cross Ratio : inner corners of themarker (future)

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18

Proposed Method

Correction of corner shifts

How cross ratio invariance can be used to �x corners shifting ?

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 10 / 18

Proposed Method

Correction of corner shifts

Idea

Displace the corners such that CR(displaced) = CRmodel

Method

H

(+dx,+dy)

(+dx,−dy)

p3

p1 p5 p6p2

p7p14

p15

p4p9

p12 p11 p16

p8p10p13

p′1

H(p1)

p′i = H(pi) + (±dx,±dy)

(dx, dy) = argmindx,dy

∑(a,b,c,d)∈I

(CR(p′a, p′b, p′c, p′d)− CRmodel)

2

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18

Proposed Method

Correction of corner shifts

Idea

Displace the corners such that CR(displaced) = CRmodel

Method

H

(+dx,+dy)

(+dx,−dy)

p3

p1 p5 p6p2

p7p14

p15

p4p9

p12 p11 p16

p8p10p13

p′1

H(p1)

p′i = H(pi) + (±dx,±dy)

(dx, dy) = argmindx,dy

∑(a,b,c,d)∈I

(CR(p′a, p′b, p′c, p′d)− CRmodel)

2

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18

Proposed Method

Correction of corner shifts

New Corner Positions

H(+dx,+dy)

(+dx,−dy)

p3

p1 p5 p6p2

p7p14

p15

p4p9

p12 p11p16

p8p10p13

p′1H(p1)

correct(pi) = H−1(H(pi) + (±dx,±dy)︸ ︷︷ ︸

p′i

)

corner in original image corrected cornerVincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 12 / 18

Proposed Method

Optimization Process

Minimization

argmindx,dy

∑(a,b,c,d)∈I

(CR(p′a, p′b, p′c, p′d)− CRmodel)

2

Levenberg-Marquardt algorithm

not optimal : jacobian not provided, estimated from aforward-di�erence approximation

5 ms for each image : SciPy , (Core i7@2.2GHz), 10 lines for CR

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 13 / 18

Validation

ValidationProtocol

Metric

Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

Validation

ValidationProtocol

Metric

Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

Validation

ValidationProtocol

Metric

Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)

Pi

O

Πcheck

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

Validation

ValidationProtocol

Metric

Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)

Pi

P proji

O

Πcheck

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

Validation

ValidationExperiments

Chessboarddistance

cross ratio

Before correction

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

Validation

ValidationExperiments

cross ratio

After correction

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

Validation

ValidationExperiments

After correction

distance

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

Validation

ValidationExperiments

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

Logitech c© HD ProWebcam C9101920×1080

Microsoft c© Lifecam 1280×720 Logitech c© Procam 640×480

Validation

ValidationExperiments

3 cameras × 5 orientations (range 40-60cm) × 8 light conditions3 groups : illumination condition.over-exposure generate more errors.

without correction with correction

Under-exposure 1.15 ± 0.22 mm 0.42 ± 0.19 mmNormal exposure 0.90 ± 0.32 mm 0.67 ± 0.22 mmOver-exposure 2.60 ± 0.70 mm 0.71 ± 0.25 mm

Table: average errors of the marker depth estimation

Cross Ratio :

Can detect extreme light condition

Can correct depth estimation (<1 mm)

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 16 / 18

Discussion

Discussion & Perspectives

Discussion

estimated homography H → corner correction

assumption detected corners are collinear

assumption of isotropic in�uence of light after homographicrecti�cation

Future Works

use inner corners of markers

minimization : analytical formulation ?

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 17 / 18

Q & A

Thank you for your attention

Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 18 / 18

vincent.agnus@ircad.fr