![Page 1: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/1.jpg)
Illumination Independent Marker Tracking using
Cross-Ratio Invariance
Vincent Agnus Stéphane Nicolau Luc Soler
Institut de Recherche Contre les Cancers de l'Appareil Digestif
ACM Symposium on Virtual Reality Software and Technology(VRST2014)
November 11, 2014
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 1 / 18
![Page 2: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/2.jpg)
Outline
Causes & E�ects of light condition on corner detection
How cross-ratio can be used to correct corner shifting
Method & validation
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 2 / 18
![Page 3: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/3.jpg)
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
![Page 4: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/4.jpg)
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
![Page 5: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/5.jpg)
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
![Page 6: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/6.jpg)
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
![Page 7: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/7.jpg)
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
![Page 8: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/8.jpg)
Motivation
Medical applications using markers
A low cost and accurate guidance system for
laparoscopic surgery : Validation on an
abdominal phantom [Nicolau VRST2005]
A low cost simulator to practice ultrasound
image interpretation and probe manipulation :
Design and �rst evaluation [Nicolau ISVRI2011]
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18
![Page 9: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/9.jpg)
Motivation
Medical applications using markers
A low cost and accurate guidance system for
laparoscopic surgery : Validation on an
abdominal phantom [Nicolau VRST2005]
A low cost simulator to practice ultrasound
image interpretation and probe manipulation :
Design and �rst evaluation [Nicolau ISVRI2011]
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18
Error on tag position
⇒ Larger position errors for :
Needle tip
UltraSound virtual plane
![Page 10: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/10.jpg)
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
![Page 11: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/11.jpg)
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance Measured Brightness
RadiometricResponse
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
![Page 12: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/12.jpg)
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance Measured Brightness
RadiometricResponse
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
![Page 13: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/13.jpg)
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance Measured Brightness
Rasterisation Spatial & IntensityRadiometric
Response
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
![Page 14: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/14.jpg)
Proposed Method
Key Idea
Light Condition
underexposure : black region ↗, white region ↘overexposure : black region ↘, white region ↗
⇒ error on estimated position of the marker
How we can detect and correct this phenomena ?
Our method
use the marker geometry : inner & outer corners
check that white/black region proportion are preserved
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 6 / 18
![Page 15: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/15.jpg)
Proposed Method
Corner Detection ShiftDetection & Correction
Cross Ratio (CR)
A projective invariant feature de�nedfor collinear points sets
A B C D
A′ B′ C ′ D′
CR =(AB/AC)
(DB/DC)=
(A′B′/A′C ′)
(D′B′/D′C ′)
Cross Ratio on markers
ABC
D
A BC D
CR should be constant :position & light
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18
![Page 16: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/16.jpg)
Proposed Method
Corner Detection ShiftDetection & Correction
Cross Ratio (CR)
A projective invariant feature de�nedfor collinear points sets
A B C D
A′ B′ C ′ D′
CR =(AB/AC)
(DB/DC)=
(A′B′/A′C ′)
(D′B′/D′C ′)
Cross Ratio on markers
ABC
D
A BC D
CR should be constant :position & light
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18
![Page 17: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/17.jpg)
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
![Page 18: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/18.jpg)
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
![Page 19: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/19.jpg)
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
![Page 20: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/20.jpg)
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
![Page 21: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/21.jpg)
Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of themarker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18
![Page 22: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/22.jpg)
Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of themarker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18
![Page 23: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/23.jpg)
Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of themarker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18
![Page 24: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/24.jpg)
Proposed Method
Correction of corner shifts
How cross ratio invariance can be used to �x corners shifting ?
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 10 / 18
![Page 25: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/25.jpg)
Proposed Method
Correction of corner shifts
Idea
Displace the corners such that CR(displaced) = CRmodel
Method
H
(+dx,+dy)
(+dx,−dy)
p3
p1 p5 p6p2
p7p14
p15
p4p9
p12 p11 p16
p8p10p13
p′1
H(p1)
p′i = H(pi) + (±dx,±dy)
(dx, dy) = argmindx,dy
∑(a,b,c,d)∈I
(CR(p′a, p′b, p′c, p′d)− CRmodel)
2
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18
![Page 26: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/26.jpg)
Proposed Method
Correction of corner shifts
Idea
Displace the corners such that CR(displaced) = CRmodel
Method
H
(+dx,+dy)
(+dx,−dy)
p3
p1 p5 p6p2
p7p14
p15
p4p9
p12 p11 p16
p8p10p13
p′1
H(p1)
p′i = H(pi) + (±dx,±dy)
(dx, dy) = argmindx,dy
∑(a,b,c,d)∈I
(CR(p′a, p′b, p′c, p′d)− CRmodel)
2
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18
![Page 27: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/27.jpg)
Proposed Method
Correction of corner shifts
New Corner Positions
H(+dx,+dy)
(+dx,−dy)
p3
p1 p5 p6p2
p7p14
p15
p4p9
p12 p11p16
p8p10p13
p′1H(p1)
correct(pi) = H−1(H(pi) + (±dx,±dy)︸ ︷︷ ︸
p′i
)
corner in original image corrected cornerVincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 12 / 18
![Page 28: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/28.jpg)
Proposed Method
Optimization Process
Minimization
argmindx,dy
∑(a,b,c,d)∈I
(CR(p′a, p′b, p′c, p′d)− CRmodel)
2
Levenberg-Marquardt algorithm
not optimal : jacobian not provided, estimated from aforward-di�erence approximation
5 ms for each image : SciPy , (Core [email protected]), 10 lines for CR
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 13 / 18
![Page 29: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/29.jpg)
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
![Page 30: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/30.jpg)
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
![Page 31: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/31.jpg)
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Pi
O
Πcheck
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
![Page 32: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/32.jpg)
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Pi
P proji
O
Πcheck
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
![Page 33: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/33.jpg)
Validation
ValidationExperiments
Chessboarddistance
cross ratio
Before correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
![Page 34: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/34.jpg)
Validation
ValidationExperiments
cross ratio
After correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
![Page 35: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/35.jpg)
Validation
ValidationExperiments
After correction
distance
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
![Page 36: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/36.jpg)
Validation
ValidationExperiments
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
Logitech c© HD ProWebcam C9101920×1080
Microsoft c© Lifecam 1280×720 Logitech c© Procam 640×480
![Page 37: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/37.jpg)
Validation
ValidationExperiments
3 cameras × 5 orientations (range 40-60cm) × 8 light conditions3 groups : illumination condition.over-exposure generate more errors.
without correction with correction
Under-exposure 1.15 ± 0.22 mm 0.42 ± 0.19 mmNormal exposure 0.90 ± 0.32 mm 0.67 ± 0.22 mmOver-exposure 2.60 ± 0.70 mm 0.71 ± 0.25 mm
Table: average errors of the marker depth estimation
Cross Ratio :
Can detect extreme light condition
Can correct depth estimation (<1 mm)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 16 / 18
![Page 38: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/38.jpg)
Discussion
Discussion & Perspectives
Discussion
estimated homography H → corner correction
assumption detected corners are collinear
assumption of isotropic in�uence of light after homographicrecti�cation
Future Works
use inner corners of markers
minimization : analytical formulation ?
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 17 / 18
![Page 39: Illumination Independent Marker Tracking using Cross-Ratio Invariance](https://reader034.vdocuments.net/reader034/viewer/2022042614/559b38361a28abf23e8b45e5/html5/thumbnails/39.jpg)
Q & A
Thank you for your attention
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 18 / 18