Curating Subject ID Labels using KeypointSignatures

Laurent Chauvin1 and Matthew Toews1

1Ecole de Technologie Superieure, Montreal, Canadalaurent.chauvin0@gmail.com

Subject ID labels are unique, anonymized codes that can be used to groupall images of a subject while maintaining anonymity. ID errors may be inadver-tently introduced manually error during enrollment and may lead to systematicerror into machine learning evaluation (e.g. due to double-dipping) or potentialpatient misdiagnosis in clinical contexts. Here we describe a highly efficient sys-tem for curating subject ID labels in large generic medical image datasets, basedon the 3D image keypoint representation, which recently led to the discovery ofpreviously unknown labeling errors in widely-used public brain MRI datasets [1].

Method: ID errors are identified using a pair-wise (dis)similarity measure d(A,B)to evaluate and flag all image pairs (A,B) that are unexpectedly similar ordissimilar. The primary computational challenge is the O(N2) algorithmic com-plexity of pairwise comparisons for a set of N images, which becomes intractableas N → ∞. d(A,B) is defined by the log of the Jaccard distance [3] betweenkeypoint sets A = {fi} and B = {fj}, as illustrated in Figure 1. Salient 3D SIFT-Rank keypoints [7] are detected in individual images, where each keypoint is ageneric local anatomical pattern and is represented as a compact 64-element de-scriptor derived from rank-ordered gradient histogram elements [6]. Enumeratingnearest neighbor (NN) descriptor matches, i.e. equivalent anatomical patternsbetween N2 image pairs is achieved in O(N log N) complexity via approximateK-NN search using the KD-tree structure [5].

I. Keypoint Detection(3D SIFT-Rank)

II. Keypoint Matching (Nearest Neighbors)

III. Set Similarity Measurement (Jaccard Overlap)

A

B

A

B

3D Images(e.g. MRI, CT)

J(A,B)=

|A∩B|

|A∪B|

Fig. 1. Computing the log Jaccard distance d(A,B) = − log J(A,B) between a pairof generic medical image volumes (A,B) involves (left to right) SIFT-Rank keypointdetection [7], K-NN keypoint matching and Jaccard overlap evaluation.

arX

iv:2

110.

0405

5v1

[cs

.CV

] 7

Oct

202

1

2 Laurent Chauvin and Matthew Toews

Results: A total of N = 7536 MRIs of 3334 unique subjects are pooled from fourlarge public neuroimaging databases and used in evaluation: the Human Connec-tome Project (HCP) [8], the Open Access Series of Imaging Studies (OASIS) 1and 3 [4], and the Alzheimer’s Disease Neuroimaging Initiative (ADNI) [2]. Eachof N(N−1)/2 = 28, 391, 880 image pairs (A,B), is associated with a unique pair-wise relationship label: same subjects (SM), monozygotic twins (MZ), dizygotictwins (DZ), full-sibling (FS), and unrelated (UR), images in different databasesare assumed to be unrelated. As expected, pairwise distance distributions in Fig-ure 2 are lowest and highest for the same (SM, pink) and unrelated subjects (UR,blue), respectively, with twin and non-twin sibling distances falling in between.Obvious outliers (blue, pink points) reveal labeling errors (see Figure 2), whichwere visually confirmed to be either images of the same subject mislabeled asdifferent (blue dots) or images of different subjects labeled as the same (pink).

Freq

uenc

y

Unrelated Subjects (UR)(N=28,381,485)

Same Subjects (SM)(N=9,583)

UR pairs mislabeled as SMSM pairs mislabeled as UR & MZ

Full-Siblings (FS)(N=607)

Dizygotic Twins (DZ)(N=71)

Monozygotic Twins (MZ)(N=134)

d(A,B)

a)

b)

c)

Fig. 2. Distributions of the pairwise log Jaccard distance d(A,B) conditional on re-lationship labels, dots indicate obvious label inconsistencies, with visual examples ofsame subject images mislabelled as different (a,b) and different subject images labeledas the same (c).

Discussion: For a typical 2563-voxel image, keypoint detection requires 1-2 sec-onds per image via a GPU implementation 1 and produces 1000-5000 keypointsper image depending on the content. Enumerating all NN images requires 100-350 milliseconds per image. The memory footprint for keypoints is 100x smallerthan image data and could be incorporated as a DICOM image header. TheHCP has since removed duplicate subjects.

1 https://github.com/3dsift-rank/3DSIFT-Rank

Curating Subject ID Labels using Keypoint Signatures 3

References

1. Chauvin, L., Kumar, K., Wachinger, C., Vangel, M., de Guise, J., Desrosiers, C.,Wells, W., Toews, M.: Neuroimage signature from salient keypoints is highly spe-cific to individuals and shared by close relatives. NeuroImage 204(20) (9 2020).https://doi.org/10.1016/j.neuroimage.2019.116208

2. Jack, C.R.C., Bernstein, M.A., Fox, N.C.N., Thompson, P., Alexander, G., Harvey,D., Borowski, B., Britson, P.J., L. Whitwell, J., Ward, C., Dale, A.M., Felmlee, J.P.,Gunter, J.L., Hill, D.L., Killiany, R., Schuff, N., Fox-Bosetti, S., Lin, C., Studholme,C., DeCarli, C.S., Gunnar Krueger, G., Ward, H.A., Metzger, G.J., Scott, K.T.,Mallozzi, R., Blezek, D., Levy, J., Debbins, J.P., Fleisher, A.S., Albert, M., Green,R., Bartzokis, G., Glover, G., Mugler, J., Weiner, M.W., Gunnar Krueger, H.A.,Ward, H.A., Metzger, G.J., Scott, K.T., Mallozzi, R., Blezek, D., Levy, J., Deb-bins, J.P., Fleisher, A.S., Albert, M., Green, R., Bartzokis, G., Glover, G., Mu-gler, J., Weiner, M.W.: The Alzheimer’s disease neuroimaging initiative (ADNI):MRI methods. Journal of Magnetic Resonance Imaging 27(4), 685–691 (4 2008).https://doi.org/10.1002/jmri.21049

3. Levandowsky, M., Winter, D.: Distance between Sets. Nature 234(5323), 34–35 (111971). https://doi.org/10.1038/234034a0

4. Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G., Morris, J.C.,Buckner, R.L.: Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and DementedOlder Adults. Journal of Cognitive Neuroscience 19(9), 1498–1507 (9 2007).https://doi.org/10.1162/jocn.2007.19.9.1498

5. Muja, M., And, D.L.I.t.o.p.a., 2014, U.: Scalable nearest neighbor algorithms forhigh dimensional data. ieeexplore.ieee.org (2014)

6. Toews, M., Wells, W., William, W.: Sift-rank: Ordinal description for invariantfeature correspondence. In: 2009 IEEE Computer Society Conference on ComputerVision and Pattern Recognition Workshops, CVPR Workshops 2009. pp. 172–177(2009). https://doi.org/10.1109/CVPRW.2009.5206849

7. Toews, M., Wells, W.M.: Efficient and robust model-to-image alignment us-ing 3D scale-invariant features. Medical Image Analysis 17(3), 271–282 (2013).https://doi.org/10.1016/j.media.2012.11.002

8. Van Essen, D., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T., Bucholz,R., Chang, A., Chen, L., Corbetta, M., Curtiss, S., Della Penna, S., Fein-berg, D., Glasser, M., Harel, N., Heath, A., Larson-Prior, L., Marcus, D.,Michalareas, G., Moeller, S., Oostenveld, R., Petersen, S., Prior, F., Schlag-gar, B., Smith, S., Snyder, A., Xu, J., Yacoub, E.: The Human ConnectomeProject: A data acquisition perspective. NeuroImage 62(4), 2222–2231 (10 2012).https://doi.org/10.1016/j.neuroimage.2012.02.018