Deep structure (Matching) Arjan Kuijper Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Deep structure What was found in the deep structure: Spatial critical pointsSpatial critical points In L(x,y;t c ): L = 0In L(x,y;t c ): L = 0 Critical curvesCritical curves In L(x,y;t): L = 0In L(x,y;t): L = 0 Catastrophe pointsCatastrophe points In L(x,y;t): det(H) = 0In L(x,y;t): det(H) = 0 Scale space saddlesScale space saddles In L(x,y;t): L = 0, D L = 0In L(x,y;t): L = 0, D L = 0 Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec What to match We could match several things: Regions in n-D Regions in n-D Regions in (n+1)-D Regions in (n+1)-D Points in (n+1)-D Points in (n+1)-D Hierarchies in (n+1)-D Hierarchies in (n+1)-D The advantage of Gaussian scale space is that it blurs everything awayThe advantage of Gaussian scale space is that it blurs everything away in a pre-defined way. in a pre-defined way. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Blurred region matching Segment the original images or their blurred versions at some scale and try to match similar areas Basically using non-scale space approaches Basically using non-scale space approaches Rigid or non-rigid registration methods Rigid or non-rigid registration methods Possibly match blurred images Possibly match blurred images Hardly attempted Hardly attempted Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Scale space region matching Construct a volume in scale space. The volume is a measure of importance. For example: Lindebergs blob algorithm. No matching algorithms known. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Matching Points Find the catastrophe points and / or the scale space saddles of two images and try to match them Handle the difference in number of points Handle the difference in number of points Handle the difference in location Handle the difference in location spatialspatial scalescale Earth Movers Distance may be useful. Preliminary results by Frans Kanters Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Hierarchy matching This is really fully using the deep structure Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Combining manifolds and critical curves At a scale space saddle two manifolds intersect. At a scale space saddle two manifolds intersect. Manifolds are limited: they have a top. Manifolds are limited: they have a top. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Nesting of manifolds in scale space Manifolds are related to other manifolds. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Void scale space saddles Beware that not all scale space saddles connect two separate manifolds. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Hierarchical Algorithm Initializing Build a scale space.Build a scale space. Find the critical points at each scale level.Find the critical points at each scale level. Construct the critical branches.Construct the critical branches. Find the catastrophe points.Find the catastrophe points. Construct and label the critical curves, including the one remaining extremum.Construct and label the critical curves, including the one remaining extremum. Find the scale space saddles.Find the scale space saddles. Determining the manifolds Find for each annihilations extremum its critical iso-intensity manifold.Find for each annihilations extremum its critical iso-intensity manifold. Construct the dual manifolds.Construct the dual manifolds. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Hierarchical Algorithm Labeling Assign to each extremum the dual manifolds to which it belongs, sorted on intensity.Assign to each extremum the dual manifolds to which it belongs, sorted on intensity. Build a tree: Start with the remaining extremum at the coarsest scale as root.Start with the remaining extremum at the coarsest scale as root. Trace to finer scale until at some value it is labeled to a dual manifold.Trace to finer scale until at some value it is labeled to a dual manifold. Split into two branches, on the existing extremum, one the extremum having the critical manifold.Split into two branches, on the existing extremum, one the extremum having the critical manifold. Continue for all branches / extrema until all extrema are added to the tree.Continue for all branches / extrema until all extrema are added to the tree. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Consider the blobs Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec De 5 De 3 De 1 De 2 Ce 2 Ce 1 Ce 3 Ce 5 e 4 e 2 e 1 e 3 e 5 R Example: manifolds and hierarchy Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec A real example Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Noise addition Mathematica If time. Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Several Trees Trees can be build up using the Catastrophe PointsCatastrophe Points Scale Space SaddlesScale Space Saddles Iso-Manifolds through themIso-Manifolds through them Different paradigms can be used Gaussian scale space itselfGaussian scale space itself Watersheds based on GSSWatersheds based on GSS Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Matching trees The next step is to match these Multi-Scale Singularity Trees. This is part of the reseach done at the DSSCV project.This is part of the reseach done at the DSSCV project. Eindhoven (NL)Eindhoven (NL) 3DLab, Kopenhagen University (DK)3DLab, Kopenhagen University (DK) Image Group, ITU (DK)Image Group, ITU (DK) Algorithms Group, ITU (DK)Algorithms Group, ITU (DK) Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Sources Scale Space Hierarchy A. Kuijper, L.M.J. Florack, M.A. Viergever Journal of Mathematical Imaging and Vision 18 (2): , 2003.Scale Space Hierarchy A. Kuijper, L.M.J. Florack, M.A. Viergever Journal of Mathematical Imaging and Vision 18 (2): , The hierarchical structure of images A. Kuijper, L.M.J. Florack IEEE Transactions on Image Processing 12 (9): , 2003.The hierarchical structure of images A. Kuijper, L.M.J. Florack IEEE Transactions on Image Processing 12 (9): , The deep structure of Gaussian scale space images Arjan KuijperThe deep structure of Gaussian scale space images Arjan Kuijper Generic Image Structure Ole Fogh OlsenGeneric Image Structure Ole Fogh Olsen Scale-Space Theory in Computer Vision Tony LindebergScale-Space Theory in Computer Vision Tony Lindeberg Multiscale Hierarchical Segmentation Bram Platel, Luc Florack, Frans Kanters, Bart ter Haar Romeny, ASCI 2003 Conference, June 4-6, 2003.Multiscale Hierarchical Segmentation Bram Platel, Luc Florack, Frans Kanters, Bart ter Haar Romeny, ASCI 2003 Conference, June 4-6, Content Based Image Retrieval using Multiscale Top Points Frans Kanters, Bram Platel, Luc Florack, Bart ter Haar Romeny, Scale Space 03, LNCS 2695: , 2003Content Based Image Retrieval using Multiscale Top Points Frans Kanters, Bram Platel, Luc Florack, Bart ter Haar Romeny, Scale Space 03, LNCS 2695: , 2003 Deep Structure Matching; PhD course on Scale Space, Cph 1-5 Dec Sinterklaas