detection, rectification and segmentation of coplanar repeated patterns

Detection, Rectification and Segmentation of Co-planar Repeated

PatternsJames Pritts

Ondrej Chum and Jiri Matas

Center for Machine Perception (CMP)Czech Technical University in Prague

Faculty of Electrical Engineering Department of Cybernetics

3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns

Introduction Repetitive patterns are ubiquitous in images Unless considered, they usually decrease vision algorithm

performance Seek a model-based approach to precisely locate and

segment general co-planar repeats

2/25


GOAL: Create a short-list of non-random matches to query image

Because of “burstiness”, repeated elements are over-counted:

High frequency words from repeats skew scoring

Co-occurring features are not independent

Image form H. Jegou and Matthijs Douze, On the burstiness of visual elements. In CVPR, 2009.

Problems with Repetitions: Image Retrieval

Query

Match??3/25


Problems with Repetitions: Stereo Matching

Cannot disambiguate tentative correspondences F-estimate invalid

Epipolar constraint provides only weak spatial verification (even with good F)

4/25

mismatchedmismatched


Prior Work Detecting repeats is well studied

Rectification is nearly universal (vanishing lines)

Exploit some constraint that is valid in rectified space

Lattice Schaffalitzky, F., Img. Vis. Comp.

2000 Lattice, Axial Symmetry

Wu et al, CVPR 2011 Symmetry

Hong et al, IJCV 2004 Congruency

Liebowitz et al, CVPR 1998 Aiger et al, Comp. Graph. Forum

20125/25


The State of the Art TILT: Zhang et al., IJCV 2012.

Find homography minimizing image rank

Manual cueing of pattern required

Fails with significant perspective warp, occlusions or if repeats are sparse

Aiger et al, Comp. Graph. Forum 2012 Joint maximization of

congruent line segments has no convergence guarantee

Systems of rational equations sampled by Hough transform (slow)

6/25


Problem Formulation

Task: Segment imaged co-planar repeats with pixel level accuracy Some scene element repeats on a plane Need not have any regularity or be densely sampled Work without image structure modulo the repeat

- A common assumption is the existence of vanishing lines in the image that can be used to rectify the scene plane

Work when repeats cover only a small part of the image Fully automated: no cueing is required Can segment pattern to pixel level accuracy

Assumptions Repeated scene elements are coplanar Scene elements can be mapped to each other by rigid

transforms Imaged by perspective camera Scene element is repeated at least 3 times

7/25


Proposed method A method for detection, precise alignment and segmentation of

general co-planar repeated patterns

8/25


Intra-Image Feature Correspondence

Extremal regions (MSERs) detected for local representation of images

Local Affine Frames (LAFs) derived from extremal regions to concisely capture local geometry

Affine frames described by SIFTs

T(Ax)=AT(x)Affine covariance

11/25


Feature Correspondence to Clusters Cluster: set of LAFs that are photometrically consistent.

12/25


From Clusters to Repeats: Rectification

Spatial verification of photometric clustering is needed Perspective and affine imaging does not preserve scale or

congruency Need general rectification method for rigidly transformed repeats

13/25

image rectified


Rectification Stratum Translated and rotated co-

planar pattern

affinity similaritysimilarity w\scale ambiguity

14/25



planar pattern

Translation: Affine rectification


14/25



planar pattern


Rotation: Similarity rectification


14/25



planar pattern


Reflection: Similarity with scale ambiguity along reflection axis

Rotation: Similarity rectification


14/25


Affine Rectification (Chum et al [3]) Assumption: repeated elements in real world are equiareal. Constraint: images of repeated elements should be equiareal after affine

rectification.

Source imageCoordinates and scales are known

Destination imageOnly scales are known (no positions)

H

Result: unit area ratio, but not necessarily equal angles and extent length ratios

18/25


Similarity Rectification Assumption: repeated elements in real world have equal extent lengths. Constraint: images of repeated elements should have equal extent lengths. Result: Equal area ratios, relative extent lengths preserved, equal angles

Rotation Reflection

Imaged

Scene

2 LAFS needed 1 LAF needed

19/25


Affinity removal with reflections

20/25


Rectification Results

22/25


Repeats to Motifs Repeat: photometrically consistent cluster of local affine frames

(LAFs) that satisfies scale constraint

Motif: is a collection of repeats that are spatially coherent

Instance: An occurrence of the motif in the pattern

Goal: Estimate a motif and set of transforms between

23/25


Motif Estimation Open Problem: Formulate cost that balances model complexity and

motif cardinality

24/25


Greedy Motif Construction

25/25


Generative Model Generate the imaged pattern from the motif

motif

27/25


Generative Model Generate the imaged pattern from the motif Estimate pattern and rectification from image

SIFTs extracted from image and clustered Rectification estimated from linear constraints Clusters verified against scale constraint to make repeats Geometric hashing of LAFS in rectified space to construct

motif

motif

27/25


Generative Model Generate the imaged pattern from the motif Estimate pattern and rectification from image

SIFTs extracted from image and clustered Rectification estimated from linear constraints Clusters verified against scale constraint to make repeats Geometric hashing of LAFS in rectified space to construct

motif Refine pattern, rectification and lens distortion by minimizing

pattern re-projection error

motif

27/25


Motif Construction

28/25


Cows

30/25


Multiple motifs

34/25


Future Work Seek join estimation of photometric clustering and rectification

Sequential estimation is error prone, especially for multiple planes

Failure modes

Infer rectification jointly from more constraints Broaden the class of images from which patterns can be

extracted Model complexity cost to principally estimate number of planes Formulate optimization problem for motif construction Integrate into image retrieval engine

37/25


Conclusions Demonstrated effectiveness of new linear constraints

valid for nearly all man-made patterns effective in a fast RANSAC framework

Improved the state-of-the-art (TILT, Aiger et al): Rectifies patterns that are: a small part of the image, of low

texture Localizes pattern automatically Affine distortion can be removed with as few as 1 repeat

Explicitly model the pattern Segmentation of pattern at pixel-level SIFT variance decreased by using refined pattern to resample

image Multiple motifs can be used to jointly optimize rectification

38/25


Questions

Thanks for your attention

***Cosegmentations performed by J. Cech, J. Matas, and M. Perdoch. Efficient sequential correspondence selection by cosegmentation. In CVPR, 2008.

39/25

detection, rectification and segmentation of coplanar repeated patterns

Science