mixed-resolution patch-matching
DESCRIPTION
Mixed-Resolution Patch-Matching. Harshit Sureka Guide: Dr. P. J. Narayanan. Patch-Matching. 01/08/13. 2. Scope of Patch-Matching Algorithms. Image credits: Image Montage (Simakov et al. 2008). Object Detection, - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/1.jpg)
Mixed-Resolution Patch-Matching
Harshit Sureka
Guide: Dr. P. J. Narayanan
![Page 2: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/2.jpg)
Patch-Matching
201/08/13
![Page 3: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/3.jpg)
Scope of Patch-Matching Algorithms
Image credits: Image Montage (Simakov et al. 2008). Object Detection,Forgery Detection and Image Reshuffling (Barnes et al. 2010). Texture synthesis and Image completion (Xiao et al. 2011)01/08/13 3
![Page 4: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/4.jpg)
Brute force v/s state-of-the-art
• Find the matching patches to the patch in red• Brute-force: 2125 msec
• State-of-the-art approaches: 1.5msec (1400x faster)
4
Image Credit: Kumar et al. (ECCV 2008)
01/08/13
![Page 5: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/5.jpg)
The Patch-Matching Problem
Image Credit: Barnes et al. (SIGGRAPH 2009)
01/08/13 5
![Page 6: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/6.jpg)
Types of Patch-Matching• Sparse
– Finding matches for a few key-points
• Dense– Matches for all patches in the
source Image• Local
– Search for matches in a specific region in the target image
• Global– Search everywhere for matches in
the target image
• Local is computationally less expensive
• But, will it suffice?
Sparse
Local Global
Dense
601/08/13
![Page 7: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/7.jpg)
Where do similar patches lie?
• Our focus dense and global• Since a lot of patches lie at large distances
701/08/13
![Page 8: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/8.jpg)
Next..
• What is a patch?• The formal patch-matching problem• Motivation for making patch-matching
algorithms faster• Types of patch-matching
Till now..
• Parameters of Patch-matching algorithms• Related Work
801/08/13
![Page 9: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/9.jpg)
Parameters of Patch-Matching Algorithms
901/08/13
![Page 10: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/10.jpg)
Previous Efforts
• To use general-purpose Approximate Nearest Neighbor (ANN) algorithms– KD trees– PCA trees– etc.
• They provide good solutions to general ANN problems
• Do not take advantage of the structure and properties specific to images
1001/08/13
![Page 11: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/11.jpg)
Previous Efforts: PatchMatch
• Barnes et al., SIGGRAPH 2009• Randomized cooperative Hill-Climbing Search• Based on the property of coherence in image patches• 3 steps of the algorithm:
Image credit: Barnes et al. 2009
• Up-to 2x faster than all previous approaches
• Enables several interactive applications
• Not as accurate as one would hope
• Coherency Assumption: Not true in highly textured images
1101/08/13
![Page 12: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/12.jpg)
Previous state-of-the-art: Coherency Sensitive Hashing (CSH)
• Korman and Avidan, ICCV 2011• Based on PatchMatch– Replaces the Random step of PatchMatch with a Hash
Table
• Combines cues of location (of coherence) and appearance in a novel manner
Type 1 Type 2 Type 312
Appearance Coherence
01/08/13
![Page 13: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/13.jpg)
Coherency Sensitive Hashing (CSH)• Targeted propagation of information
• Lower RMS error values in early iterations compared to PatchMatch
• Faster and more accurate than Patchmatch
• Less dependent on the coherency assumption
• Lowest reconstruction errors– Current state-of-the-art
1301/08/13
![Page 14: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/14.jpg)
Next..
• Introduction to Patch-Matching
• Previous techniques– General purpose ANN – PatchMatch– Coherency Sensitive Hashing (CSH)
Till now..
• Image Pyramids• Pyramids for Patch-Matching
1401/08/13
![Page 15: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/15.jpg)
Pyramids• Image represented at multiple scales
15
Level 0
Level 1
Level 2
Level 3
Level 4512x512 32x32
01/08/13
![Page 16: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/16.jpg)
Pyramids
• Pyramid coarse-to-fine framework is well established (Burt et al. 1984)
• Help a variety of problems– Image compression– Image alignment– Optical flow– Object tracking–Matching: do away with fine details, look at only
high level information
16Can they also be used for Patch-Matching? Why not!
01/08/13
![Page 17: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/17.jpg)
Pyramid Patch-Matching• Novel patch-matching algorithm• Pyramid representation of original images
– Coarsest level image size determined by a threshold ‘Pthold’• Exhaustive global search at small images of coarsest level• Search localization at finer levels by up-sampling the
matches found at coarser levels
1701/08/13
![Page 18: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/18.jpg)
Pyramid Patch-Matching (PPM)
• Coarsest level
18
• Finer levels
a* is the patch in IA(p+1) corresponding to a01/08/13
![Page 19: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/19.jpg)
PPM: Real Image example
1901/08/13
![Page 20: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/20.jpg)
Pyramid Patch Matching• Increase K– Best match at coarsest resolution may not be the best
match at original resolution– Store more number of matches– Wider search range at finer levels– Up to 2% better accuracy
20
• Reduce Patch-Size– Small size patches matched
at coarser resolutions
• Early Termination– Of distance calculation if it exceeds the distance of the
farthest patch currently in the matched list of patches– Up to 1.7x speed-up
01/08/13
![Page 21: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/21.jpg)
Parallel Implementation• PPM processes each patch independently• Pyramid construction– Standard parallel implementation
• Coarsest level– Each thread assigned a block of source patches– Searches for best matches in the whole target image
• Finer levels– Up-sampling of matches; candidate-list generation– Each thread searches the candidate space of the patches
in its block• Near-linear time speed up w.r.t. number of cores
2101/08/13
![Page 22: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/22.jpg)
GPU Implementation
• Pyramid construction on the GPU– Standard implementation
• Each patch is matched independently
• Distance calculation of each query patch with its candidate matches is also independent
2201/08/13
![Page 23: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/23.jpg)
GPU Implementation• Algorithm– Exhaustive search at the coarsest image resolution– Localized search at finer resolution levels– For each patch
• threads equal to the number of candidate patches are invoked
• Each thread handles one vector-distance calculation– Up-sampling of matches; candidate-list generation
also in parallel• Additional 70x compared to implementation run
on 4-core (~250x compared to single-core)
2301/08/13
![Page 24: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/24.jpg)
GPU Timings
• Values for CSH Dataset Images (shown in next slide)
• Brute force for Lena (256x256 image)• Most gain in speed achieved for brute-force
global exhaustive search
24
Search CPU GPU Speed-up
3 x 3 13.46 sec 278 msec 48x
5 x 5 20.45 sec 684 msec 30x
7 x 7 29.68 sec 1113 msec 26x
Brute-Force(Lena, 512x512 image)
356.63 sec 4.89 sec 72x
01/08/13
![Page 25: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/25.jpg)
CSH Dataset for Experiments• 133 image-pairs from the dataset provided by Korman and Avidan, ICCV 2011 (CSH)• Both images of the same scene of a movie
• between 1 and 30 frames apart in the video
2501/08/13
![Page 26: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/26.jpg)
ParametersVarying KNN and Patch-size
26
Varying Pthold
Varying Search-range
Error: RMS distance between RGB values of patches, averaged over all matches of a patch and over all patches in the image
Platform: Intel i7 920 processor, 2.67 GHz
Values over all 133 pairs of images in the CSH dataset (shown in next slide)
01/08/13
![Page 27: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/27.jpg)
Experiment I• Proximity to ground-truth– Brute-force search for ground-truth matches– PPM captures up to 4% more ground truth
matches– Better performance for more number of matches
required
2701/08/13
![Page 28: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/28.jpg)
Experiment II• Error (RMS error between RGB values of patches and their matches)
– Parameters set to achieve similar run-times– PPM achieves consistently lower error values than
CSH• Better performance than state-of-the-art– Much closer to ground truth (Calculated by Brute-force on GPU)
2801/08/13
![Page 29: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/29.jpg)
Next..
• Patch-Matching and previous techniques• Pyramid Patch-Matching– Improvements: Increase K, Reduce patch-size,
early-termination– Experiments: State-of-the-art performance
Till now..
• Mixed-Resolution Vectors (MR-vectors)– Motivation for mixing resolutions
• Mixed-Resolution Patch-Matching (MRPM) 2901/08/13
![Page 30: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/30.jpg)
Mixed-Resolution: Motivation
• Images at coarser resolutions contain approximate information
• Small objects may get smoothed out due to downsampling
• A match missed at the coarsest resolution will never be recovered later in the algorithm
• Solution: More information already exists at finer resolutions. Use it!
3001/08/13
![Page 31: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/31.jpg)
Mixed-Resolution Vectors• Idea– Concatenate corresponding patch-vectors of finer
resolutions and current resolution• Including Information– Currently we pick a same-sized patch around the
center of the corresponding up-sampled patch– Available prior knowledge about the image or patches
can be used to pick information more wisely!
3101/08/13
![Page 32: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/32.jpg)
MR-Vectors
• Primary contribution• Can include information from multiple levels
3201/08/13
![Page 33: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/33.jpg)
Mixed-Resolution Patch-Matching
• In the framework of PPM, we replace traditional patch-vectors with MR-vectors
• Rest of the algorithm remains same
• We call this the “Mixed-Resolution Patch-Matching” Algorithm (MRPM).
3301/08/13
![Page 34: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/34.jpg)
Comparison with increasing search range
• Accuracy increases with cost of time when using a wider search range around up-sampled matches
• Similar behavior is observed while mixing resolutions
34
Mixed-Resolution achieves lower error values with less cost of time
01/08/13
![Page 35: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/35.jpg)
Parallel Implementation
• Fast and efficient parallel implementations of MRPM follow from parallel implementations of PPM
• Multicore: Near-linear time speed-up• GPU: All levels already on GPU memory
35
Search CPU GPU Speed-up
3 x 3 19 sec 442 msec 43x
5 x 5 30 sec 937 msec 32x
7 x 7 45 sec 1.73 sec 26x
01/08/13
![Page 36: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/36.jpg)
PPM v/s MRPM
• Lower error values with marginal cost of time• Less effect of time cost on GPU performance
• Note: error not averaged over patch-size36
(l = 2)
01/08/13
![Page 37: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/37.jpg)
Experiment I
• Proximity to ground truth
• Up to 8% more of the ground truth matches are captured by MRPM than CSH
• 4% more than PPM
3701/08/13
Best K Matches CSH PPM MRPM
Best Match 92.43% 92.48% 93.57%
Best 5 Matches 88.06% 89.7% 91.51%
Best 10 Matches 81.72% 85.69% 89.87%
![Page 38: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/38.jpg)
Experiment II
• Error– Parameters set to achieve similar run time–MRPM achieves near optimal patch-matching
3801/08/13
![Page 39: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/39.jpg)
Next..
• Patch-Matching and previous techniques• Pyramid Patch-Matching• Mixed-Resolution Vectors• Mixed-Resolution Patch-Matching
Till now..
• Applications• Conclusion
3901/08/13
![Page 40: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/40.jpg)
Applications
• A variety of applications have a patch-matching engine at their core
• MRPM can act as that engine and help improve quality and speed of several of these applications
• We discuss 3 of these– Image de-noising– Image summarization (Retargeting)– Auto-crop
4001/08/13
![Page 41: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/41.jpg)
Image denoising• Performs a weighted averaging
over similar patches to de-noise the image (Non-local means, Buades et al., 2005)
• Currently local due to computational constraints– Also improves accuracy in some
cases
• MRPM can be used to extend it to global
4101/08/13
![Page 42: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/42.jpg)
Experiments
• Dataset of 36 images collected for evaluating denoising performance
42
Dataset of 36 images
Combining matches of local (non-local means) and global (MRPM)
01/08/13
![Page 43: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/43.jpg)
Self-similar Image
• Evaluation: Method-noise• Less structure removed from Image
43
PSNR: 33.35
PSNR: 34.03
01/08/13
![Page 44: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/44.jpg)
Image Retargeting (Summarization)
• Simakov et al. 2008, Bidirectional Similarity
4401/08/13
![Page 45: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/45.jpg)
Results
• MRPM achieves better visual qualityof results
4501/08/13
![Page 46: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/46.jpg)
Auto-crop
• Similar to Bi-directional Similarity– Coherence term muted: since all cropping
windows would anyway be coherent with the image
• Cropping based on completeness term similarity
4601/08/13
![Page 47: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/47.jpg)
More Results
4701/08/13
![Page 48: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/48.jpg)
Limitation: Image Reconstruction• Given an Image B and a dense patch-matching from A
to B. Reconstruct image A• Each pixel is replaces with the average of the
corresponding pixels that it is mapped to by all patches that contain it
4801/08/13
![Page 49: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/49.jpg)
Summary
• Presented a Pyramid based approach to the problem of patch-matching– 3 Improvements to improve accuracy and efficiency
• Achieved state-of-the-art matching accuracy• Introduced the concept of Mixed-Resolution
Vectors• Proposed Mixed-Resolution Patch-Matching
outperforming state-of-the-art by up-to 8% more ground-truth matches captured
4901/08/13
![Page 50: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/50.jpg)
Future work
• Include coherency cues in the framework of the algorithm– Up-sampling maintains some coherency
• Matching across all rotations and scale• Matching in videos
• Explore the scope of MR-vectors– We believe it is a generic concept and the idea can be
used in several other problems that use the multi-resolution technique
5001/08/13
![Page 51: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/51.jpg)
Publication(s) Harshit Sureka, P. J. Narayanan. Mixed-Resolution Patch-
Matching. In the 12th European Conference on Computer Vision, ECCV 2012, 7th to 13th October 2012, Florence, Italy.
5101/08/13
![Page 52: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/52.jpg)
Thank you!!
5201/08/13
![Page 53: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/53.jpg)
Backup Slides
5301/08/13
![Page 54: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/54.jpg)
Downsampling Method
• All, but the nearest-neighbor interpolation perform well
• Best: Gaussian Averaging
5401/08/13
![Page 55: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/55.jpg)
Increase K• Storing more number of matches at coarser
resolutions than originally required• Increase in accuracy up-to 2%• More pronounced effects when less number of
matches are required (i.e. ‘K’ is low)
5501/08/13
![Page 56: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/56.jpg)
Early Termination
• Early Termination– Of distance calculation if it exceeds the distance of
the farthest patch currently in the matched list of patches
– Up to 1.7x speed-up
5601/08/13
![Page 57: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/57.jpg)
GPU Implementation Details
• Each patch represented as a 1-D vector called the Patch-vector
• Bottleneck: Vector-distance calculation
• Each thread performs only one vector-distance calculation of query patch vector with target image candidate patch vector
5701/08/13
![Page 58: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/58.jpg)
GPU Implementation
• Distance stored in a distance array of the query patch
• Distances are stored in independent arrays for each source patch (in parallel, using Thrust library)
• Up-sampling of patches also done in parallel
5801/08/13
![Page 59: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/59.jpg)
MRPM Algorithm
5901/08/13
![Page 60: Mixed-Resolution Patch-Matching](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149de550346895db7041b/html5/thumbnails/60.jpg)
Bidirectional Similarity
6001/08/13