Slide 1

Mixed-Resolution Patch- Matching (MRPM) Harshit Sureka and P.J. Narayanan (ECCV 2012) Presentation by Yaniv Romano 1

Slide 2

Problem Definition Patch Matching \ Nearest Neighbor Fields: For every patch in image A, find the K most similar patches in image B. NNF AB 2

Slide 3

Motivation BM3D Denoising [K. Dabov et al, IEEE-TIP 2007] 3

Slide 4

Motivation Super Resolution [M. Protter and M. Elad, IEEE-TIP 2009] 4 - known pixel Up-scaling by a factor of 2

Slide 5

Motivation Deinterlace [M. Protter and M. Elad, IEEE-TIP 2009] 5

Slide 6

6

Slide 7

Motivation Image Editing [Barnes et al, SIGRAPH 2009] 7

Slide 8

8

Slide 9

9

Slide 10

Challenges SIMPLE & ELEGANT Short RUN-TIME* High ACCURACY Strong IMAGE PRIOR Supports different SIMILARITY MEASURES ? 10 *very-low extra-memory

Slide 11

Todays talk Problem Definition. Motivation. Challenges. Related works. Pyramid Patch-Matching (PPM). Mixed Resolution Pyramid Patch-Matching (MRPM). Experimental Results. Pros & Cons. Future ideas. 11

Slide 12

An iterative randomized algorithm. Based on local propagation and mostly-local search. Heavily relies on the fact that images are generally coherent. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] 12 current match of left neighbor left neighbor candidate

Slide 13

An iterative randomized algorithm. Based on local propagation and mostly-local search. Heavily relies on the fact that images are generally coherent. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] 13 current match of top neighbor current match of left neighbor candidates left neighbor top neighbor

Slide 14

SIMPLE & ELEGANT Medium RUN-TIME* Medium ACCURACY Medium IMAGE PRIOR Supports different SIMILARITY MEASURES 14 *Compared to the state-of-the-art (CSH) Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] *very low extra-memory

Slide 15

1.Hash patches of both images into a table. 2.Find matches, based on: Appearance. Coherence. Type 1Type 2 Type 3 Appearance Appearance & CoherenceAppearance Image Space & Appearance Space CSH algorithm [Korman et al, ICCV 2011] PM 15

Slide 16

Image Space & Appearance Space CSH algorithm [Korman et al, ICCV 2011] Not SIMPLE at all Medium-Low RUN-TIME* Good ACCURACY Good IMAGE PRIOR Doesnt Support different SIMILARITY MEASURES (easily) 16 *Compared to previous methods. *medium-low extra-memory

Slide 17

17 Pyramid Patch-Matching

Slide 18

Down sampling the input image(s) Gaussian averaging, area averaging, cubic interpolation, and others. Image Pyramid 18

Slide 19

Pyramid Patch-Matching (PPM) 19

Slide 20

Pyramid Patch-Matching (PPM) 20

Slide 21

Pyramid Patch-Matching (PPM) 21

Slide 22

Pyramid Patch-Matching (PPM) 22

Slide 23

Pyramid Patch-Matching (PPM) Source Image Target Image

Slide 24

Pyramid Patch-Matching (PPM) Find matches 3 Exhaustive Search Upsampling & Expanding Search Upsampling 12 Source Image Target Image

Slide 25

Pyramid Patch-Matching (PPM) Exhaustive Search Upsampling & Expanding Search Find matches Upsampling 12 3 4 Source Image Target Image

Slide 26

Pyramid Patch-Matching (PPM) *Possible to use other patch-matching and nearest-neighbor algorithms for search.

Slide 27

PPM Pros Very simple. Fast exhaustive search at the coarsest level. Local searches at finer levels. 27

Slide 28

PPM Cons Small regions can disappear. The vicinity of a missed low resolution patch may not be searched in higher resolutions. Lower resolution versions may lead to: A smooth distance function. Poor localization of matches. 28

Slide 29

Mixed Resolution Pyramid Patch-Matching Upsampling ?

Slide 30

Mixed Resolution Pyramid Patch-Matching The algorithm is the same as before with the following major difference: 30 ? ? Use Mixed-Resolution Vectors instead of the traditional patch vectors

Slide 31

Pyramid Patch-Matching Exhaustive Search Upsampling & Expanding Search Find matches Upsampling 12 3 4

Slide 32

Mixed Resolution Pyramid Patch-Matching Exhaustive Search Upsampling & Expanding Search Upsampling 1 4 2 Find matches 3

Slide 33

Mixed Resolution Pyramid Patch-Matching Exhaustive Search Upsampling & Expanding Search Upsampling 1 4 2 Find matches 3

Slide 34

Search Range vs. Mixed Resolutions Search Range No. of Resolutions Mixed (L) Values: [Avg. Error* (Time)] L=1 (no mixing)L=2L=3 3X3109.47 (13 sec)96.71 (19 sec)93.93 (24 sec) 5X599.92 (22 sec)90.38 (30 sec)87.74 (35 sec) 7X796.81 (30 sec)88.62 (45 sec)86.18 (53 sec) 9X994.8 (41 sec)87.36 (63 sec)85.03 (73 sec) *Error = Average Euclidean distance between source and matched patches Lower error values are achieved with less cost of time by mixing resolutions Mixing resolutions is more effective than increasing the search range. 34

Slide 35

GPU Implementation Mixed resolution patch matching processes each pixel independently. Perfect for the heavy parallel architecture of GPU. 35

Slide 36

GPU Implementation Image SizeSearchCPUGPUSpeed-Up 1920X1080 3X313.46 sec278 msec48X 5X520.45 sec684 msec30X 7X729.68 sec1113 msec26X 256X256Exhaustive356.63 sec4.89 sec72X 36

Slide 37

Experimental Results Ground Truth - exhaustive search: K=10 NN. patch size = 8X8. Error = Avg. Euclidian distance between the patches and their matches. Mixed pixels from two resolution levels. 3X3 search range. 37

Slide 38

How many of the ground truth matches are found? Higher % implies better accuracy: 38 Best K MatchesCSHMRPM (Proposed)%MPRM %CSH Best Match92.43%93.57%1.14% Best 5 Matches88.06%91.51%3.45% Best 10 Matches81.72%89.87%8.15% Advantages of the MRPM method are more pronounced when several nearest matches are needed

Slide 39

Average Error With a similar computation time to CSH they obtain lower average error: 39

Slide 40

Image Reconstruction Find a dense patch matching between A & B: Reconstruct image A using image B. Reconstruction Mean Square Error, lower is better: 40 Ground TruthProposedCSHPatch Match 5.816.476.297.62 The proposed Image Prior is weaker than CSH (coherency)

Slide 41

Bottom line Very SIMPLE & ELEGANT State-of-the- art RUN-TIME* State-of- the-art ACCURACY Medium- Good IMAGE PRIOR Medium Support different SIMILARITY MEASURES *Compared to CSH *medium extra-memory 41

Slide 42

Mixed Resolution Pyramid Patch-Matching Pros: The algorithm: Faster than the state-of-the-art. Fast exhaustive search at the coarsest level. Local searches at finer levels. More accurate than the state-of-the-art. Take into consideration the fine details of the patches. Very simple and easy to implement. Parallel property, matching is independent. The article: Self-contained. Provides code (both CPU and GPU implementations). The comparisons to previous works are sufficient. Builds the story very well.

Slide 43

Mixed Resolution Pyramid Patch-Matching

Slide 44

Future ideas Future idea #1: Instead of image pyramid 44

Slide 45

Future ideas Future idea #1: Use the mixed vectors with Wavelets. The mixed vectors contain the whole image details compared to image pyramid

Slide 46

Future ideas Future idea #1 reduce data redundancy: Use the mixed vectors with Wavelets. Future idea #2 strengthen the image prior: Use image coherency. Exhaustive Search Upsampling & Expanding Search Upsampling 1 4 2 Find matches 3 46

Slide 47

47

Slide 48

We Are Done ! Thank You 48