Optimized neighbor embeddings for single-image super-resolutionMEHMET TÜRKAN, DOMINIQUE THOREAU, PHILIPPE GUILLOTEL, CHARLY POYAC

2

Summary• Basic concepts• Principle• Quality Improvements• Complexity improvements• Results• Conclusion & remaining work

3

Basic concepts

To resize an image, standard algorithms are interpolation based (linear, cubic…), they are fast but they cannot recovery high frequencies and the result is often blur.

Original low res image

Nearest neighbor interpolation

Bicubic interpolation

4

Basic conceptsSuper-resolution is class of techniques that enhance the resolution of an imaging system.

Methods for SR can be broadly classified into two families of methods: The classical multi-image super-resolution[3], and the Example-Based super-resolution[4][5][7][8].

Objectives of the internship

5

Starting from a reference algorithm designed by Mehmet Turkan, Dominique Thoreau, and Phillipe Guillotel, my job was:

• Switch original software developed from Matlab to C/C++.• Speed up the processing time and enhance the quality as much as possible.

Final objective is to upsample FullHD sequences to UHD in a suitable time.

6

Summary• Basic concepts• Principle• Quality Improvements• Complexity improvements• Results• Conclusion & remaining work

7

Principle

Small patches are very likely to repeat themselves within and across different scales of an image.

We can reconstruct a coherent high resolution image with patches correspondence between low res and downsampled images.

8

Principle : minimization

𝐺𝑘=(⟨𝐿𝑅1 ,𝐿𝑅1 ⟩ ⟨𝐿𝑅1 ,𝐿𝑅2 ⟩ … ⟨𝐿𝑅1 ,𝐿𝑅𝑘 ⟩⟨𝐿𝑅2 ,𝐿𝑅1 ⟩ ⟨𝐿𝑅2 ,𝐿𝑅2 ⟩ ⋯ ⟨𝐿𝑅2 ,𝐿𝑅𝑘 ⟩

⋮ ⋮ ⋱ ⋮⟨𝐿𝑅𝑘 ,𝐿𝑅1 ⟩ ⟨𝐿𝑅𝑘 ,𝐿𝑅2 ⟩ ⋯ ⟨𝐿𝑅𝑘 ,𝐿𝑅𝑘 ⟩)h𝑊𝑒𝑖𝑔 𝑡𝑠𝑘=

𝐺𝑘− 1  1

1𝑇𝐺𝑘−1   1

LLE (locally linear embedding) is computed with low res patches to approximate the input patch and weights obtained are used to merge high res patches.

The best K patches combination is obtained by a downsampling of the merged patch and a SSD comparison with the input LR patch.

With the knn-search, we have K couples of low res and high res patches.

9

10

Summary• Basic concepts• Principle• Quality Improvements• Complexity improvements• Results• Conclusion & remaining work

11

Quality Improvements: patch orientation

To increase the result of the knn-search, it’s possible to seek the input patch plus his transformations (transpose, symmetric X, symmetric Y and combinations).

Result is better but the knn-search is about 8 times slower.

12

Quality Improvements: textural features

standard texturalWe can improve the opportunity of find similar patches in the knn-search by using a textural version of all the patches.

In the same way, the LLE is computed with textural patches, and the mean value of the input patch is added to the result because the upsampling of the mean is itself.

13

Quality Improvements: attempts• Using a bicubic neighborhood instead of padding for the patch downsampling.

• Compute a first pass with a big value of K, and a second pass with k varying and using the neighborhood of the first pass.

• Downsampling result and compare it with the input image to detect error areas in order to recompute with better parameters.

• Continue to minimize the patch combination until it’s over a maximum error.

• Nonlinear minimization instead of LLE.

14

Summary• Basic concepts• Principle• Quality Improvements• Complexity improvements• Results• Conclusion & remaining work

15

Complexity improvements: windows search

Instead of the full phases surface, it’s possible to significantly speedup the knn-search using a window.

Results are close to a full search because corresponding patches are often beside the input patch position.

16

Complexity improvements: single channel SR

Because the Human visual system is principally sensible in luminance, it’s possible to convert an image from RGB to YUV colorspace and work only in Y channel, U and V channels are interpolated with Lanczos algorithm.

Visually, results are very close to a multi channel processing but the compute time is faster.

HR image

LR image channels

17

Complexity improvements: minimization

Histogram of best value of K

Standard minimization : K takes all value until

Optimized minimization, only small values of k are computed + max value

→ Result is very close to a full minimization with all values of K.

Number of patches

K

18

Summary• Basic concepts• Principle• Quality Improvements• Speed improvements• Results• Conclusion & remaining work

19

Results : Upsampling x4 for Oldman

LANCZOS OUR[8]ORIGINAL NEAREST NEIGHBOR

20

LANCZOS OUR ORIGINAL

21

Summary• Basic concepts• Principle• Quality Improvements• Complexity improvements• Results• Conclusion & remaining work

22

Conclusion & Remaining work

Remaining work:• CUDA implementation:

Knn-search, matrix inversion and patch combination are very parallelizable.

• Discontinuity problems with sequences: Lot of artifacts appear, especially on edges.

Advantages:• Results are excellent compared to standards interpolation methods (bicubic, Lanczos…), average

is 1.5dB better, many details appear.• It’s visually better than all other super resolution algorithms.

Drawbacks:• Very slow to compute, about 1 hour to upsample a FullHD image into UHD, but it’s 20 times

faster than the reference Matlab software.

23

References1. H. Chang, D.-Y. Yeung, and Y.Xiong, “Super-resolution through neighbor embedding,” in IEEE Comp.

Soc. Conf. Comp. Vis. Pattern Recogn., 2004, pp. 275-282.2. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, “Learning low-level vision,” Int. J. Comp. Vis., vol.

40, no. 1, pp. 83-97, 2004.3. J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE

Trans. Image Process., vol. 19, pp. 2861-2873, 2010.4. D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” in IEEE Int. Conf. Comp.

Vis., 2009, pp. 349-356.5. G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Trans.

Graph., vol. 28, no. 3, pp.1-10, 2010.6. S. Roweis and L. Saul, “Nonlinear dimensionality reduction by locally linear embedding,” Science,

vol. 290, pp. 2323-2326, 2000.7. M. Turkan, D. Thoreau, and P. Guillotel, “Optimized neighbor embeddings for single-image super-

resolution,” To be presented in ICIP 2013.8. M. Turkan, D. Thoreau, and P. Guillotel, “Self-content super-resolution for ultra-HD up-sampling,” in

Proc. European Conf. Visual Media Prod., 2012, pp. 49–58.