A Sampled Texture Prior for Image Super-ResolutionLyndsey C. Pickup, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University of Oxford

Goal: Super-resolution aims to produce a high-resolution image from a set of low-resolution images by recovering or inventing plausible high-frequency information. With more than one low-resolution image, high-frequency information can be recovered by making use of sub-pixel image displacements. Given some basic knowledge about the image subject, it should be possible to improve the super-resolution estimate by incorporating information available from similar images.

Introduction The Sample-Based Prior

There are K low-resolution images y(k), generated by y(k) = W(k)x + G(k), where G is a vector of

i.i.d. Gaussians such that G » N(0,-1), and G is the noise precision. If x has N pixels and each y(k) has M pixels, W(k) is an M£N matrix which encompasses the warping, blurring and decimation of x to produce y(k). We denote the registration and blurring parameters of y(k) by (k).

Low-resolution images (inputs)

The goal is to reconstruct the high-resolution image to the right. It is assumed that the low-resolution images have been generated from this image through a process that blurs and subsamples the high-resolution image, and adds Gaussian noise.

A few results

Maximum Likelihood Solution

The maximum likelihood images are very noisy because the problem can be ill-conditioned.

where g is an approximation to the image gradient at each pixel in four different directions (horizontal, vertical, both diagonals), and is the Huber function, defined

Since this is an image of a fairly specific type, it should be possible to do better. Rather than using a generic Huber-style prior based on image statistics, samples from similar images can be used to build a more appropriate prior.

An approach that has been shown to work well in texture synthesis is used to build the image prior. Rather than developing a parametric model, samples are taken from other images.

Small sample image patches of edge length 5 to15 pixels are taken from a training image containing similar textures to the images being super-resolved.

For each patch, neighbourhood information is stored along with a central pixel value. The patches are weighted by a 2D Gaussian so pixels near the centre carry more weight.

To evaluate this texture-based prior

•Take each pixel xi in the super-resolution image x, and find its neighbourhood R(xi).

•Find the closest matching neighbourhood in the patches sampled from the training image.

•Look up the central pixel value of that neighbourhood, LR(xi).

•Assume xi is Gaussian distributed about LR(xi) with precision T.

•This gives us:

Solving for the super-resolution image

• Start with an initial estimate of x, for instance the average image. We assume that the image registration has already been estimated accurately.

• Sample patches from an image in the same class as the super-resolution image being recovered. Store the neighbourhood and patch centre data for use with the texture-based prior.

• Use SCG to optimize the MAP version of the super-resolution model incorporating the new p(x):

where encompasses the T and G terms from the prior and data likelihood. = 0.16

= 0.64

= 0.01 = 0.04

Original imageSample texture

With a weak prior, the input data contains enough evidence for the prior to look like the ground truth, but as increases, the prior term dominates.

Maximum A Posteriori Solution

Pixel neighbourhood

Central pixel

Using the wrong texture

Text truth Text low-res

Text training image

Brick training image

Brick truth Brick low-res

Beads training image

Beads truth Beads low-res

A plot of root mean square (RMS) errors obtained from texture-based prior against those from the Huber-MAP images, where the error is calculated with respect to the ground truth images. In all cases, the texture-based prior’s image has a lower RMS error.

For each image type, nine datasets were generated, using varying numbers of inputs and different noise levels. For each dataset, the best of several runs (while varying free model parameters) is shown. Note that training images do not overlap with image sections used as ground truth.

The model

Image x. Warp, withparameters

Blur by point-spread function.

Decimate by zoom factor.

Corrupt with additive Gaussian noise.

0.5 2.5 7.5 10

5

10

20

Noise (grey levels)

Num

ber

of im

ages

0.5 2.5 7.5 10

5

10

20

Noise (grey levels)

Num

ber

of im

ages