Pixel Interpolation and Its Applications

ByDr S Patnaik

1

Inverse MappingInverse MappingLocate output image pixel grid location in output space. Then for each output pixel on the grid:

• Apply the inverse spatial transformation to determine the corresponding location in input space: (uk,vk) = T-1{(xk,yk)}.

• Using the input image pixels nearest to (uk,vk), interpolate to get an approximate value for the input image at (uk,vk).

• Use that value for the k-th output pixel.

2

3

Image InterpolationImage Interpolation• Introduction

– What is image interpolation?Model based recovery of continuous data from discrete data within a known range of abscissa.

• Interpolation Techniques– 1D zero-order, first-order, third-order– 2D = two sequential 1D– Directional(Adaptive) interpolation*

• Interpolation Applications– Digital zooming (resolution enhancement)– Image in painting (error concealment)– Geometric transformations– Warping

4

IntroductionIntroduction• What is image interpolation?

–An image f(x,y) tells us the intensity values at the integral lattice locations, i.e., when x and y are both integers

– Image interpolation refers to the “guess”of intensity values at missing locations, i.e., x and y can be arbitrary

–Note that it is just a guess (Note that all sensors have finite sampling distance)

5

Engineering MotivationsEngineering Motivations• Why do we need image interpolation?

–We want BIG images• When we see a video clip on a PC, we like to

see it in the full screen mode–We want GOOD images

• If some block of an image gets damaged during the transmission, we want to repair it

–We want COOL images• Manipulate images digitally can render fancy

artistic effects as we often see in movies

6

Scenario I: Resolution Scenario I: Resolution EnhancementEnhancement

Low-Res.

High-Res.

7

Scenario II: Image Scenario II: Image InpaintingInpainting

Non-damaged Damaged

8

Scenario III: Image Scenario III: Image WarpingWarping

You will work on image warping

9

1D Zero-order (Replication)

nf(n)

x

f(x)

10

1D First-order Interpolation (Linear)

nf(n)

x

f(x)

11

Linear Interpolation FormulaLinear Interpolation Formula

a 1-a

f(n)

f(n+1)f(n+a)

f(n+a)=(1-a)f(n)+af(n+1), 0<a<1

Heuristic: the closer to a pixel, the higher weight is assignedPrinciple: line fitting to polynomial fitting (analytical formula)

Note: when a=0.5, we simply have the average of two

12

Numerical ExamplesNumerical Examplesf(n)=[0,120,180,120,0]

f(x)=[0,60,120,150,180,150,120,60,0], x=n/2

f(x)=[0,20,40,60,80,100,120,130,140,150,160,170,180,…], x=n/6

Interpolate at 1/2-pixel

Interpolate at 1/3-pixel

13

n

x

f(n)

f(x)

Cubic spline fitting

1D Third-order Interpolation (Cubic)*

http://en.wikipedia.org/wiki/Spline_interpolation

14

Graphical InterpretationGraphical Interpretationof Interpolation at Halfof Interpolation at Half--pelpel

row column

f(m,n) g(m,n)

15

Numerical ExamplesNumerical Examplesa b

c d

a a b b

a a b b

c c d d

c c d d

zero-order

first-order

a (a+b)/2 b(a+c)/2 (a+b+c+d)/4 (b+d)/2c (c+d)/2 d

16

Numerical Examples (ConNumerical Examples (Con’’t)t)

X(m,n)row m

row m+1

Col n Col n+1

X(m,n+1)

X(m+1,n+1)X(m+1,n)

Ya 1-ab

1-b

Q: what is the interpolated value at Y?Ans.: (1-a)(1-b)X(m,n)+(1-a)bX(m+1,n)+a(1-b)X(m,n+1)+abX(m+1,n+1)

Bilinear Interpolation Bilinear Interpolation --TriangleTriangle• Given value of function at vertices oftriangle, interpolate values inside• Interpolate between corners to get two points on

sides collinear with middle point. Then interpolatebetween these

• p = a*p1 + b*p2 + c*p3, a+b+c=1

• More will be discussed during image warping

17

BiBi--cubic Interpolationcubic Interpolation• Instead of weighting by distance, d,

weight by:1-3d2+2d3

• Smooth: continuous slope• Symmetric: 1-g(x)=g(1-x)

18

Cubic interpolationCubic interpolation• Cubic Convolution Interpolation determines the grey level

value from the weighted average of the 16 closest pixels tothe specified input coordinates, and assigns that value tothe output coordinates.

• The one-dimension cubic convolution interpolation kernelis:

u(s)={ 3/2|s|3 – 5/2|s|2 + 1 if 0<= |s| < 1={ -1/2|s|3 + 5/2|s|2 –4|s|+ 2 if 1<= |s| < 2={ 0 if 2 < |s|

19

20

Limitation with Limitation with bilinear/bicubicbilinear/bicubic

• Edge blurring• Jagged artifacts

X Z

Jagged artifacts

X

Z

Edge blurring

21

Pixel Replication

low-resolutionimage (100×100)

high-resolutionimage (400×400)

22

low-resolutionimage (100×100)

Bilinear Interpolation

high-resolutionimage (400×400)

23

Bicubic Interpolation

low-resolutionimage (100×100)

high-resolutionimage (400×400)

24

EdgeEdge--Directed InterpolationDirected Interpolation(Li&Orchard(Li&Orchard’’2000)2000)

low-resolutionimage (100×100)

high-resolutionimage (400×400)

Image Image DemosaicingDemosaicing• Used to reconstruct a full color image

from the incomplete color samples

25

26

Image DemosaicingImage Demosaicing((ColorColor--FilterFilter--Array InterpolationArray Interpolation))

Bayer Pattern

Bays Pattern Bays Pattern DemosaicingDemosaicing

27

28

Error Concealment*Error Concealment*

damaged interpolated

29

Image Image InpaintingInpainting**

Image Image MosaicingMosaicing**

30

31

Geometric TransformationGeometric Transformation

In the virtual space, you can have any kind of apple you want!

MATLAB functions: griddata, interp2, maketform, imtransform

32

Basic PrincipleBasic Principle• (x,y) (x’,y’) is a geometric

transformation• We are given pixel values at (x,y)

and want to interpolate the unknown values at (x’,y’)

• Usually (x’,y’) are not integers and therefore we can use linear interpolation to guess their valuesMATLAB implementation: z’=interp2(x,y,z,x’,y’,method);

33

RotationRotation

yx

yx

cossinsincos

''

x

y

x’

y’

θ

34

MATLAB ExampleMATLAB Example

% original coordinates[x,y]=meshgrid(1:256,1:256);

z=imread('cameraman.tif');

% new coordinatesa=2;for i=1:256;for j=1:256;x1(i,j)=a*x(i,j);y1(i,j=y(i,j)/a;end;end% Do the interpolation z1=interp2(x,y,z,x1,y1,'cubic');

35

Rotation ExampleRotation Example

θ=3o

36

ScaleScale

yx

aa

yx

/100

''

a=1/2

37

Affine TransformAffine Transform

y

x

dd

yx

aaaa

yx

2221

1211

''

square parallelogram

38

Affine Transform ExampleAffine Transform Example

10

25.15.

''

yx

yx

39

ShearShear

y

x

dd

yx

syx

101

''

square parallelogram

40

Shear ExampleShear Example

10

15.01

''

yx

yx

41

Projective TransformProjective Transform

1'

87

321

yaxaayaxa

x

1'

87

654

yaxa

ayaxay

quadrilateralsquare

A B

CD

A’

B’

C’

D’

42

Projective Transform ExampleProjective Transform Example

[ 0 0; 1 0; 1 1; 0 1] [-4 2; -8 -3; -3 -5; 6 3]

43

Polar TransformPolar Transform

22 yxr

xy1tan

44

Iris Image UnwrappingIris Image Unwrapping

r

45

Use Your ImaginationUse Your Imaginationr -> sqrt(r)

http://astronomy.swin.edu.au/~pbourke/projection/imagewarp/

46

Free Form DeformationFree Form Deformation

Seung-Yong Lee et al., “Image Metamorphosis Using Snakes and Free-Form Deformations,”SIGGRAPH’1985, Pages 439-448

47

Application into Image Application into Image MetamorphosisMetamorphosis

48

Summary of Image Summary of Image InterpolationInterpolation

• A fundamental tool in digital processing of images: bridging the continuous world and the discrete world

• Wide applications from consumer electronics to biomedical imaging