image interpolation

EE465: Introduction to Digital Image Processing

1

Image InterpolationImage Interpolation• Introduction

– What is image interpolation?– Why do we need it?

• Interpolation Techniques– 1D zero-order, first-order, third-order– 2D zero-order, first-order, third-order– Directional interpolation*

• Interpolation Applications– Digital zooming (resolution enhancement)– Image inpainting (error concealment)– Geometric transformations

EE465: Introduction to Digital Image Processing

2

IntroductionIntroduction• What is image interpolation?

– An image f(x,y) tells us the intensity values at the integral lattice locations, i.e., when x and y are both integers

– Image interpolation refers to the “guess” of intensity values at missing locations, i.e., x and y can be arbitrary

– Note that it is just a guess (Note that all sensors have finite sampling distance)

EE465: Introduction to Digital Image Processing

3

Introduction (Con’t)Introduction (Con’t)• Why do we need image

interpolation?– We want BIG images

• When we see a video clip on a PC, we like to see it in the full screen mode

– We want GOOD images• If some block of an image gets damaged

during the transmission, we want to repair it

– We want COOL images• Manipulate images digitally can render fancy

artistic effects as we often see in movies

EE465: Introduction to Digital Image Processing

4

Scenario I: Resolution Scenario I: Resolution EnhancementEnhancement

Low-Res.

High-Res.

EE465: Introduction to Digital Image Processing

5

Scenario II: Image InpaintingScenario II: Image Inpainting

Non-damaged Damaged

EE465: Introduction to Digital Image Processing

6

Scenario III: Image WarpingScenario III: Image Warping

EE465: Introduction to Digital Image Processing

7

Image InterpolationImage Interpolation• Introduction

– What is image interpolation?– Why do we need it?

• Interpolation Techniques– 1D zero-order, first-order, third-order– 2D zero-order, first-order, third-order– Directional interpolation*

• Interpolation Applications– Digital zooming (resolution enhancement)– Image inpainting (error concealment)– Geometric transformations

EE465: Introduction to Digital Image Processing

8

1D Zero-order (Replication)

n

f(n)

x

f(x)

EE465: Introduction to Digital Image Processing

9

1D First-order Interpolation (Linear)

n

f(n)

x

f(x)

EE465: Introduction to Digital Image Processing

10

Linear Interpolation FormulaLinear Interpolation Formula

a 1-a

f(n)

f(n+1)f(n+a)

f(n+a)=(1-a)f(n)+af(n+1), 0<a<1

Basic idea: the closer to a pixel, the higher weight is assigned

Note: when a=0.5, we simply have the average of two

EE465: Introduction to Digital Image Processing

11

Numerical ExamplesNumerical Examples

f(n)=[0,120,180,120,0]

f(x)=[0,60,120,150,180,150,120,60,0], x=n/2

f(x)=[0,20,40,60,80,100,120,130,140,150,160,170,180,…], x=n/6

Interpolate at 1/2-pixel

Interpolate at 1/3-pixel

EE465: Introduction to Digital Image Processing

12

n

x

f(n)

f(x)

Cubic spline fitting

1D Third-order Interpolation (Cubic)

EE465: Introduction to Digital Image Processing

13

From 1D to 2D

Just like separable 2D transform (filtering) that can be implemented by two sequential 1D transforms (filters) along row and column direction respectively, 2D interpolation can be decomposed into two sequential 1D interpolations.

The ordering does not matter (row-column = column-row)

Such separable implementation is not optimal but enjoys low computational complexity

EE465: Introduction to Digital Image Processing

14

Graphical InterpretationGraphical Interpretationof Interpolation at Half-pelof Interpolation at Half-pel

row column

f(m,n) g(m,n)

EE465: Introduction to Digital Image Processing

15

Numerical ExamplesNumerical Examplesa b

c d

a a b b

a a b b

c c d d

c c d d

zero-order

first-order

a (a+b)/2b(a+c)/2 (a+b+c+d)/4 (b+d)/2c (c+d)/2 d

EE465: Introduction to Digital Image Processing

16

Numerical Examples (Con’t)Numerical Examples (Con’t)

X(m,n)row m

row m+1

Col n Col n+1

X(m,n+1)

X(m+1,n+1)X(m+1,n)

Ya 1-a

b

1-b

Q: what is the interpolated value at Y?Ans.: (1-a)(1-b)X(m,n)+(1-a)bX(m+1,n)+a(1-b)X(m,n+1)+abX(m+1,n+1)

EE465: Introduction to Digital Image Processing

17

Bicubic Interpolation*Bicubic Interpolation*

EE465: Introduction to Digital Image Processing

18

Limitation with Limitation with bilinear/bicubicbilinear/bicubic

• Edge blurring• Jagged artifacts

X Z

Jagged artifacts

X

Z

Edge blurring

EE465: Introduction to Digital Image Processing

19

Directional Interpolation*Directional Interpolation*

Step 1: interpolate the missing pixels along the diagonal

black or white?

Step 2: interpolate the other half missing pixels

a b

c d

Since |a-c|=|b-d|x

x has equal probabilityof being black or white

a

b

c

dSince |a-c|>|b-d|

x=(b+d)/2=black

x

EE465: Introduction to Digital Image Processing

20

Image InterpolationImage Interpolation• Introduction

– What is image interpolation?– Why do we need it?

• Interpolation Techniques– 1D zero-order, first-order, third-order– 2D zero-order, first-order, third-order– Directional interpolation*

• Interpolation Applications– Digital zooming (resolution enhancement)– Image inpainting (error concealment)– Geometric transformations

EE465: Introduction to Digital Image Processing

21

Pixel Replication

low-resolutionimage (100×100)

high-resolutionimage (400×400)

EE465: Introduction to Digital Image Processing

22

low-resolutionimage (100×100)

Bilinear Interpolation

high-resolutionimage (400×400)

EE465: Introduction to Digital Image Processing

23

Bicubic Interpolation

low-resolutionimage (100×100)

high-resolutionimage (400×400)

EE465: Introduction to Digital Image Processing

24

Edge-Directed InterpolationEdge-Directed Interpolation(Li&Orchard’2000)(Li&Orchard’2000)

low-resolutionimage (100×100)

high-resolutionimage (400×400)

EE465: Introduction to Digital Image Processing

25

Image DemosaicingImage Demosaicing((Color-Filter-Array InterpolationColor-Filter-Array Interpolation))

Bayer Pattern

EE465: Introduction to Digital Image Processing

26

Image ExampleImage Example

Ad-hoc CFA Interpolation Advanced CFA Interpolation

EE465: Introduction to Digital Image Processing

27

Error ConcealmentError Concealment

damaged interpolated

EE465: Introduction to Digital Image Processing

28

Image InpaintingImage Inpainting

EE465: Introduction to Digital Image Processing

29

Geometric TransformationGeometric Transformation

Widely used in computer graphics to generate special effects

MATLAB functions: griddata, interp2, maketform, imtransform

EE465: Introduction to Digital Image Processing

30

Basic PrincipleBasic Principle• (x,y) (x’,y’) is a geometric

transformation• We are given pixel values at (x,y)

and want to interpolate the unknown values at (x’,y’)

• Usually (x’,y’) are not integers and therefore we can use linear interpolation to guess their valuesMATLAB implementation: z’=interp2(x,y,z,x’,y’,method);

EE465: Introduction to Digital Image Processing

31

RotationRotation

y

x

y

x

cossin

sincos

'

'

x

y

x’

y’

θ

EE465: Introduction to Digital Image Processing

32

MATLAB ExampleMATLAB Example

% original coordinates[x,y]=meshgrid(1:256,1:256);

z=imread('cameraman.tif');

% new coordinatesa=2;for i=1:256;for j=1:256;x1(i,j)=a*x(i,j);y1(i,j=y(i,j)/a;end;end

% Do the interpolation z1=interp2(x,y,z,x1,y1,'cubic');

EE465: Introduction to Digital Image Processing

33

Rotation ExampleRotation Example

θ=3o

EE465: Introduction to Digital Image Processing

34

ScaleScale

y

x

a

a

y

x

/10

0

'

'

a=1/2

EE465: Introduction to Digital Image Processing

35

Affine TransformAffine Transform

y

x

d

d

y

x

aa

aa

y

x

2221

1211

'

'

square parallelogram

EE465: Introduction to Digital Image Processing

36

Affine Transform ExampleAffine Transform Example

1

0

25.

15.

'

'

y

x

y

x

EE465: Introduction to Digital Image Processing

37

ShearShear

y

x

d

d

y

x

sy

x

1

01

'

'

square parallelogram

EE465: Introduction to Digital Image Processing

38

Shear ExampleShear Example

1

0

15.

01

'

'

y

x

y

x

EE465: Introduction to Digital Image Processing

39

Projective TransformProjective Transform

1'

87

321

yaxa

ayaxax

1'

87

654

yaxa

ayaxay

quadrilateralsquare

A B

CD

A’

B’

C’

D’

EE465: Introduction to Digital Image Processing

40

Projective Transform ExampleProjective Transform Example

[ 0 0; 1 0; 1 1; 0 1] [-4 2; -8 -3; -3 -5; 6 3]

EE465: Introduction to Digital Image Processing

41

Polar TransformPolar Transform

22 yxr

x

y1tan

EE465: Introduction to Digital Image Processing

42

Iris Image UnwrappingIris Image Unwrapping

r

EE465: Introduction to Digital Image Processing

43

Use Your ImaginationUse Your Imaginationr -> sqrt(r)

http://astronomy.swin.edu.au/~pbourke/projection/imagewarp/

EE465: Introduction to Digital Image Processing

44

Free Form DeformationFree Form Deformation

Seung-Yong Lee et al., “Image Metamorphosis Using Snakes and Free-Form Deformations,”SIGGRAPH’1985, Pages 439-448

EE465: Introduction to Digital Image Processing

45

Application into Image Application into Image MetamorphosisMetamorphosis

EE465: Introduction to Digital Image Processing

46

Summary of Image Summary of Image InterpolationInterpolation

• A fundamental tool in digital processing of images: bridging the continuous world and the discrete world

• Wide applications from consumer electronics to biomedical imaging

• Remains a hot topic after the IT bubbles break