Why do we Need Statistical Model in the first place? Any image processing algorithm has to

work on a collection (class) of images instead of a single one

Mathematical model gives us the abstraction of common properties of the images within the same class

Model is our hypothesis and images are our observation data In physics, can F=ma explain the relationship

between force and acceleration? In image processing, can this model fit this class of images?

Introduction to Texture Synthesis Motivating applications Texture synthesis vs. image

denoising Statistical image modeling revisited

Modeling correlation/dependency Transform-domain texture synthesis Nonparametric texture synthesis Performance evaluation issue

Computer Graphics in SPORE

What is Image/Texture Model?

speech

Analysis

Synthesis

Pitch, LPCResidues …

texture

Analysis

Synthesis

P(X): parametric/nonparametric

How do we Tell the Goodness of a Model? Synthesis (in statistical language,

it is called sampling)

Hypothesizedmodel

Does the generatedsample (experimentalresult) look like the data of our interests?

A fair coin?Does the generatedsequence (experimentalresult) contain the samenumber of Heads and Tails?

Flipthe coin

Computersimulation

Discrete Random Variables (taken from EE465)

Example III: For a gray-scale image (L=256), we can use the notation p(rk), k = 0,1,…, L - 1, to denote the histogram of an image with L possible gray levels, rk, k = 0,1,…, L - 1, where p(rk) is the probability of the kth gray level (random event) occurring. The discrete random variables in this case are gray levels.

Question: What is wroning with viewing all pixels as being generated from an independent identically distributed (i.i.d.) random variable

To Understand the Problem

Theoretically, if all pixels are indeed i.i.d., then random permutation of pixels should produce another image of the same class (natural images)

Experimentally, we can write a simple MATLAB function to implement and test the impact of random permutation

Permutated image with identical histogram to lena

Random Process Random process is the foundation

for doing research in the field of communication and signal processing (that is why EE513 is the core requirement for qualified exam)

Random processes is the vector generalization of (scalar) random variables

Correlation and Dependency (N=2)

If the condition

holds, then the two random variables are said to be uncorrelated. From our earlier discussion, we know that if x and y are statistically independent, then p(x, y) = p(x)p(y), in which case we write

Thus, we see that if two random variables are statistically independent then they are also uncorrelated. The converse of this statement is not true in general.

Covariance of two Random Variables

The moment µ11

is called the covariance of x and y.

Recall: How to Calculate E(XY)?

… …

… …

X

Y

N

nnnYX

NXYE

1

1)(Empirical solution:

Note: When Y=X, we are getting autocorrelation

Stationary Process*

T T+K

P(X1,…,XN)=P(XK+1,…,XK+N) for any K,N (all statistics is time invariant)

N N

space/time location

order of statistics

Gaussian Process

With mean vector m and covariance matrix CFor convenience, we often assume zero mean (if it is nonzero mean, we can subtract the mean)

The question is: is the distribution of observation data Gaussian or not?

For Gaussian process, it is stationary as long asits first and second order statistics are time-invariant

The Curse of Dimensionality Even for a small-size image such as 64-by-

64, we need to model it by a random process in 4096-dimensional space (R4096) whose covariance matrix is sized by 4096-by-4096

Curse of dimensionality was pointed out by E. Bellman in 1960s; but even computing resource today cannot handle the brute-force search of nearest-neighbor search in relatively high-dimensional space.

Markovian Assumption

Andrei A. Markov1856 - 1922

Pafnuty L. Chebyshev1821 - 1894

Andrey N. Kolmogorov1903 - 1987

A Simple Idea

The future is determined by the present but is independent of the past

Note that stationarity and Markovianity are two “orthogonal” perspectives of imposing constraintsto random processes

Markov Process),...,|()|()(),...,( 111211 XXXPXXPXPXXP MMM

),...,|(),...,|( 111 Nkkkkk XXXPXXXP

N-th order Markovian

N past samples

Parametric or non-parametric characterization

Autoregressive (AR) Model Parametric model (Linear

Prediction)

An infinite impulse response (IIR) filter

N

nknknk wXaX

1

N

n

nn zazAzAzH

zWzHzX

1

1)(),(/1)(

),()()(

z-transform

Example: AR(1)

iik

i

kkkkkk

wa

wawXawaXX

...

122

1

Autocorrelation function

...2,1,0,)( kakr k

a=0.9

k

r(k)

Yule-Walker Equation

N

nknknk wXaX

1

N

nlknknlkk XXEaXXE

1

)()(

N

k

a

a

a

rrNrrr

rrNrrr

Nr

kr

r

1

)0()1()1()1()0(

)0()1()1(......)1()0(

)(

)(

)1(

Covariance C

Wiener FilteringIn practice, we do not know autocorrelation functions but only observation data X1,…,XM

),...,()( 11 1

2N

M

k

N

nnknk aafXaXMSE

Approach 1: empirically estimate r(k) from X1,…,XM

Approach 2: Formulate the minimization problem of

Niaaaf iN 1,0/),...,( 1

Exercise: you can verify they end up with the same results

Least-Square Estimation

N

nknknk wXaX

1

N

k

a

a

a

NMXNMXMXNMXMX

NXXX

MX

kX

X

1

)()1()1()1(......)2(

......)1(......)1()0(

)(

)(

)1(

M equations, N unknown variables11 NNMM aCy

Least-Square Estimation (Con’d)

11 NNMM aCy

aCCyC TT

)()( 1 yCCCa TT

If you write it out, it is exactly the empirical wayof estimating autocorrelation functions – nowyou have got the third approach

Rxx rx

From 1D to 2D

Xm,n1

23 4

5Xm,n1

2 3 4

5

6

Causal neighborhood Noncausal neighborhood

678

Causality of neighborhood depends on differentapplications (e.g., coding vs. synthesis)

Experimental Justifications

original

Analysis

Synthesisrandomexcitation

AR modelparameters

Failure Example (I)

Analysisand

Synthesis

N=8,M=4096Another way to look at it: if X and Y are two imagesof disks, will (X+Y)/2 produce another disk image?

Failure Example (II)

Analysisand

Synthesis

Note that the failure reason of this example is different from the last example (N is not large enough)

N=8,M=4096

Summary for AR Modeling We start from AR models because they are

relatively simple and well understood (not just for images but also for speech coding, stock market prediction …)

AR model parameters are related to the second-order statistics by Yule-Walker equation

AR model is equivalent to IIR filtering (linear prediction decorrelates the input signal)

Improvement over AR Model Doubly stochastic process*

In stationary Gaussian process, second-order statistics are time/spatial invariance

In doubly stochastic process, second-order statistics (e.g., covariance) are modeled by another random process with hidden variables

Windowing technique To estimate spatially varying statistics

Why do We need Windows? Nothing to do with Microsoft All images have finite dimensions – they

can be viewed as the “windowed” version of natural scenes

Any empirical estimation of statistical attributes (e.g., mean, variance) is based on the assumption that all N samples observe the same distribution However, how do we know this assumption

is satisfied?

1D Rectangular Window

X(n)

n

W=(2T+1)

Tnk

Tnkkn X

TX

)12(1

2D Rectangular Window

W=(2T+1)

W=(2T+1)

Loosely speaking, parameterestimation from a localizedwindow is a compromisedsolution to handle spatiallyvarying statistics

Such idea is common toother types of non-stationarysignals too (e.g., short-time speech processing)

ExampleAs window slidesthough the image,we will observe thatAR model parametersvary from locationto location

A

B

C

Q: AR coefficientsat B and C differfrom those at A butfor different reasons,Why?

What is Next? Apply linear transformations

A detour of wavelet transforms Wavelet-space statistical models Application into texture synthesis

From parametric to nonparametric Patch-based nonparametric models Texture synthesis examples Application into image inpainting