numerical quadrature for high-dimensional integrals lászló szirmay-kalos

Numerical quadrature for Numerical quadrature for high-dimensional integralshigh-dimensional integrals

László Szirmay-Kalos

Brick ruleBrick rule

f (z) dz f (zm )

z = 1/M

f(zm )0 1 z = 1/M

f (z) dz 1/M

f(zm )

Equally spaced abscissa:uniform grid

Error analysis of the brick ruleError analysis of the brick rule

0 1

f (z) dz 1/M

f(zm )

fError = f/2/M·1/M·M=f/2/M=O(1/M)

Trapezoidal ruleTrapezoidal rule

f (z) dz

f (zm )+ f (zm+1 ))/2 z = 1/M f(zm) w(zm)

w(zm) =0 1

f (z) dz 1/M f(zm )

w(zm)

1 if 1 < m < M1/2 if m = 1 or m = M

Error = O(1/M)

Brick rule in higher dimensionsBrick rule in higher dimensions

f (x,y) dxdy 1/n j

f (x,yj) dx 1/n2 i jf (xi,yj) = 1/M f(zm )

n points

[0,1]2f (z) dz 1/M f(zm )

M=n2 zm =(xi,yj)

Error analysis for higher Error analysis for higher dimensionsdimensions

n2 samples

f (x,y) dxdy = 1/n j

f (x,yj) dx fy /2/n 1/n j 1/n (if (xi,yj) fx /2/n ) fy /2/n = 1/M f(zm ) (fx +fy ) /2 · M-0.5

Classical rules in Classical rules in DD dimensions dimensions

Error: f/2 M- 1/D = O(M -1/D)Required samples for the

same accuracy: O( (f/error)D )Exponential core

– big gaps between rows and columns

Monte-Carlo integrationMonte-Carlo integrationTrace back the integration to an expected value problem

[0,1]Df (z) dz= [0,1]Df (z) 1 dz =[0,1]Df (z) p(z) dz = E[f (z) ]

p(z)= 1

f (z)

E[f (z) ]variance: D2 [f (z) ]

Real line

pdf of f (z)

Expected value estimation by Expected value estimation by averagesaverages

f (z)

E[f (z) ] f *=1/M f(zm )E[f *]=E[1/M·f(zm)]= 1/M·E[f(zm)]=E[f (z) ]

D2[f *]= D2[1/M ·f(zm)]= 1/M 2 ·D2[f(zm)]=if samples are independent! = 1/M 2·M·D2[f(zm)]= 1/M·D2[f(z)]

Distribution of the averageDistribution of the average

E[f (z) ] pdf of f *=1/M f(zm )

M=10M=40

M=160

Central limit theorem: normal distribution

Three 9s law: Pr{| f *- E[f *] | < 3D[f *]} > 0.999

Probabilistic error bound (0.999 confidence): |1/M f(zm ) - [0,1]Df (z) dz | < 3D[f ]/M

Classical versus Monte-Carlo Classical versus Monte-Carlo quadraturesquadratures

Classical (brick rule): f/2 M-1/D

f : variation of the integrand– D dimension of the domain

Monte-Carlo: 3D[f ]/M– 3D[f ]: standard deviation (square root of the

variance) of the integrand– Independent of the dimension of the

domain of the integrand!!!

Importance samplingImportance samplingSelect samples using non-uniform densities

f (z) dz= f (z)/p(z) ·p(z) dz = E[f (z)/p(z)] 1/M f(zm)/p(zm)

f (z)/p(z)

E[f (z)/p(z)] f *=1/M f(zm

)/p(zm)

Optimal probability densityOptimal probability density

Variance of f (z)/p(z) should be small Optimal case f (z)/p(z) is constant, variance is

zero p(z) f (z) and p(z) dz = 1 p(z) = f (z) /

f (z) dz Optimal selection is impossible since it needs the

integral Practice: where f is large p is large

Numerical integrationNumerical integration

f (z) dz 1/M f (zi )

Good sample points? z1, z2, ..., zn

Uniform (equidistribution) sequences: assymptotically correct result for any Riemann integrable function

Uniform sequence: Uniform sequence: necessary requirementnecessary requirement

Let f be a brick at the center

1

f

A

f (z) dz = V (A)

V(A) m(A) 1/M f (zi) = m (A)/M

lim m (A)/M = V(A)

10

DiscrepancyDiscrepancy Difference between the

relative number of points and the relative size of the area

D*(z1, z2,..., zn) = max | m(A)/M - V(A) |

V m points

M points

Uniform sequences Uniform sequences lim lim DD((zz11,,......,, zznn) =0) =0

Necessary requirement: to integrate a step function discrepancy should converge to 0

It is also sufficient

+=

Other definition of Other definition of uniformnessuniformness

Scalar series: z1, z2,..., zn.. in [0,1]

1-uniform: P(u < zn < v)=(v-u)u v 10

1-uniform sequences1-uniform sequences Regular grid: D = 1/2 M random series: D loglogM/2M Multiples of irrational numbers modulo 1

– e.g.: {i 2 } Halton (van der Corput) sequence in base b

– D b2/(4(b+1)logb) logM/M if b is even– D (b-1)/(4 logb) logM/M if b is odd

Discrepancy of a random seriesDiscrepancy of a random series

Theorem of large numbers (theorem of iterated logarithm):1 , 2 ,…, M are independent r.v.

with mean E and variance :

Pr( limsup | i/M - E | 2 loglogM/M ) = 1

x is uniformly distributed:i (x) = 1 if i(x) < A and 0 otherwise:

i/M = m(A)/ M, E = A, 2 = A-A2 < 1/4

Pr( limsup | m(A)/ M - A | loglogM/2M ) = 1

Halton (Van der Corput) seq: Halton (Van der Corput) seq: HHii is the radical inverse of is the radical inverse of iii binary form of i radical inverse Hi 0 0 0.0 01 1 0.1 0.52 10 0.01 0.253 11 0.11 0.754 100 0.001 0.1255 101 0.101 0.6256 110 0.011 0.375

0 12 34

Uniformness of the Halton Uniformness of the Halton sequencesequence

i binary form of i radical inverse Hi 0 0 0.000 01 1 0.100 0.52 10 0.010 0.253 11 0.110 0.754 100 0.001 0.1255 101 0.101 0.6256 110 0.011 0.375

All fine enough interval decompositions:each interval will contain a sample before a second sample is placed

Progam: generation of the Progam: generation of the Halton sequenceHalton sequence

class Halton { double value, inv_base;

Number( long i, int base ) { double f = inv_base = 1.0/base; value = 0.0; while ( i > 0 ) { value += f * (double)(i % base); i /= base; f *= inv_base; }}

Incemental generation of the Incemental generation of the Halton sequenceHalton sequence

void Next( ) { double r = 1.0 - value - 0.0000000001; if (inv_base < r) value += inv_base; else { double h = inv_base, hh; do { hh = h; h *= inv_base; } while ( h >= r ); value += hh + h - 1.0;}

2,3,…2,3,…-uniform sequences-uniform sequences

2-uniform:

P(u1 < zn < v1, u2 < zn+1 < v2) = (v1-u1) (v2-u2)

(zn ,zn+1 )

-uniform sequences-uniform sequences

Random series of independent samples– P(u1<zn< v1, u2< zn+1< v2) = P(u1<zn<

v1) P( u2< zn+1< v2)

Franklin theorem: with probability 1:– fractional part of n -uniform

is a transcendent number (e.g. )

Sample points for integral quadratureSample points for integral quadrature

1D integral: 1-uniform sequence 2D integral:

– 2-uniform sequence– 2 independent 1-uniform sequences

d-D integral– d-uniform sequence– d independent 1-uniform sequences

Independence of 1-uniform sequences: Independence of 1-uniform sequences:

pp1, 1, pp2 2 are relative primesare relative primes

p1n columns:

samples uniform with period p1

n

p2m rows:

samples uniform with period p2

m

p1n p2

m cells: samples uniform with period SCM(p1

n, p2

m)SCM= smallest common multiple

Multidimensional sequencesMultidimensional sequences

Regular grid Halton with prime base numbers

– (H2(i) , H3(i), H5(i), H7(i), H11(i), …)

Weyl sequence: Pk is the kth prime– (iP1, iP2, iP3, iP4, iP5, …)

Low discrepancy sequencesLow discrepancy sequences Definition:

– Discrepancy: O(logD M/M ) =O(M-(1-)) Examples

– M is not known in advance: Multidimensional Halton sequence: O(logD M/M )

– M is known in advance: Hammersley sequence: O(logD-1 M/M )

Optimal?– O(1/M ) is impossible in D > 1 dimensions

OO(log(logD D M/M M/M ) =) =OO((M M --((1-1-))) ?) ?

O(logD M/M ) dominated by c logD M/M – different low-discrepancy sequences have

significantly different c If M is large, then logD M < M

– logD M/M < M/M = M -(1-) – Cheat!: D=10, M = 10100

logD M = 10010 M= 1010

Error of the integrandError of the integrand

How uniformly are the sample point distributed?

How intensively the integrand changes

Variation of the function: VitaliVariation of the function: Vitali

f Vv

f

Vv=limsup fxi+1 fxi

01

| df (u)/du | du

xi

Vitali Variation in higher dimensionsVitali Variation in higher dimensions

f

Vv=limsup fxi+1,yi+1fxi+1, yifxi, yi +1fxi, yi

| 2f (u,v)/ u v | du dv

Zero if f is constant along 1 axis

f

Hardy-Krause variation of Hardy-Krause variation of discontinuous functionsdiscontinuous functions

f f

Variation:

Variation:

fxi+1,yi+1fxi+1, yifxi, yi +1fxi, yi

Koksma-Hlawka inequalityKoksma-Hlawka inequality error( error( f f ) < V) < VHKHK • •DD((zz11,, zz22,,......,, zznn))

1. Express:f (z) from its derivative

e(u) e(u-z)

zu u

f(1)-f(z) = z1

f ’(u)du f(z)=f(1)- z1

f ’(u)du

f(z) = f(1)- 01

f ’(u) e(u-z) du

Express Express 1/1/M M ff((zzii ))1/M f (zi ) =

= f(1)- 01

f ’(u) ·1/M e(u-zi ) du =

= f(1)- 01

f ’(u) · m(u) /M du

ExpressExpress 001 1 ff((zz)d)dz z

using partial integrationusing partial integration

01 f (u) · 1 du = f (u) · u|0

1 -

0

1 f ’(u) · u du =

= f(1)- 01

f ’(u) · u du

ab’= ab - a’b

ExpressExpress

||1/1/M M ff((zzii ) -) - 0011ff((zz)d)dz| z|

| 1/M f (zi ) - 01 f (z)dz| =

= | 01

f ’(u) · (m(u)/M - u) du |

= 01 | f ’(u) · (m(u)/M - u)| du

= 01 | f ’(u)| du · maxu| (m(u)/M - u) | =

= VHK · D(z1, z2,..., zn)

upperbound

Importance sampling in Importance sampling in quasi-Monte-Carlo integrationquasi-Monte-Carlo integration

Integration by variable transformation: z = T(y)

f (z) dz = f (T(y)) | dT(y)/dy | dy

p(y) = |dT (y)/dy|T

y

z

Optimal selectionOptimal selection

Variation of the integrand is 0:f (T(y)) ·| dT(y)/dy | = const f (z) ·| 1/ (dT-1(z)/dz) | = const

y = T-1(z) = z f (u) du /const

Since y is in [0,1]: T-1(zmax) = f (u) du /const = 1const = f (u) du

z = T(y) = (inverse of z f (u) du/ f (u) du) (y)

Comparing to MC importance Comparing to MC importance samplingsampling

– f (z) dz = f (z)/p(z)], p(z) f (z)– 1. normalization: p(z) = f (z)/ f (u) du – 2. probability distributions

P(z)= zp(u) du– 3. Generation of uniform random variable r.– 6. Find sample z by transforming r by the inverse

probability distribution: z = P-1(r)

MC versus QMC?MC versus QMC?

What can we expect from quasi-Monte Carlo quadrature if the integrand is of infinite variation?

Initial behavior of quasi-Monte Carlo– 100, 1000 samples per pixel in computer

graphics– They are assymptotically uniform.

QMC for integrands of QMC for integrands of unbounded variationunbounded variation

N

N

|Dom|=l / N

Length ofdiscontinuity

l

Number of samples in discontinuityM = l N

Decomposition of the Decomposition of the integrandintegrand

f s d= +

integrand finite variation part discontinuity

f

Error of the quadratureError of the quadrature

error ( f ) error( s) + error(d )QMC MC

VHK•D(z1,z2,...,zn) |Dom| • 3 f / M =3 f l • N -3/4

In d dimensions: 3 f l • N -(d+1)/2d

Applicability of QMCApplicability of QMC

QMC is better than MC in lower dimensions– infinite variation– initial behaviour

(n > base = d-th prime number)

numerical quadrature for high-dimensional integrals lászló szirmay-kalos

Documents