1

Pseudorandom Number Pseudorandom Number Generators and the Metropolis Generators and the Metropolis

AlgorithmAlgorithm

Jane RenJane Ren

2

OutlineOutline• Introduction to random number Introduction to random number

generatorsgenerators• The Marsaglia random number The Marsaglia random number

generatorgenerator• SPRNG (A Scalable Library for SPRNG (A Scalable Library for

Pseudorandom Number Generation)Pseudorandom Number Generation)• Textbook assignment 3.2.3 with Textbook assignment 3.2.3 with

Marsaglia random numbersMarsaglia random numbers• Assignment 3.2.3 with random Assignment 3.2.3 with random

numbers from another type of numbers from another type of random number generator.random number generator.

• Test resultsTest results• ConclusionsConclusions

3

Properties of Good Random Properties of Good Random Number GeneratorsNumber Generators

• RandomnessRandomness• Uniformity Uniformity • Computational EfficiencyComputational Efficiency• Long period lengthLong period length• UnpredictabilityUnpredictability• RepeatabilityRepeatability• PortabilityPortability• HomogeneityHomogeneity• Good random numbers are not Good random numbers are not

easy to findeasy to find

4

The Marsaglia Random Number The Marsaglia Random Number GeneratorGenerator• The programs in our textbook The programs in our textbook

use the Marsaglia generator.use the Marsaglia generator.• A combinations of two A combinations of two

generatorsgenerators• Lagged Fibonacci seriesLagged Fibonacci series

– IInn = I = In-r n-r – I– In-sn-s mod 2 mod 22424, r = 97, s = 33, r = 97, s = 33

• Arithmetic SeriesArithmetic Series– I – k, I – 2k, I – 3k…, mod (2I – k, I – 2k, I – 3k…, mod (22424 - 3), k - 3), k

= 7654321= 7654321

• The final stepThe final step– Take xTake xnn from a lagged Fibonacci from a lagged Fibonacci

seriesseries– Take yTake ynn from a n arithmetic series from a n arithmetic series– If x >= y, the final number is x – yIf x >= y, the final number is x – y– Else the final number is x – y + 1Else the final number is x – y + 1

5

Sample Output from the Marsaglia GeneratorSample Output from the Marsaglia Generator

• Assignment a0102_02 with seed1 = Assignment a0102_02 with seed1 = 1 and seed2 = 6.1 and seed2 = 6.

• Run 1Run 1RANMAR INITIALIZED.RANMAR INITIALIZED.idat= 1, xr= 0.894103587idat= 1, xr= 0.894103587idat= 2, xr= 0.034489155idat= 2, xr= 0.034489155idat= 3, xr= 0.330096781idat= 3, xr= 0.330096781idat= 4, xr= 0.613509536idat= 4, xr= 0.613509536extra xr= 0.710489452extra xr= 0.710489452

• Run 2Run 2 MARSAGLIA CONTINUATION.MARSAGLIA CONTINUATION.idat= 1, xr= 0.710489452idat= 1, xr= 0.710489452idat= 2, xr= 0.429685593idat= 2, xr= 0.429685593idat= 3, xr= 0.142630935idat= 3, xr= 0.142630935idat= 4, xr= 0.369295061idat= 4, xr= 0.369295061extra xr= 0.894588530extra xr= 0.894588530

6

Some Advantages of he Some Advantages of he Marsaglia GeneratorMarsaglia Generator• The lagged Fibonacci series has The lagged Fibonacci series has

a period of (2a period of (22424 – 1) * 2 – 1) * 29494 ≈≈22110110. . This is very good for a This is very good for a pseudorandom number pseudorandom number generator.generator.

• The Marsaglia random number The Marsaglia random number generator is portable because generator is portable because it uses the IEEE standard it uses the IEEE standard representation of 32 bit representation of 32 bit floating point numbers.floating point numbers.

7

Some Drawbacks of the Marsaglia GeneratorSome Drawbacks of the Marsaglia Generator• Low discretization step size 2Low discretization step size 2-24-24

– We have already seen this problem from We have already seen this problem from assignment a0204_03.assignment a0204_03.

• Not as fast as some other generators, such as Not as fast as some other generators, such as the congruential generator.the congruential generator.

• It fails the birthday spacing test.It fails the birthday spacing test.– The birthday spacing test chooses m birthdays in a The birthday spacing test chooses m birthdays in a

year of n days. If a certain day appeared more than year of n days. If a certain day appeared more than once, then that value should be asymptotically once, then that value should be asymptotically Poisson distributed with mean m**3/(4n). Poisson distributed with mean m**3/(4n).

8

3.2.3 Assignment for section 3.2 Page 3.2.3 Assignment for section 3.2 Page 152152• a0302_01a0302_01• ““Write a Metropolis program to Write a Metropolis program to

generate the normal distribution generate the normal distribution through the Markov process through the Markov process

x x x’ = x + 2a(x x’ = x + 2a(xrr – 0.5) – 0.5) where xwhere xr r is a uniformly distributed is a uniformly distributed

random number in the range [0,1) random number in the range [0,1) and a is a real constant. Use a = 3.0 and a is a real constant. Use a = 3.0 and the initial value x = 0.0 to test and the initial value x = 0.0 to test your program. (i) Generate a chain your program. (i) Generate a chain of 2000 events. Plot the empirical of 2000 events. Plot the empirical peaked distribution function for peaked distribution function for these event in comparison with the these event in comparison with the exact peaked distribution function. exact peaked distribution function. Monitor also the acceptance rate. (ii) Monitor also the acceptance rate. (ii) Repeat the above for a chain of Repeat the above for a chain of 20000 events.”20000 events.”

9

Assignment a0302_01Assignment a0302_01

• Output for a chain of 20 000 eventsOutput for a chain of 20 000 eventsRANMAR INITIALIZED.RANMAR INITIALIZED.

Metropolis Generation of Gaussian random numbersMetropolis Generation of Gaussian random numbers

Ndat=20000, a=3.0000Ndat=20000, a=3.0000

Acceptance rate found 0.491700Acceptance rate found 0.491700

• CPU time for running the Metropolis Program CPU time for running the Metropolis Program with Marsaglia = 0.126u (average of 10 runs)with Marsaglia = 0.126u (average of 10 runs)

• Comparison between the empirical peaked distribution Comparison between the empirical peaked distribution and the exact peaked normal distributionand the exact peaked normal distribution

10

Scalable Library for Scalable Library for Pseudorandom Number Pseudorandom Number Generation (SPRNG)Generation (SPRNG)• Software package released under Software package released under

the GNU General Public License.the GNU General Public License.• There are 6 kinds of random There are 6 kinds of random

number generators in SPRNGnumber generators in SPRNG1.1. Combined Multiple Recursive GeneratorCombined Multiple Recursive Generator

2.2. 48 Bit Linear Congruential Generator 48 Bit Linear Congruential Generator with Prime Addendwith Prime Addend

3.3. 64 Bit Linear Congruential Generator 64 Bit Linear Congruential Generator with Prime Addendwith Prime Addend

4.4. Modified Lagged Fibonacci GeneratorModified Lagged Fibonacci Generator

5.5. Multiplicative Lagged Fibonacci Multiplicative Lagged Fibonacci GeneratorGenerator

6.6. Prime Modulus Linear Congruential Prime Modulus Linear Congruential GeneratorGenerator

* Mersenne Twister Generator (will be * Mersenne Twister Generator (will be added)added)

• Link Link http://sprng.cs.fsu.edu/

11

Why SPRNG?Why SPRNG?• The random number generators in The random number generators in

SPRNG are considered the top SPRNG are considered the top generators.generators.

• The generators have never failed the The generators have never failed the standard tests.standard tests.– Including some of the largest random Including some of the largest random

number tests with up to 10**13number tests with up to 10**13• The tests used were collisions, coupon, The tests used were collisions, coupon,

equidistribution, gap, maximum of t, equidistribution, gap, maximum of t, permutations, poker, runs up, serial, sum permutations, poker, runs up, serial, sum of distributions, Ising: Metropolis, Ising: of distributions, Ising: Metropolis, Ising: Wolff, and Random Walk.Wolff, and Random Walk.

• What if one of the SPRNG What if one of the SPRNG generators was used for assignment generators was used for assignment a0302_01?a0302_01?

• All of the random number generators All of the random number generators passed the empirical tests, so which passed the empirical tests, so which one do we want to use?one do we want to use?

12

Descriptions of Some of the Descriptions of Some of the Tests SPRNG Generators passedTests SPRNG Generators passed• Permutation TestPermutation Test

– Divide the random sequence into Divide the random sequence into subsequences of a certain length.subsequences of a certain length.

– Rank each value in the subsequence by Rank each value in the subsequence by magnitude.magnitude.

– Count the frequency each permutation Count the frequency each permutation occurs.occurs.

– Conduct a Chi-Square test with these Conduct a Chi-Square test with these counts.counts.

– Compare the results.Compare the results.

• Collision TestCollision Test– Measure the number of times the number Measure the number of times the number

falls at a position that already had a falls at a position that already had a number in it.number in it.

• Coupon Collector’s TestCoupon Collector’s Test– Compare actual period with expected Compare actual period with expected

periodperiod

13

The GeneratorsThe Generators1.1. Combined Multiple Recursive Combined Multiple Recursive

GeneratorGenerator• Period Period ≈ 2≈ 2219219

• Fast because it is a linear congruential Fast because it is a linear congruential generatorgenerator

2.2. 48 Bit Linear Congruential Generator 48 Bit Linear Congruential Generator with Prime Addendwith Prime Addend• Period Period = 2= 24848 • Number of distinct streams is of the Number of distinct streams is of the

order of order of 221919 • Fast because it is a linear congruential Fast because it is a linear congruential

generatorgenerator

3.3. 64 Bit Linear Congruential Generator 64 Bit Linear Congruential Generator with Prime Addendwith Prime Addend• Period Period = 2= 26464 • Number of distinct streams is of the Number of distinct streams is of the

order of order of 101088 • Fast because it is a linear congruential Fast because it is a linear congruential

generatorgenerator

14

The GeneratorsThe Generators4. 4. Modified Lagged Fibonacci Modified Lagged Fibonacci

GeneratorGenerator• Period Period = 2= 23131(2(2l l – 1), where – 1), where ll is the lag. is the lag.

• The default period is 2The default period is 213101310

• Number of distinct streams is of the Number of distinct streams is of the order of order of 2231(31(ll – 1)-1– 1)-1 , default = , default = 2213101310

5.5. Multiplicative Lagged Fibonacci Multiplicative Lagged Fibonacci GeneratorGenerator• Period Period = 2= 26161(2(2l l – 1), default = 2– 1), default = 28181 • Number of distinct streams is of the Number of distinct streams is of the

order of order of 2263(63(ll – 1)-1– 1)-1 , default = , default = 228181

6.6. Prime Modulus Linear Congruential Prime Modulus Linear Congruential GeneratorGenerator• Period Period = 2= 26464 - 2 - 2• Number of distinct streams Number of distinct streams ≈≈ 225858 • Fast because it is a linear congruential Fast because it is a linear congruential

generatorgenerator

15

Is there a better generator?Is there a better generator?• The current 6 random number The current 6 random number

generators in SPRNG work generators in SPRNG work sufficiently well.sufficiently well.

• The Mersenne Twister The Mersenne Twister generator MT19937 is not a generator MT19937 is not a part of SPRNG yet, but it is part of SPRNG yet, but it is being added to SPRNG.being added to SPRNG.

• I chose to use Mersenne I chose to use Mersenne Twister as a substitute for the Twister as a substitute for the Marsaglia generator for Marsaglia generator for a0302_01. a0302_01.

16

Why Mersenne Twister?Why Mersenne Twister?• Astronomical period of Astronomical period of 221993719937 – 1 – 1

– No other generator has achieved this!No other generator has achieved this!– Marsaglia has 2Marsaglia has 2110110

• 623 dimensional equidistribution with up to 623 dimensional equidistribution with up to 32 bit accuracy in a working area of only 32 bit accuracy in a working area of only 624 words.624 words.– No other generator has achieved this!No other generator has achieved this!

• Can be employed in practical important Can be employed in practical important simulations because it is based on a good simulations because it is based on a good definition of randomness. definition of randomness. – Many other generators are only random Many other generators are only random

in a particular area.in a particular area.• Performance exceeds that of integer-large-Performance exceeds that of integer-large-

modulus generators.modulus generators.• Efficient random number generation.Efficient random number generation.• Passed tests many stringent tests, Passed tests many stringent tests,

including the diehard tests (invented by including the diehard tests (invented by Marsaglia) and tests on the higher Marsaglia) and tests on the higher dimensional uniformity, including the dimensional uniformity, including the spectral test and the k-distribution test. spectral test and the k-distribution test.

• Most suitable for Monte Carlo because it Most suitable for Monte Carlo because it emphasizes on the most significant bits.emphasizes on the most significant bits.

17

Mersenne Twister – The AlgorithmMersenne Twister – The Algorithm• Based on the recurrence (mod 2)Based on the recurrence (mod 2)

• l, s, t, and u are integers, b and c are bit masks of l, s, t, and u are integers, b and c are bit masks of word size, x[0:n-1] is an array of n unsigned integers word size, x[0:n-1] is an array of n unsigned integers of word size, and u, ll, and a are unsigned constant of word size, and u, ll, and a are unsigned constant integers of word size.integers of word size.

18

Tempering MatrixTempering Matrix• The Mersenne Twister uses a tempering The Mersenne Twister uses a tempering

matrix, so that multiplication by the matrix, so that multiplication by the tempering matrix is very efficient and tempering matrix is very efficient and fast.fast.– Example of a tempering MatrixExample of a tempering Matrix

• Let x = (xLet x = (xw-1w-1, x, xw-2w-2, x, xw-3w-3,…, x,…, x00) )

a = (aa = (aw-1w-1, a, aw-2w-2, a, aw-3w-3,…, a,…, a00))

• xA can be calculated with only binary xA can be calculated with only binary operations.operations.

– xA = shiftright(x) xA = shiftright(x) if xif x00 = 0 = 0

– xA = shiftright(x) XOR axA = shiftright(x) XOR a if xif x00 = 1 = 1

19

Mersenne Twister – How does It Mersenne Twister – How does It Work?Work?• Step 0Step 0

– Create a mask for the upper bits and another Create a mask for the upper bits and another one for the lower bits. one for the lower bits.

• Step 1Step 1

– The x array is initialized with a seed The x array is initialized with a seed

• Step 2Step 2

– y is the upper bits of x[i] concatenated with y is the upper bits of x[i] concatenated with the lower bits of x[i+1].the lower bits of x[i+1].

20

Mersenne Twister – How does It Mersenne Twister – How does It Work?Work?• Step 3Step 3

– y * A.y * A.– A is chosen carefully so it can be easily A is chosen carefully so it can be easily

calculated with right shift and exor.calculated with right shift and exor.• Step 4Step 4

– Multiply with T (tempering matrix) for better Multiply with T (tempering matrix) for better equidistribution and good acurracy.equidistribution and good acurracy.

• Steps 5 & 6Steps 5 & 6• Increment Increment ii by 1 and repeat the by 1 and repeat the

processprocess

21

Mersenne TwisterMersenne Twister• We use an improved version of We use an improved version of

the Mersenne Twister from it’s the Mersenne Twister from it’s original authors (released original authors (released 2/2002).2/2002).

• Some disadvantages of the Some disadvantages of the original versionoriginal version– Most significant bits only affect Most significant bits only affect

most significant bits of the array most significant bits of the array state.state.

– Non-overlapping subsequences on Non-overlapping subsequences on successive runs is not possible.successive runs is not possible.

– Not as fast as the improved version.Not as fast as the improved version.

22

Sample Output from the Mersenne Sample Output from the Mersenne TwisterTwisterRun 1Run 1

MERSENNE TWISTER INITIALIZED.MERSENNE TWISTER INITIALIZED.idat= 1, xr= 0.814723692 idat= 1, xr= 0.814723692 idat= 2, xr= 0.135477004 idat= 2, xr= 0.135477004 idat= 3, xr= 0.905791934 idat= 3, xr= 0.905791934 idat= 4, xr= 0.835008590 idat= 4, xr= 0.835008590 extra xr= 0.126986812extra xr= 0.126986812

• Run 2Run 2MERSENNE TWISTER MERSENNE TWISTER

CONTINUATION.CONTINUATION.idat= 0, xr= 0.126986812idat= 0, xr= 0.126986812 idat= 1, xr= 0.968867771 idat= 1, xr= 0.968867771 idat= 2, xr= 0.913375856 idat= 2, xr= 0.913375856 idat= 3, xr= 0.221034043 idat= 3, xr= 0.221034043 extra xr= 0.632359250extra xr= 0.632359250

23

Mersenne Twister in a302_01Mersenne Twister in a302_01

• Output for a chain of 20 000 Output for a chain of 20 000 eventseventsRANMAR INITIALIZED.RANMAR INITIALIZED.

Mersenne Twister Generation of Mersenne Twister Generation of Gaussian random numbersGaussian random numbers

Ndat=20000, a=3.0000Ndat=20000, a=3.0000

Acceptance rate found 0.497850Acceptance rate found 0.497850

• The acceptance rate is The acceptance rate is 0.4917000 with the Marsaglia 0.4917000 with the Marsaglia generatorgenerator

• CPU Time for the Metropolis CPU Time for the Metropolis Program with Mersenne Twister Program with Mersenne Twister = 0.124 s (Average of 10 runs)= 0.124 s (Average of 10 runs)– Marsaglia = 0.126 sMarsaglia = 0.126 s

24

Mersenne Twister in a302_01Mersenne Twister in a302_01

• Comparison between the Comparison between the empirical peaked distribution empirical peaked distribution and the exact peaked normal and the exact peaked normal distributiondistribution

25

Results from Marsaglia and Mersenne Twister Results from Marsaglia and Mersenne Twister CombinedCombined

26

Mersenne Twister and Marsaglia Execution Mersenne Twister and Marsaglia Execution TimesTimes

• The CPU times for running the The CPU times for running the Metropolis program with Marsaglia Metropolis program with Marsaglia and Mersenne Twister are very and Mersenne Twister are very close.close.

• Does it mean both generators have Does it mean both generators have roughly the same speed? roughly the same speed?

• Mersenne Twister is faster than Mersenne Twister is faster than Marsaglia.Marsaglia.

• I wrote a program to only generate I wrote a program to only generate 4,294,967,295 Marsaglia numbers 4,294,967,295 Marsaglia numbers and another one that only generates and another one that only generates 4,294,967,295 Mersenne Twister 4,294,967,295 Mersenne Twister numbers. numbers. – Mersenne Twister = 244.710uMersenne Twister = 244.710u– Marsaglia = 312.580uMarsaglia = 312.580u– MT is approximately 22% faster.MT is approximately 22% faster.

27

Gaussian Difference TestGaussian Difference Test

• Data from the Metropolis Data from the Metropolis programprogram

• MarsagliaMarsaglia– Mean = 0.003543Mean = 0.003543– Error bar = 0.007169Error bar = 0.007169

• Mersenne TwisterMersenne Twister– Mean = -0.008677Mean = -0.008677– Error bar = 0.006998Error bar = 0.006998

• Q = 0.222553Q = 0.222553

28

GNUPLOT FOR DISTRIBUTION FUNCTION.• STMC_DF_GNU(Ndat, data1);• STMC_DF_GNU(Ndat2, data2);

29

The Two-Sided Kolmogorov The Two-Sided Kolmogorov TestTest• N = N1 = N2 = 50000N = N1 = N2 = 50000• KOLM2_AS2(N1, N2, DAT1, DAT2, KOLM2_AS2(N1, N2, DAT1, DAT2,

DEL, Q)DEL, Q)– DAT1 = Marsaglia, DAT2 = Mersenne DAT1 = Marsaglia, DAT2 = Mersenne

TwisterTwister• DEL = 0.001172, Q = 0.8820840.001172, Q = 0.882084

– DAT1 = Marsaglia1, DAT2 = Marsaglia2DAT1 = Marsaglia1, DAT2 = Marsaglia2• DEL = 0.001148, Q = 0.8965650.001148, Q = 0.896565

– DAT1 = MT1, DAT2 = MT2DAT1 = MT1, DAT2 = MT2• DEL = 0.002050, Q = 0.2439140.002050, Q = 0.243914

• DEL is the maximum deviation from DEL is the maximum deviation from the exact distribution functionthe exact distribution function

• Q is the approximation to the Q is the approximation to the probability that this deviation is due probability that this deviation is due to chance.to chance.

30

ConclusionsConclusions• The Mersenne Twister The Mersenne Twister

generator is more suitable for generator is more suitable for Monte Carlo Simulations than Monte Carlo Simulations than the Marsaglia random number the Marsaglia random number generator.generator.– Mersenne Twister is faster.Mersenne Twister is faster.– Mersenne Twister more accurate.Mersenne Twister more accurate.– Mersenne Twister has a linear Mersenne Twister has a linear

discretization.discretization.

• Mersenne Twister is probably Mersenne Twister is probably the best generator at the the best generator at the present time.present time.

31

ReferencesReferences1.1. Berg, Bernd. Berg, Bernd. Markov Chain Monte Carlo Markov Chain Monte Carlo

Simulations and Their Statistical AnalysisSimulations and Their Statistical Analysis. . World Scientific, 2004.World Scientific, 2004.

2.2. Karaoglu, Hihat. Karaoglu, Hihat. Mersenne Twister: A Mersenne Twister: A Study on Random Number Generators.Study on Random Number Generators. Project Prestudy. Vrije Universiteit Project Prestudy. Vrije Universiteit Brussel, Apr 2004.Brussel, Apr 2004.

3.3. Korab, Holly. Korab, Holly. Best of the Random Number Best of the Random Number GeneratorsGenerators. The National Center for . The National Center for Supercomputing Applications. Featured Supercomputing Applications. Featured story, Feb 1999.story, Feb 1999.

4.4. L’Ecuyer Pierre. L’Ecuyer Pierre. Uniform Random Number Uniform Random Number Generation.Generation. The Annals of Operations The Annals of Operations Research (1994).Research (1994).

5.5. Lee, J.C. Lee, J.C. Random Number and Statistical Random Number and Statistical Tests.Tests. Talk at Chuo University, Japan. Jan, Talk at Chuo University, Japan. Jan, 2003.2003.

6.6. Matsumoto, Makoto and Takuji Nishimura. Matsumoto, Makoto and Takuji Nishimura. Mersenne Twister: A 623-Dimensionally Mersenne Twister: A 623-Dimensionally Equidistributed Uniform Pseudo-Random Equidistributed Uniform Pseudo-Random Number GeneratorNumber Generator. ACM Transactions on . ACM Transactions on Modeling and Computer Simulations, Vol. 8, No. Modeling and Computer Simulations, Vol. 8, No. 1, Jan 1998, Pages 3-30.1, Jan 1998, Pages 3-30.

32

References (continued)References (continued)7.7. Matsumoto, Makoto and Takuji Nishmura. Matsumoto, Makoto and Takuji Nishmura.

rng_afm.c. An Improved version of the rng_afm.c. An Improved version of the Mersenne TwisterMersenne Twister. . http://www.math.keio.ac.jp/matumoto/emt.html

8.8. Mascagni, Michael, and Srinivasan, Ashok. Mascagni, Michael, and Srinivasan, Ashok. Algorithm 806: SPRNG: A Scalable Library Algorithm 806: SPRNG: A Scalable Library for Pseudorandom Number Generation. for Pseudorandom Number Generation. ACM Transactions on Mathematical ACM Transactions on Mathematical Software, Vol 26, No. 3, September 2000, Software, Vol 26, No. 3, September 2000, pages 436-461.pages 436-461.

9.9. RSA Laboratories’ Bulletin – News and RSA Laboratories’ Bulletin – News and Advice on Data Security and Cryptography. Advice on Data Security and Cryptography. Number 8. Sept 3, 1998. Number 8. Sept 3, 1998.

10.10. Sinclair, Bart. Sinclair, Bart. Permutation Test. Permutation Test. Class Class notes for Elec 428 Computer Systems notes for Elec 428 Computer Systems Performance. Performance. http://www.owlnet.rice.edu/~elec428/rng/perexp.html

11.11. The Scalable Parallel Random Number The Scalable Parallel Random Number Generator Library (SPRNG) for ASCI Monte Generator Library (SPRNG) for ASCI Monte Carlo Computations. Carlo Computations. http://sprng.cs.fsu.edu/

33

QuestionsQuestions