maximum entropy, maximum entropy production and their application to physics and biology
DESCRIPTION
Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology. Roderick C. Dewar Research School of Biological Sciences The Australian National University. Part 1: Maximum Entropy (MaxEnt) – an overview Part 2: Applying MaxEnt to ecology - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/1.jpg)
Maximum Entropy,
Maximum Entropy Production
and their
Application to Physics and Biology
Roderick C. Dewar
Research School of Biological Sciences
The Australian National University
![Page 2: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/2.jpg)
Part 1: Maximum Entropy (MaxEnt) – an overview
Part 2: Applying MaxEnt to ecology
Part 3: Maximum Entropy Production (MEP)
Part 4: Applying MEP to physics & biology
![Page 3: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/3.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 4: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/4.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 5: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/5.jpg)
system
energy in
What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …
environment
matter in
many interacting degrees of freedom
energy out
matter out
opennon-equilibrium
![Page 6: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/6.jpg)
Poleward heat transport
SW
LW
Latitudinal heat
transport H = ?
170 W m-2
300 W m-2
T
![Page 7: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/7.jpg)
Cold plate, Tc
Hot plate, Th
Ra < 1760
conduction
T
Cold plate, Tc
Hot plate, Th
Ra > 1760
convection
H = ?
Turbulent heat flow (Raleigh-Bénard convection)
![Page 8: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/8.jpg)
Fsw
Flw + H + E C, H20, O2, N
T,
Ecosystem energy & mass fluxes
![Page 9: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/9.jpg)
system
energy in
What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …
environment
matter in
many interacting degrees of freedom
energy out
matter out
opennon-equilibrium
many degrees of freedom statistical mechanicsstatistical mechanics
Global Circulation Models, Dynamic Ecosystem Models ….
![Page 10: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/10.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 11: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/11.jpg)
W(A) = number of microstates that give macrostate A
Microstate i1 Macrostate A = less detailed description
Ludwig Boltzmann (1844 - 1906)
The most probable macrostate A is the one with the largest W(A) (assume microstates are a priori equiprobable)
SB(A) = kBlog W(A) = Boltzmann entropy of macrostate A
The most probable macrostate is the one of maximum entropy
Boltzmann microstate counting
Microstate i2
![Page 12: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/12.jpg)
Example: N independent distinguishable particles with fixed total energy E
Macrostate A = {nj particles are in state j}
Microstate i = {the mth particle is in state jm}ε3
ε2
ε1
j
jjj
jj
jj nnnnNN εβαloglogΦ
j
jn
NAW
!
!Nn
jj
Enj
jj ε: maximise S = kBlog W subject to
j
jj
jj
ZN
nβεexp
1
βεexp
βεexp
0
Φ
jn
(large N)
![Page 13: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/13.jpg)
Boltzmann entropy Clausius entropy
β 1/kBT
Given E, Smax = kBlogWmax = kB(βE + NlogZ)
under δE = δQ, Smax changes by δSmax = kBβ(δQ)
cf. Clausius thermodynamic entropy δSTD = δQ/T
Smax STD
BUT: microstate counting only works for non-interacting particles
![Page 14: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/14.jpg)
pi = probability that system is in microstate i Macroscopic predictions via
J Willard Gibbs (1839 - 1903)
Gibbs algorithm
i
iiQpQ
The Gibbs algorithm
(MaxEnt)
Maximise H = -i pi log pi with respect to {pi} subject to the
constraints (C) on the system
But how do we construct pi ?
‘minimise the index of probability of
phase’
![Page 15: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/15.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 16: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/16.jpg)
• Closed, isolated
• Closed
• Open
Three applications of MaxEnt (equilibrium systems)
• Microcanonical
• Canonical
• Grand-canonical
System constraints (C) Distribution (pi)
![Page 17: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/17.jpg)
Example 1: closed, isolated system in equilibrium
C: N and E fixed
Microstate i = any N-particle state with total energy Ei restricted to E
Precise description of i and Ei depends on microscopic physics (CAN include particle interactions)
1
1
NE
iip
,ΩMaximise
i
NE
ii ppH log
,Ω
1
subject to
NE
iii
NE
ii ppp
,Ω,ΩαlogΦ
11
NEpi ,Ω
10
ip
Φbasis for Boltzmann’s microstate counting
![Page 18: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/18.jpg)
Example 2: closed system in equilibrium
Microstate i = any N-particle state (no restriction on Ei)
E
Maximise subject to
C: N and fixedE
iC
i EZ
p βexp 1
0
ip
Φ
β 1/kBT
N
iii ppH
Ωlog
1
1
1
N
iip
Ω
EEp
N
iii
Ω
1
N
iii
N
iii
N
ii Epppp
ΩΩΩβαlogΦ
111
Hmax STD
![Page 19: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/19.jpg)
Example 3: open system in equilibrium
Microstate i = any physically allowed microscopic state (no restriction on Ei or Ni)
E N
Maximise
1i
ip
Ω
log1i
ii ppH subject to EEpi
ii
NNpi
ii
C: and fixedEN
iiGC
i NEZ
p γβexp 1
0
ip
Φ β 1/kBT
γ -μ/kBT
ΩΩΩΩ
γβαlogΦ1111 i
iii
iii
iii
i NpEpppp
Hmax STD
![Page 20: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/20.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 21: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/21.jpg)
Frequency interpretation (Venn, Pearson, Fisher …)
System has Ω a priori equiprobable microstates
N independent identical systems, ni = no. of systems in state i
pi describes a physical property of the real world (frequency)
! ... !!
!
Ωnnn
N
21
W = no. of microstates giving {n1,n2 … nΩ}
pi = ni /N = frequency of microstate i
NHppWN i
ii as loglogΩ
1
1
MaxEnt coincides with large-N limit of Maximum Probability for multinomial W
![Page 22: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/22.jpg)
• pi represents our state of knowledge of the real world
• basic axioms for uncertainty H associated with pi the unique uncertainty function is
• Applies to any discrete set of outcomes i
pi = 1/6 (i = 1…6) H = log 6maximum uncertainty :
pi = 0 (i = 1,2...5), p6 = 1 H = 0
minimum uncertainty :
Ω
1log
iii ppH
Information theory interpretation (Shannon 1948, Jaynes 1957 …)
Claude Shannon(1916-2001)
![Page 23: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/23.jpg)
Jaynes (1957b, 1978)
Q (= ΣipiQi) reproducible under C
it is sufficient to encode only the information C into pi …
all information other than C is thrown away
… but this is precisely what MaxEnt does!
MaxEnt = max H subject to C
H = -i pi log pi = missing information about i
Behaviour that is experimentally reproducible under conditions C must be
theoretically predictable from C aloneEdwin Jaynes (1922-1998)
![Page 24: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/24.jpg)
Assumed constraints C
experimental conditionsconservation laws
microstates (e.g. QM)
Reproducible behaviour Q
Max H subject to C pi
The prediction game
i
iiQpQ
Observed behaviour Qobs
Qobs Q missing constraint
MaxEnt
testC C'
ESSENTIAL PHYSICS PREDICTION
OBSERVATION
![Page 25: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/25.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 26: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/26.jpg)
Edwin Jaynes (aged 14 months)
Information theory interpretation of MaxEnt
general algorithm for predicting reproducible behaviour
under given constraints
can be extended to non-equilibrium systems
(same principle, different constraints)
‘Maximum caliber principle’ (Jaynes 1980, 1996)
Γ micropaths
ΓΓ logmax ppH
cf. Feynman path integral formalism of QM!
![Page 27: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/27.jpg)
A B
The second law in a nutshell
AB reproducible WB WA' = WA SB SA
WB
WA WA'
. .microscopic
path in phase-space
after Jaynes 1963, 1988
S = kBlog W
Liouville Theorem(Hamiltonian dynamics)
reproducible macroscopic change
![Page 28: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/28.jpg)
• The problem: to predict “complex system” behaviour
• The solution: statistical mechanics
- Boltzmann microstate counting (maximum probability)
- Gibbs algorithm (MaxEnt)
• Applications of MaxEnt to equilibrium systems
- micro-canonical, canonical, grand-canonical distributions
• Physical interpretation of MaxEnt
- frequency interpretation
- information theory interpretation (Jaynes)
• Extension to non-equilibrium systems (Jaynes)
• General properties of MaxEnt distributions
Part 1: MaxEnt – an overview
![Page 29: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/29.jpg)
Some general properties of MaxEnt distributions
λexpλ
1
kkiki f
Zp
1i
ip
m k,Ffpf ki
kiik 1
k
kk FλZH λlogmax
k
kZ
Fλ
λlog
kjkjkj
kjk
jjk Cffff
ZFC
λλ
λlog
λ
2
subject to m + 1 constraints C
i
ii ppH logmax
Fλ
k
k F
FHF
maxλ FHmax
i kkik fZ λexpλ
;
λ max2
kjkjj
kjk B
FF
FH
FB
1CB
Response-fluctuation & reciprocity relations:
Stability-convexity relation:
Constitutive relation: Orthogonality:
Partition function:
![Page 30: Maximum Entropy, Maximum Entropy Production and their Application to Physics and Biology](https://reader035.vdocuments.net/reader035/viewer/2022062518/56814c2d550346895db93397/html5/thumbnails/30.jpg)
Summary of Lecture 1 …
The problem
to predict the behaviour of non-equilibrium systems with many degrees of freedom
The proposed solution
MaxEnt: a general information-theoretical algorithm for predicting reproducible behaviour under given constraints
Boltzmann
Gibbs
Shannon
Jaynes