complex networks, synchronization and …complex networks, synchronization and cooperative behaviour...
TRANSCRIPT
![Page 1: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/1.jpg)
Complex networks, synchronization and
cooperative behaviour
Johan Suykens
KU Leuven, ESAT-SCD/SISTAKasteelpark Arenberg 10
B-3001 Leuven (Heverlee), BelgiumEmail: [email protected]
http://www.esat.kuleuven.be/scd/
VUB Leerstoel 2012-2013 - Oct. 31 2012
Complex networks, synchronization and cooperative behaviour
![Page 2: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/2.jpg)
Introduction
http://www.youtube.com”synchronization of metronomes”
(a modern version of the synchronization of two pendulum clocks observedby Christiaan Huygens, 1665)
Complex networks, synchronization and cooperative behaviour 1
![Page 3: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/3.jpg)
Overview
• Chaotic systems synchronization, Lur’e systems
• Cluster synchronization and community detection in complex networks
• Optimization using coupled local minimizers, cooperative behaviour
Complex networks, synchronization and cooperative behaviour 1
![Page 4: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/4.jpg)
Circuits and systems: Chua’s circuit
+ +
− −
vC2C2 C1
iL
LvC1
gNR(vC1
)
NR
G
Ga
Gb
Ga
−EE
gNR(vC1
)
vC1
Chua’s circuit [Chua et al., 1986]: in dimensionless form
x = α(y − x − f(x))y = x − y + z
z = −βy
where
f(x) = m1x +1
2(m0 − m1) (|x + 1| − |x − 1|)
(depending on α, β: bistability, limit cycles, chaos)
Complex networks, synchronization and cooperative behaviour 2
![Page 5: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/5.jpg)
Bifurcation to Chaos
(vC1, vC2)-plane:
Power spectrum vC1:
−→ birth of the double scroll attractor −→
Complex networks, synchronization and cooperative behaviour 3
![Page 6: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/6.jpg)
Lur’e system
L
L(s)
N
σ(·)
m(t) = 0
u
y(t)++
k1
k2
σ(y)
y
• Lur’e system:
x = Ax + Bu
y = Cx
u = σ(y)→ x = Ax + Bσ(Cx)
where x ∈ Rn and σ(·) : R
h → Rh satisfies a sector condition.
• Chua’s circuit: h = 1
Complex networks, synchronization and cooperative behaviour 4
![Page 7: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/7.jpg)
More hidden units
• Multi-stability & Multi-scroll chaos:Extend the nonlinearity andcreate additional equilibrium points[Suykens & Vandewalle, 1991; Arena, 1996; Yalcin, 2001; Lu, 2006]
• Multilayer neural networks are universal approximators [Hornik, 1989]
(Chua’s circuit has 1 hidden unit (h = 1), more hidden units for multi-scrolls)
Complex networks, synchronization and cooperative behaviour 5
![Page 8: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/8.jpg)
A gallery of multi-scroll attractors
[Suykens & Vandewalle, 1991; Yalcin et al., 2001]
Complex networks, synchronization and cooperative behaviour 6
![Page 9: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/9.jpg)
Lur’e systems: examples
• Lur’e system:
x = Ax + Bu
y = Cx
u = σ(y)→ x = Ax + Bσ(Cx)
• Many examples of Lur’e systems in different areas:- Recurrent neural networks (Hopfield network: A = −I, C = I) [Hopfield, 1985]
- Cellular neural networks (sparse and structured matrices A, B, C) [Chua, 1988]
- Actuator saturation in control systems
- Chua’s circuit, multi-scroll circuits
- Arrays of coupled networks
- Genetic oscillator models
L
L(s)
N
σ(·)
m(t) = 0
u
y(t)++
k1
k2
σ(y)
y
Complex networks, synchronization and cooperative behaviour 7
![Page 10: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/10.jpg)
Genetic oscillators
A general genetic oscillator form [Li, Chen, Aihara, 2006]:
x(t) = Ax(t) +
l∑
i=1
Bifi(x(t))
where
• x(t) ∈ Rn: concentrations of proteins, RNAs, chemical complexes
• fi(x(t)) = [fi1(x1(t)); ...; fin(xn(t))]: regulatory function (monotonicallyincreasing or decreasing: e.g. Michaelis-Menten or Hill form)
Examples: Goodwin model, repressilator, toggle switch, circadian oscillators
Complex networks, synchronization and cooperative behaviour 8
![Page 11: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/11.jpg)
Stability analysis and LMIs (1)
• Linear system:x = Ax
Quadratic Lyapunov function:
V = xTPx, P = P T > 0
• Stability analysis:
V = xTPx + xTP x = xT (ATP + PA)x < 0
Global asymptotic stability for
ATP + PA < 0
Linear matrix inequality (LMI) for a given matrix A [Boyd et al., 1994]
Complex networks, synchronization and cooperative behaviour 9
![Page 12: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/12.jpg)
Stability analysis and LMIs (2)
• Lur’e system:x = Ax + Bσ(Cx)
Try e.g. a quadratic Lyapunov function (leading to a sufficient stabilitycondition):
V = xTPx, P = P T > 0
• Stability analysis: exploit the fact that σ belongs to sector [0, k]
V = xTPx + xTP x
≤ xTPx + xTP x−∑
i 2λiσi(σi − kcTi x) = [xTσT ]Z
[
x
σ
]
If
Z =
[
ATP + PA PB + kCTΛBTP + kΛC −2Λ
]
< 0
then globally asymptotically stable (any initial state x(0) convergesto the origin), where Λ = diag{λi} with λi ≥ 0.
Complex networks, synchronization and cooperative behaviour 10
![Page 13: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/13.jpg)
Synchronization of Lur’e systems
• Master-slave synchronization scheme (drive-response):
M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + K(x − z)
Master system M drives slave system S (follows behaviour imposed bythe master system): under which conditions do the systems M and Ssynchronize?
(studies in synchronization of chaotic systems, and applications to secure
communications [Pecora & Carroll, 1990; Chen & Dong, 1998; Yalcin et al., 2005])
• Mutual synchronization scheme:
M1 : x = Ax + Bσ(Cx) + K1(x − z)M2 : z = Az + Bσ(Cz) + K2(x − z)
Systems M1 and M2 mutually influence each other.
Complex networks, synchronization and cooperative behaviour 11
![Page 14: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/14.jpg)
Synchronization example
Master system
No synchronization Synchronization
Complex networks, synchronization and cooperative behaviour 12
![Page 15: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/15.jpg)
Error system
• Consider the error e = x − z relative between the master M and theslave S system:
e = (A − K)e + B[σ(C(e + z)) − σ(Cz)]
• Assume a sector condition on σ(C(e + z)) − σ(Cz)[Suykens & Vandewalle, IJBC 1997; Curran, Suykens, Chua, IJBC 1997]
• A sufficient condition for global asymptotic stability of the error systemcan be obtained by taking e.g. a quadratic Lyapunov function
V (e) = eTPe, P = P T > 0
and derive under which condition dVdt
< 0, ∀e ∈ Rn0 .
Complex networks, synchronization and cooperative behaviour 13
![Page 16: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/16.jpg)
Interpretation as a control problem
• Master-slave synchronization scheme:
M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + u
C : u = K(x − z)
with control signal u.
• Control objective: for given matrices A,B,C design a controller C withmatrix K such that synchronization is achieved.
• For Lur’e systems synchronization can be characterized by LMIs.
• Synchronization can be achieved for any choice of initial states x(0), z(0):for all initial state choices the systems synchronize in the sense that‖x(t) − z(t)‖ → 0 when time t → ∞.
Complex networks, synchronization and cooperative behaviour 14
![Page 17: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/17.jpg)
Different control problems and approaches
• Dynamic measurement feedback control instead of full state feedback:if one cannot measure complete state vectors x, z.
• Robust synchronization: A, B,C matrices non-identical for master andslave system: it is possible to synchronize two systems up to a smallsynchronization error (e.g. limit cycle versus chaos); control in thepresence of disturbances or noise (e.g. H∞ control)
• Control via impulses (sporadic coupling, only from time to time andnon-equidistantly in time) instead of continuously controlling
• Control in systems with time-delays
• Other forms of synchronization: partial synchronization, clustersynchronization, phase synchronization, connection with graph topology
[Chen et al.; Wu et al.; Suykens et al.; Nijmeijer et al.; Yalcin et al.]
Complex networks, synchronization and cooperative behaviour 15
![Page 18: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/18.jpg)
Problems in synchronization theory
IMPULSIVECOUPLING
Robust
Impulsive
Time-delaySynchronization
Synchronization
Synchronization
Synchronization
Synchronization
Nonlinear H∞
Robust Nonlinear H∞
EXTERNALINPUT
MISMATCHPARAMETER
AutonomousNon-autonomous
Design Purposes
Master-slave Synchronization Schemes
DELAY
Chaotic Lur’e Systems
[Yalcin et al., 2005]
Complex networks, synchronization and cooperative behaviour 16
![Page 19: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/19.jpg)
Overview
• Chaotic systems synchronization, Lur’e systems
• Cluster synchronization and community detection in complexnetworks
• Optimization using coupled local minimizers, cooperative behaviour
Complex networks, synchronization and cooperative behaviour 16
![Page 20: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/20.jpg)
Complex networks
Random network Scale−free network
Number of links Number of linksNumber of links
Num
ber
of n
odes
Num
ber
of n
odes
[log scale]
[log
sca
le]
Num
ber
of n
odes
[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]
- Random networks: bell curve distribution- Scale-free networks: power law distribution
Robust against accidental failures, but vulnerable to coordinated attacks
Biological networks: growth (gene duplication) and preferential attachement
(rich-gets-richer mechanism: new nodes prefer to link to the more connected nodes)
Complex networks, synchronization and cooperative behaviour 17
![Page 21: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/21.jpg)
Map of protein-protein interactions
[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]Highly linked proteins (network hubs) tend to be crucial for cell survival.
Only few proteins are able to physically attach to a huge number.
www.nd.edu/∼networks
Complex networks, synchronization and cooperative behaviour 18
![Page 22: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/22.jpg)
Wave phenomena in neuronal networks
- Hodgkin-Huxley type model of oscillatory activity in the bursting neurons of a snail
- Burst waves of antiphase spiking excitation in a 200× 200 lattice of electrically coupled
nonidentical neurons (snapshots at different times)
[Komarov, Osipov, Suykens, Chaos 2008]
Complex networks, synchronization and cooperative behaviour 19
![Page 23: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/23.jpg)
Synchronization in complex networks
• Synchronization of chaotic systems [Pecora & Carroll, 1990]:mainly low dimensional systems and regular network topologies
• Complex networks: larger networks, different network topologies
• Complex networks:- relation between network topology and synchronization into clusters?- how to design to achieve desired clusters?- how to cope with time delays or communication constraints?- how to enhance synchronizability of complex networks- how to rewire the network?- ...
[Suykens & Osipov, Focus issue, Chaos 2008; Arenas et al., PR 2008]
Complex networks, synchronization and cooperative behaviour 20
![Page 24: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/24.jpg)
Link between synchronization and spectral clustering
• (generalized) Kuramoto model: N coupled phase oscillators
dθi
dt= ωi +
∑
j
Kij sin(θj − θi), i = 1, ..., N
Special case: ωi = ω, Kij = σaij with adjacency matrix [aij]
• Linearized dynamics (Laplacian matrix L)
dθi
dt= −σ
∑
j
Lijθj, i = 1, ..., N
• Relationship between topological scales and dynamic time scalesModular structures emerge at different time scales
[Arenas et al., PRL 2006, PR 2008]
Complex networks, synchronization and cooperative behaviour 21
![Page 25: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/25.jpg)
Complex networks
Synchronization
Spectral clustering
Complex networks, synchronization and cooperative behaviour 22
![Page 26: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/26.jpg)
Spectral clustering
SVM, kernel methods
Data
Complex networks, synchronization and cooperative behaviour 23
![Page 27: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/27.jpg)
Complex networks
Synchronization
Spectral clustering
SVM, kernel methods
Data
Complex networks, synchronization and cooperative behaviour 24
![Page 28: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/28.jpg)
Community detection from synchronization
• Kuramoto model: θi = ω + σ∑
j aij sin(θj − θi)
• Follow the evolution of
ρij(t) = 〈cos[θi(t) − θj(t)]〉
averaged over different initial conditions.
• Community detection based on a binary dynamic connectivity matrix
[Dt(T )]ij = 1 if ρij(t) > T, zero otherwise
T large enough: one finds set of disconnected clustersT smaller: inter-community connections become visible
• Other approach: matrix DT (t) unravels the topological structure of thenetwork at different time scales.
[Arenas et al., PRL 2006, PR 2008]
Complex networks, synchronization and cooperative behaviour 25
![Page 29: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/29.jpg)
Finding communities in weighted networks (1)
0 0.2 0.4 0.60
0.5
1
1.5
2
2.5
3
time
Qw
0
0.5
1
1.5
2
2.5
3
Qw
=0.4947
Synthetic example [Lou & Suykens, Chaos 2011]: community detectionby considering [D]ij = tij if ρij(t) > T and zero otherwise, where tij isthe time needed for nodes i and j to synchronization in the sense thatρij(t) > T .
Complex networks, synchronization and cooperative behaviour 26
![Page 30: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/30.jpg)
Finding communities in weighted networks (2)
1030
24
16
19
23
21
15
9
1312
20
17
11
22
4
7
5
6
28 25 32
2926
14
8 3
2
1
18
31
3334
27
C3
C2C
1
C4
(a)
6 7 17 5 11 1 12 18 2 8 14 20 3 4 13 22 9 10 31 15 33 34 19 21 16 24 28 27 30 23 25 26 32 290
0.5
1
1.5
2
2.5
3
3.5
time
(b)
Qw
=0.4439
on the Zachary’s karate club network [Lou & Suykens, Chaos 2011]
Complex networks, synchronization and cooperative behaviour 27
![Page 31: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/31.jpg)
Finding communities in weighted networks (3)
on the American football team network [Lou & Suykens, Chaos 2011]
Complex networks, synchronization and cooperative behaviour 28
![Page 32: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/32.jpg)
”Programming” clusters into complex networks
- cluster design on a 20 × 60 lattice of identical Rossler oscillators.- cluster ”CHAOS” obtained from randomly distributed initial conditions.
[Belykh, Osipov, Petrov, Suykens, Vandewalle, Chaos 2008]
Complex networks, synchronization and cooperative behaviour 29
![Page 33: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/33.jpg)
Overview
• Chaotic systems synchronization, Lur’e systems
• Cluster synchronization and community detection in complex networks
• Optimization using coupled local minimizers,cooperative behaviour
Complex networks, synchronization and cooperative behaviour 29
![Page 34: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/34.jpg)
Optimization
Local optimization
+ fast
- local optimum
Newton, QN, LM, CG
Complex networks, synchronization and cooperative behaviour 30
![Page 35: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/35.jpg)
Optimization
Local optimization
+ fast
- local optimum
Newton, QN, LM, CG
Global optimization
- slow
+ global search
GA, SA, swarms
Complex networks, synchronization and cooperative behaviour 30
![Page 36: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/36.jpg)
Optimization
Local optimization
+ fast
- local optimum
Newton, QN, LM, CG
???
+ fast
+ global search
???
Global optimization
- slow
+ global search
GA, SA, swarms
Complex networks, synchronization and cooperative behaviour 30
![Page 37: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/37.jpg)
Local optimization
• Consider the unconstrained optimization problem:
minx∈Rn
U(x)
with cost function U(·) continuously differentiable.
• Simple continuous-time steepest descent algorithm:
x = −η∇xU(x)
converging to a local optimum.
• Better local optimization methods:momentum term, Newton method, conjugate gradients, ...
Complex networks, synchronization and cooperative behaviour 31
![Page 38: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/38.jpg)
Coupled local minimizers
• Essential idea for Coupled Local Minimizers (CLM):
1. consider two (or more) local optimizers and let them interact2. enforce that the optimizers should reach the same final state,
i.e. require state synchronization
• Realizing cooperative behaviour for optimization: based on coupling ofoptimization processes and master-slave synchronization
• Hierarchical scheme: objectives (cost functions) at the individual leveland at the group level
[Suykens et al., IJBC 2001]
Complex networks, synchronization and cooperative behaviour 32
![Page 39: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/39.jpg)
Coupled local minimizers
weight space
cost
Multi−start local optimization
No interaction
weight space
cost
Coupled Local Minimizers
Interaction and information exchange
[Suykens et al., IJBC 2001]
Complex networks, synchronization and cooperative behaviour 33
![Page 40: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/40.jpg)
Array consisting of coupled local minimizers
space
space
cost
cost
Complex networks, synchronization and cooperative behaviour 34
![Page 41: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/41.jpg)
CLM: a toy example
• Example: consider the following objective
minx,z
U(x) + U(z) subject to x = z
Lagrange programming network:
x = −∇xU(x) − (x − z) − λ
z = −∇zU(z) + (x − z) + λ
λ = x − z
Complex networks, synchronization and cooperative behaviour 35
![Page 42: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/42.jpg)
Toy example: double potential well
−6 −4 −2 0 2 4 60
100
200
300
400
500
600
700
800
900
x
U(x
)
0 20 40 60 80 100 120 140 160 180 200−10
−5
0
5
10
15
20
25
30
t
x,z,λ
The initial states x(0), z(0) are chosen to be in the two different valleys.The states x(t), z(t) converge to the global solution at x = z = −2.9
(blue: x(t) - red: z(t) - green: λ(t))
Complex networks, synchronization and cooperative behaviour 36
![Page 43: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/43.jpg)
Lagrange programming network
• Problem statement:
minx∈Rn
f(x) subject to h(x) = 0
• Lagrangian: L(x, λ) = f(x) + λTh(x)
• Lagrange programming network:
{
x = −∇xL(x, λ)
λ = ∇λL(x, λ)
This can be viewed as a continuous-time optimization algorithm.
[Zhang & Constantinides, 1992]
Complex networks, synchronization and cooperative behaviour 37
![Page 44: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/44.jpg)
CLM: more general formulation (1)
• Consider a group consisting of q optimizers {x(i)}qi=1
• Minimize average energy cost subject to pairwise synchronization states
minx(i)∈Rn
1
q
q∑
i=1
U(x(i))
subject to x(i) − x
(i+1) = 0, i = 1, 2, ..., q
• Boundary conditions x(0) = x
(q), x(q+1) = x
(1)
[Suykens et al., IJBC 2001]
Complex networks, synchronization and cooperative behaviour 38
![Page 45: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/45.jpg)
CLM: more general formulation (2)
• Augmented Lagrangian (synchronization as hard and soft constraint)
L(x(i)
, λ(i)
) =η
q
qX
i=1
U [x(i)
] +1
2
qX
i=1
γi ‖x(i) − x
(i+1)‖22+
qX
i=1
〈λ(i), [x
(i) − x(i+1)
]〉
• Lagrange programming network:
{
x(i) = −η
q∇
x(i)U [x(i)] + γi−1[x
(i−1) − x(i)] − γi[x
(i) − x(i+1)] + λ(i−1) − λ(i)
λ(i) = x(i) − x
(i+1) , i = 1, 2, ..., q
Complex networks, synchronization and cooperative behaviour 39
![Page 46: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/46.jpg)
Optimal cooperation
• Decrease of ensemble energy cost:
d〈U〉
dt=
1
q
qX
i=1
〈∂U [x(i)]
∂x(i)
, x(i)〉
=1
q
qX
i=1
〈∂U [x(i)]
∂x(i)
,−η
q
∂U [x(i)]
∂x(i)
+ γi−1[x(i−1) − x
(i)]
−γi[x(i) − x
(i+1)] + λ
(i−1) − λ(i)〉
• Optimal cooperation: LP problem in γ (scheduling of γi values)
minγ∈Rq
d〈U〉
dt|x,λ such that γ < γi < γ, i = 1, 2, ..., q
This incorporates the principle of master-slave synchronization.
Complex networks, synchronization and cooperative behaviour 40
![Page 47: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/47.jpg)
Example: optimization of Lennard-Jones clusters
• In predicting 3D structure of proteins from amino acid sequences,potential energy surface (PES) minimization is often related to thenative structure of the protein.Benchmark problem: optimization of Lennard-Jones (LJ) clusters [Sali,1994; Wales, 1997, 1999].
• Cost function:
ULJ = 4∑
i<j
(1
r12ij
−1
r6ij
)
with rij the Euclidean distance between atom i and j (j = 1, ..., N).
• (LJ)38 which possesses a double-funnel energy landscape and is knownto be an interesting test-case [Wales, 1997, 1999].
• Important role of p(x(0)) ∝ exp[− 12σ2x(0)T
x(0)] (similar to consideringa confining potential in effective potential minimization methods).
Complex networks, synchronization and cooperative behaviour 41
![Page 48: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/48.jpg)
Case (LJ)38
0 0.5 1 1.5 2 2.5 3
x 10−7
100
102
104
106
108
1010
1012
1014
t
ULJde
lta
Evolution of the cost function for q = 50 coupled local minimizers, reachingthe global minimum configuration for (LJ)38 with double-funnel landscape.
Complex networks, synchronization and cooperative behaviour 42
![Page 49: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/49.jpg)
Case (LJ)150
Potential for application to larger scale problems
Complex networks, synchronization and cooperative behaviour 43
![Page 50: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/50.jpg)
Example: CLM training of MLP neural networks
• CLM with state vectors x(i) (i = 1, ..., q) equal to the unknown weight
vectors θ(i) of the MLP.
• CLM training process corresponds to coupled backpropagation processeswith weight vector synchronization.
• The initial distribution of p(x(i)(0)) (i = 1, ..., q) (at time 0) plays animportant role, similar to the choice of a regularization constant (inmethods of minimizing errors and keeping the weights small).
Complex networks, synchronization and cooperative behaviour 44
![Page 51: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/51.jpg)
CLM training of neural networks (1)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
x
y
- MLP training (10 hidden units) of a sinusoidal function (green) given 20 noisy data
- Application of scaled CG without early stopping leading to overfitting (red) and best
result by Bayesian learning with regularization (blue).
Complex networks, synchronization and cooperative behaviour 45
![Page 52: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/52.jpg)
CLM training of neural networks (2)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5
−1
−0.5
0
0.5
1
1.5
2
x
y
CLM result which optimizes a sum squared error on training data withoutregularization of the cost function.
Complex networks, synchronization and cooperative behaviour 46
![Page 53: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/53.jpg)
CLM training of neural networks (3)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x 10−5
1
1.5
2
2.5
3
3.5
4
t
U
CLM evolution (group of q = 20 optimizers) of the sum squared error costfunction during optimization
Complex networks, synchronization and cooperative behaviour 47
![Page 54: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/54.jpg)
Alternative Formulation to CLMs
• Capture a group of optimizers within a ball and shrink the ball
• Objective:
minx(i)∈Rn,r∈R
〈U〉 + 12 ν r2
subject to ‖x(i) − x(i+1)‖2
2 ≤ r2, i = 1, ..., q
where 〈U〉 = 1q
∑qi=1 U [x(i)].
• Advantage: always easy to find a feasible point to the constraints duringthe optimization process.
[Suykens & Vandewalle, 2002]
Complex networks, synchronization and cooperative behaviour 48
![Page 55: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/55.jpg)
CLM: extensions
• Coupled Newton methods with applications in civil engineering[Teughels, De Roeck, Suykens, 2002]
• Additional noise can be injected into the system [Gunel et al., 2006]
• Extensions to coupled simulated annealing processes with cost functionevaluations only [Xavier-de-Souza, Suykens, Vandewalle, Bolle, IEEE-SMC-B 2010].
Successfully applied e.g. for tuning parameter selection in kernel methods,being more efficient than grid search, SA or GA [K. De Brabanter et al.,CSDA 2010].
• Stability analysis of CLMs [Lou & Suykens, IEEE-TCAS-I, in press].
• Hybrid CLMs: occasional impulsive coupling, suitable for parallelimplementations [Lou & Suykens, 2012].
Complex networks, synchronization and cooperative behaviour 49
![Page 56: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/56.jpg)
Conclusions
• Synchronization phenomena: naturally happening in a wide range ofsystems and complex networks.
• Lur’e systems: broad class of nonlinear systems, conditions for globalstability and global synchronization can be obtained.
• Community detection in complex networks: obtainable also through asynchronization process
• Coupled local minimizers: aims for global search together with fasterconvergence.
Complex networks, synchronization and cooperative behaviour 50
![Page 57: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/57.jpg)
Acknowledgements (1)
• Colleagues at ESAT-SCD (especially research units: systems, models,control - biomedical data processing - bioinformatics):
C. Alzate, A. Argyriou, J. De Brabanter, K. De Brabanter, L. De Lathauwer, B. De
Moor, M. Diehl, Ph. Dreesen, M. Espinoza, T. Falck, D. Geebelen, X. Huang, B.
Hunyadi, A. Installe, V. Jumutc, P. Karsmakers, R. Langone, J. Lopez, J. Luts, R.
Mall, S. Mehrkanoon, M. Moonen, Y. Moreau, K. Pelckmans, J. Puertas, L. Shi, M.
Signoretto, P. Tsiaflakis, V. Van Belle, R. Van de Plas, S. Van Huffel, J. Vandewalle,
T. van Waterschoot, C. Varon, S. Yu, and others
• L. Chua, P. Curran, A. Huang, T. Yang, A. Munuzuri, M. Yalcin, S.Gunel, S. Ozoguz, G. Osipov, M. Komarov, V. Belykh, V. Petrov, S.Xavier-de-Souza, X. Lou, A. Teughels, G. De Roeck, S. Arnout.
• Support from ERC AdG A-DATADRIVE-B, KU Leuven, GOA-MaNet,COE Optimization in Engineering OPTEC, IUAP DYSCO, FWO projects,IWT, IBBT eHealth, COST
Complex networks, synchronization and cooperative behaviour 51
![Page 58: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/58.jpg)
Acknowledgements (2)
Complex networks, synchronization and cooperative behaviour 52
![Page 59: Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory](https://reader036.vdocuments.net/reader036/viewer/2022062506/5f03a7cf7e708231d40a1f41/html5/thumbnails/59.jpg)
Thank you
Complex networks, synchronization and cooperative behaviour 53