1

Properties of the Duration of Transient Oscillations in a Ring Neural Network

Yo Horikawa and Hiroyuki Kitajima

Kagawa University

Japan

2

1. Background Ring network of sigmoidal neuron models with inhibitory unidirectional coupling

When the coupling gain: |g|> gth(N) ≥ 1,

･ Number of neurons: odd (N = 2M + 1) → Stable oscillation

･ Number of neurons: even (N = 2M) → Stable steady states

(e.g. x2k ≈ 1, x2k-1 ≈ -1 (0 ≤ k ≤ m)

τdx1/dt = -x1 + f(xN)

τdxn/dt = -xn + f(xn - 1) (2 ≤ n ≤ N) (A1)

xn: state of neuron n

N: number of neurons f(x) = tanh(gx): output of neurong: coupling gain (g < 0) τ: time constant

45

N12

6

78

3

f (x ) = tanh(gx ) (g = -10)

-1

0

1

-2 -1 0 1 2x

3

･ Number of neurons is odd (N = 2M + 1).

→ Ring oscillator: Ring of inverters (NOT gates)

Stable oscillations of rectangular waves

45

2M+1

12

6

7

8

3

+ +-

+ - +

- - +- + +- + -+ + -+ - -+ - +

t

τdx1/dt = -x1 + f(xN)

τdxn/dt = -xn + f(xn - 1) (2 ≤ n ≤ N) (A1)

xn: state of neuron n

N: number of neurons f(x) = tanh(gx): output of neuron g: coupling gain (g < 0) τ: time constant

xn(t)

4

･ Number of neurons is even (N = 2M).

→ Network is bistable.

However, transient states from random initial values can be extremely long.

Transient oscillations lasted more than a month in computer simulation with a workstation for N = 100.

45

2M

12

6

7

8

3

+ +- --

x = (+ - ･･･ + -) OR

x = (- + ･･･ - +)

Simulation

N = 40, g = -10.0, τ= 1.0,

xn(0) ~ N(0, 0.12) (1 ≤ n ≤ 40)

Circuit experiment

n = 40, C = 10μF, R = 10kΩ

(time constant: CR = 0.1s)

Oscillation lasts about 45 minutes.

5

The transient oscillations are traveling waves of the boundaries of separated blocks.

Scenario to the steady state The neurons are separated into two blocks in which the signs of the states of the neurons change alternately.

(+-+-+- ･･･ +-+-+--+-+-+ ･･･ -+-+).

Two boundaries (inconsistencies) of the two blocks move in the direction of the coupling by changing the signs of the states of the neurons.

(--+-+- ･･･ +-+-+-+--+-+ ･･･ -+-+)

→ (-++-+- ･･･ +-+-+-+-++-+ ･･･ -+-+)

→ (-+--+- ･･･ +-+-+-+-+--+ ･･･ -+-+)

The velocities of the boundaries differ only slightly and continue to move for a long time.

(-+-+-+ ･･･ -+-+-+--+--+-+ ･･･ -+-+)

→ (-+-+-+ ･･･ -+-+-+-+-+-+-+ ･･･ -+-+)

They finally merge and the network reaches the steady state.

6

We consider excitatory coupling instead of inhibitory coupling for simplicity.

Neurons are separated in two blocks of the same signs. (- - - - - - - - - + + + + + + + + + +)

(+ - - - - - - - - - + + + + + + + + +) (+ + - - - - - - - - - + + + + + + + +) (+ + + - - - - - - - - - + + + + + + +)

(+ + + + + + + + + + + + - - + + +) (+ + + + + + + + + + + + + - + + +) (+ + + + + + + + + + + + + + + + +)

Demonstration

τdx1/dt = -x1 + f(xN)

τdxn/dt = -xn + f(xn - 1) (2 ≤ n ≤ N) (A2)

xn: state of neuron n

f(x) = tanh(gx): output of neurong: coupling gain (g > 1)

t

7

2. Velocity of the boundariesRandom initial values of xn(0) (1 ≤ n ≤ N)

(after short time) → Propagating two blocks

(+ + + + + + + + + + + + + + + - - - - - - - - - + + + + + + + + + + + + + + +)

Difference between the velocities of two blocks: v0 – v1 < 0

→ Changes in the length of a smaller block: l(t) with dl(t)/dt = v0 – v1

→ Duration T of the oscillation with the condition l(T) = 0

v1 v0

l(t)

l(t): the length of a smaller block

The number of neurons in the block

8

Propagation time of the boundary per neuron: Δt

→ Velocity of the block per neuron: v = 1/Δt

Sigmoidal function: f(x) = tanh(gx) → Sign function: sgn(x)

τdxn/dt = -xn - 1 (xn -1 < 0)

= -xn + 1 (xn -1 > 0) (A3)sgn (x )

-1

0

1

-2 -1 0 1 2x

τdxn/dt = -xn + 1τdxn/dt = -xn - 1 τdxn/dt = -xn - 1

x(t0) ≈ - 1

n

n

9

τdxn/dt = -xn + 1 (xn -1 > 0)

xn(t1) = exp(-(t1 - t0)/τ)(xn(t0) - 1) + 1

≈ -2exp(-(t1 - t0)/τ) + 1

v1 v0

l(t)

v1 v0

l(t)

n = 1 N

v1 v0

l(t)

v1 v0

l(t)

n = 1 N

τdxn/dt = -xn - 1 (xn -1 < 0)

xn(t1 +Δt1) = exp(-Δt1/τ)(xn(t1) + 1) - 1 = 0

Δt1 =τlog{2 - 2exp[-(t1 - t0)/τ]}

=τlog2 + log{1 - exp[-(t1 - t0)/τ]}

v1 = 1/Δt1 = 1/[τlog{2 - 2exp[-(t1 - t0)/τ]}] ≈ 1/[τlog{2 - 2exp[log2 ･ l(t)]}]

v0 = 1/Δt0 = 1/[τlog{2 - 2exp[-(t0 – t-1)/τ]}] ≈ 1/[τlog{2 - 2exp[log2 ･ (N - l(t))]}]

dl(t)/dt = v0 - v1 = 1/Δt0 - 1/Δt1

≈ 1/[τlog{2 - 2exp[log2 ･ (N - l(t))]}] - 1/[τlog{2 - 2exp[log2 ･ l(t)]}]

Δt ≈τlog2 (t1 - t0) → ∞

10

dl/dt ≈ 1/[τlog{2 - 2exp[log2 ･ (N - l)]}] - 1/[τlog{2 - 2exp[log2 ･ l]}]

= 1/{τ[log2 + log(1 - 2-(N - l))]} - 1/{τ[log2 + log(1 - 2-l)]}

≈ 1/[τ(log2 - 2-(N - l))] - 1/[τ(log2 - 2-l)]

≈ 1/τ·[1/log2 + 2-(N - l)/(log2)2 - (1/log2 + 2-l/(log2)2)]

= 1/[τ(log2)2]·(2-(N - l) - 2-l)

= k(exp(-c(N - l)) - exp(-cl)) (k = 1/[τ(log2)2], c = log2) (A4)

The velocity of the boundary depends on an exponential of distance to the forward boundary.

The difference between the velocites of two boundaries decreases exponentially with the length l and N - l of the blocks.

Changes in the length of the blocks is exponentially small when the block lengths are large.

v1 v0

l(t)

v1 v0

l(t)

n = 1 N

11

3. Duration of the transient oscillations dl/dt = k(exp(-c(N - l)) - exp(-cl)) (k = 1/[τ(log2)2], c = log2)

This equation is solved as

exp(cl(t)) = exp(cN/2)tanh(-exp(-cN/2)ckt + arctanh(exp(cl0 - cN/2)))

(l0 = l(0) (0 ≤ l0 ≤ N/2)) (A5)

The two blocks merge, i.e. a smaller block disappear.

→ The oscillation ceases.

The duration T of the transient oscillations is given with l(T) = 0.

T = 1/(ck)·exp(cN/2){arctanh[exp(c(l0 - N/2))] – arctanh[exp(-cN/2)]} (A6)

0

10

20

30

40

0 2000 4000 6000 8000 10000t

l l(0) = 10l(0) = 15l(0) = 18l(0) = 22

T10

T15

T18

T22

12

A simple form of the duration T(l0) when the number N of neurons is large is obtained by letting N be infinity (N → ∞).

dl/dt = -kexp(-cl)

l(t) = 1/c·log(exp(cl0) - ckt)

T = (exp(cl0) - 1))/ck = τlog2·(2l0 - 1) (l(T) = 0)

(k = 1/[τ(log2)2], c = log2, l0 = l(0) (0 ≤ l0 ≤ N/2)) (A7)

The duration of the transient oscillations increases exponentially with the length of a smaller block, i.e. the number of neurons in the block.

Fig. A1. Duration T of the transient oscillations with the initial block length l0 (1 ≤ l0 ≤ 19) and N = 40.

0

1

2

3

4

5

6

0 5 10 15 20l0

log 1

0(T

)

simulationEq. (A6)

Eq. (A7)

13

4. Distribution of the duration under random initial conditions Random initial states xn(0) (1 ≤ n ≤ N) of neurons ~ i. i. d.

→ The initial length of a smaller block is distributed uniformly in (0, N/2).

l0 ~ U(0, N/2).

The probability density function h(T) of the duration T of the transient oscillations is derived with

h(T) = |dT(l0; N)/dl0|-1/(N/2)

= 2kexp(-cN/2)cosech[2(exp(-cN/2)ckT + arctanh(exp(-cN/2)))]∙2/N

= 2/[τ(log2)2]∙2-N/2cosech[2(2-N/2T/[τlog2] + arctanh(2-N/2))]∙2/N

(A8)

'd)'(d)2/,0(00 0

0

Tl

TThlNU

14

The distribution of the duration T under random initial condition.

h(T) = 2/[τ(log2)2]∙2-N/2cosech[2(2-N/2T/[τlog2] + arctanh(2-N/2))]∙2/N (A8)

A cut-off point: Tc = τlog2·2N/2

･ T < Tc

h(T) ≈ k/(ckT+1)∙2/N = 1/[τ(log2)2(T/(τlog2) + 1)]∙2/N

(0 ≤ T ≤ 1/ck·(exp(cN/2) - 1) = τlog2·(2N/2 - 1)) (A9)

The duration T is thus distributed in the form of 1/T.

･ T > Tc

h(T) ≈ λexp(-λT) (λ≈ 2exp(-cN/2)ck = 1/[2N/2-1∙τ(log2)]) (A10)

The cut-off point Tc increases exponentially with the number of neurons.

Proportion of the duration of the oscillations over the cut-off is inversely proportional to the number N of neurons.

Pr{T < Tc} ∝ 1/N

→ The inverse power-law distribution dominates as N increases.

15

Fig. A2. Probability density function h of the duration T of the transient oscillations in the network with the numbers N of neurons 10 (a), 20 (b), 40 (c).

(a) N=10

- 6

- 5

- 4

- 3

- 2

- 1

0 50 100 150T

log 1

0(h

)simulationEq. (A8)Eq. (A9)Eq. (A10)

(b) N=20

- 6

- 5

- 4

- 3

- 2

0 1000 2000 3000 4000T

log 1

0(h

) simulationEq. (A8)Eq. (A9)Eq. (A10)

(c) N=40

- 10

- 8

- 6

- 4

- 2

0

0 2 4 6 8log10(T)

log 1

0(h

)

simulationEq. (A8)Eq. (A9)Eq. (A10)

h ∝ 1/T

16

The mean m, the variance σ2 and the coefficient of variation CV of the duration T is derived with Eq. (A9).

m(T(N)) = 2(exp(cN/2) - 1 – cN/2)/(c2kN)

= 2τ(2N/2 - 1 - log2/2∙N)/N

σ2(T(N)) = (exp(cN) - 4exp(cN/2) + 3 + cN)/(c3k2N) - {m(T(N)}2

CV(T(N)) = σ(T(N))/m(T(N))

≈ (cN)1/2/2 = (log2)1/2/2∙N1/2 (N » 1) (A11)

Fig. A3. Mean m(T) of the duration of the transient oscillations vs Number N of neurons.

0

1

2

3

4

5

6

0 10 20 30 40N

log 1

0(m

(T))

simulationEq. (A11)

The mean duration m(T) increases exponentially with the number of neurons.

17

5. Conclusion

Properties of the transient oscillations in the ring networks of neurons with inhibitory unidirectional coupling were studied.

･ Kinematical model of the traveling waves in the networks The transient oscillations are the propagating boundaries of the blocks of the neurons at which the signs of the state of neurons are inconsistent. The kinematical equation of the propagating boundaries was derived. The difference between the velocities of the two propagating boundaries is exponentially small with the length of the blocks.

･ Exponential increases in the duration of the oscillations with the number of neuronsThe duration of the transient oscillations increases exponentially with the number of neurons.

･ Power-law distributions of the duration of the oscillations The distribution of the duration of the oscillations is in the form of the inverse power-law below the cut-off.

18

Transient states increasing exponentially with system size

･ Transient spatio-temporal chaos in coupled map lattices [A1] and reaction-diffusion equations [A2] ･ The length and number of cycles and transient time to them in asymmetric neural networks [A3] ･ Transient irregular firings in diluted inhibitory networks of pulse-coupled neurons [A4] ･ Transient well-controlled sequences in continuous-time Hopfield networks with Liapunov functions [A5].

These systems never reach their asymptotically stable states in a practical time when the system size is sufficiently large. Their functions, e.g. information processing in the nervous systems, may proceed in the transient states. The transient states thus play more important roles than the asymptotic states in actual systems.

[8] K. Kaneko, Supertransients, spatiotemporal intermittency and stability of fully developed spatiotemporal chaos, Phys. Lett. A 149 (1990) pp. 105-112. [9] A. Wacker, S. Bose and E. Schöll, Transient spatio-temporal chaos in a reaction-diffusion model, Europhysics Letters 31 (1995) pp. 257-262. [10] U. Bastolla1 and G. Parisi, Relaxation, closing probabilities and transition from oscillatory to chaotic attractors in asymmetric neural networks, J. Phys. A 31 (1998) pp. 4583-4602. [11] R. Zillmera, R. Livib, A. Politic and A. Torcini, Desynchronized stable states in diluted neural networks, Neurocomputing 70 (2007) pp. 1960-1965. [12] J. Šíma and P. Orponen, Exponential transients in continuous-time Liapunov systems, Theoretical Computer Science 306 (2003) pp. 353-372.