Efficient Binomial Channel Capacity Computation

with an Application to Molecular Communication

Richard D. Wesel, Emily E. Wesel,

Lieven Vandenberghe, Christos Komninakis, and

Muriel Medard

1

2018 Information Theory and its Applications Workshop

February 13, 2018

• The Binomial Channel

• Capacity-Achieving Distribution

• Csiszar’s Min-Max Capacity Theorem

• Dynamic Assignment Blahut-Arimoto

• Application to a Molecular Channel

2

Outline

Channel output Y is number of

successes in n Bernoulli trials.

The Binomial Channel

Channel input X is

probability of

success in a

Bernoulli trial.

• The input alphabet is the unit interval, so

it is uncountably infinite.

• …so Blahut-Arimoto is awkward.

• However, the capacity-achieving input

distribution has at most n+1 mass points

from [Witsenhausen, Dubins].

4

Input Alphabet, Capacity Achieving Support

5

Capacity Achieving Distributions

6

Csiszar’s Min-Max Capacity Theorem

This theorem instructs us to find the

capacity-achieving output distribution.

7x

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.05

1.1

1.15

1.2

1.25

1.3

1.35

1.4

D(Py|x||Py) as a function of x for capacity-achieving PX

, n=4

8

A Stopping Criterion

is an upper bound.

For any PX and corresponding PY :

I(X;Y) is a lower bound.

When -I(X;Y) is small

enough, declare victory.

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

D(Py|x||Py) as a function of x for PX

from n-1, n=4, I difference = 0.38

9

10x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.05

1.1

1.15

1.2

1.25

1.3

1.35

1.4D(Py|x||Py)from iteration 1, n=4, I difference = 0.00

Blahut-Arimoto found n=4 capacity

11

Use the n=4 solution for n=5

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5

1.55

1.6

1.65

D(Py|x||Py) as a function of x for PX

from n-1, n=5, I difference = 0.23

12

n=3 solution for n=4, after Blahut-Arimoto

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 0, n=5, I difference = 0.0181

13

Split the center, shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 1, n=5, I difference = 0.0171

14

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 2, n=5, I difference = 0.0144

15

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 3, n=5, I difference = 0.0111

16

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 4, n=5, I difference = 0.0079

17

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 5, n=5, I difference = 0.0052

18

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 6, n=5, I difference = 0.0032

19

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 8, n=5, I difference = 0.0010

20

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 9, n=5, I difference = 0.0005

21

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 10, n=5, I difference = 0.0002

22

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 11, n=5, I difference = 0.0001

23

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 13, n=5, I difference = 0.0000

24

Shift towards maxima

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 14, n=5, I difference = 0.0000

25

Done!

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.2

1.25

1.3

1.35

1.4

1.45

1.5D(Py|x||Py)from iteration 15, n=5, I difference = 0.0000

26

Now for n=6, start with n=5 solution

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.3

1.35

1.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

D(Py|x||Py) as a function of x for PX

from n-1, n=6, I difference = 0.25

27

n=5 solution for n=6, after Blahut-Arimoto

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.3

1.35

1.4

1.45

1.5

1.55

1.6

1.65D(Py|x||Py)from iteration 0, n=6, I difference = 0.1077

28

15 iterations later

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.25

1.3

1.35

1.4

1.45

1.5

1.55

D(Py|x||Py) vs. x for capacity-achieving PX

, n=6, 15 iterations

29

Now for n=9, start with n=8 solution

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.45

1.5

1.55

1.6

1.65

1.7

1.75

1.8

D(Py|x||Py) as a function of x for PX

from n-1, n=9, I difference = 0.12

30

n=8 solution for n=9, after Blahut-Arimoto

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.45

1.5

1.55

1.6

1.65

1.7

1.75D(Py|x||Py)from iteration 0, n=9, I difference = 0.0240

31

Bump in the middle. Need new mass point.

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.45

1.5

1.55

1.6

1.65

1.7

1.75D(Py|x||Py)from iteration 1, n=9, I difference = 0.0171

32

Bump in the middle. Need new mass point.

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.45

1.5

1.55

1.6

1.65

1.7

1.75D(Py|x||Py)from iteration 2, n=9, I difference = 0.0033

33

All is well, with 5 mass points for n=9.

x0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

D(P

y|x

||P

y)

1.45

1.5

1.55

1.6

1.65

1.7

1.75

D(Py|x||Py) vs. x for capacity-achieving PX

, n=9, 41 iterations

34

Capacity Achieving Distributions

35

Dynamic Assignment Blahut-Arimoto (DAB)

36

DAB rocks!

Number of Bernoulli Trials0 5 10 15 20 25

Tim

e in

Secon

ds

0

500

1000

1500Binomial channel capacity computation time using three algorithms

Number of Bernoulli Trials0 5 10 15 20 25

Num

ber

of

Ite

ratio

ns

0

1000

2000

3000

4000

5000

6000

7000

8000Number of Ellipsoid/DAB Iterations for the three algorithms

A Molecular Communication Channel

Transmitter

A Molecular Communication Channel

Generates 𝑛 particles in time 𝜏.

A Molecular Communication Channel

Generates 𝑛 particles in time 𝜏.

Particles selected for release with probability σ.

A Molecular Communication Channel

Generates 𝑛 particles in time 𝜏.

Particles selected for release with probability σ.Selected particles actually released with probability 𝛼.

A Molecular Communication Channel

Generates 𝑛 particles in time 𝜏.

Particles selected for release with probability σ.Selected particles actually released with probability 𝛼.Particles make it to receiver with probability 𝜌.

receiver

Particles detected by receiver with probability β.

Channel output Y is number of

detected particles of n generated.

A Binomial Channel!

Channel input X is

probability of

Success at every

step. Lies on

[0, 𝛼𝜌𝛽]

43

Molecular Communication Capacity Results

44

Molecular Communication Capacity Results

• Dynamic Assignment Blahut-Arimoto uses

Csiszar’s Min-Max Capacity Theorem to

compute capacity in cases where the

alphabet is uncountable but capacity is

achieved by a (small) finite support.

• It’s much faster (and easier) than the

ellipsoid algorithm.

• We used this method to study a molecular

communication channel.45

Conclusions