shannon inequalities in distributed storage - part ivijay/pdfs/2016_shannon...shannon inequalities...
TRANSCRIPT
![Page 1: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/1.jpg)
Shannon Inequalities in Distributed Storage - Part I
Birenjith Sasidharan and P. Vijay Kumar(joint work with Myna Vajha and Kaushik Senthoor)
Department of Electrical Communication Engineering,Indian Institute of Science, Bangalore
Workshop on Advanced Information TheoryCommemorating the 100th Birthday of Claude Shannon
Organized by the IISc-IEEE ComSoc Student Chapter(in association with IEEE Bangalore Section & ECE Department, IISc)
Indian Institute of Science, April 30, 2016
![Page 2: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/2.jpg)
Outline
1 Basic Definitions
2 Sets of Random Variables
3 Entropic Vectors
4 Backup Slides
![Page 3: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/3.jpg)
References
Raymond Yeung, “Facets of Entropy’http://www.inc.cuhk.edu.hk/EII2013/entropy.pdf
![Page 4: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/4.jpg)
Outline
1 Basic Definitions
2 Sets of Random Variables
3 Entropic Vectors
4 Backup Slides
![Page 5: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/5.jpg)
Entropy of a Random Variable
Let X be a discrete random variable taking on values1 from an alphabet X .Then
H(X ) := −∑x∈X
p(x) log p(x)
= −∑x
p(x) log p(x) for simplicity.
Clearly
H(X ) ≥ 0.
1All random variables encountered here will take on values from a finite alphabet.
![Page 6: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/6.jpg)
Joint and Conditional Entropy of 2 Random Variables
If X ,Y are a pair of discrete random variables we define their jointentropy via:
H(X ,Y ) := −∑x ,y
p(x , y) log p(x , y).
We define the conditional entropy of X given Y via:
H(X/Y ) :=∑y
p(y)
{−∑x
p(x/y) log p(x/y)
}= −
∑x ,y
p(x , y) log p(x/y).
Clearly
H(X ,Y ) ≥ 0
H(X/Y ) ≥ 0.
![Page 7: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/7.jpg)
Joint and Conditional Entropy of 2 RV (continued)
Morover
H(X ,Y ) := −∑x ,y
p(x , y) log p(x , y)
= −∑x ,y
p(x , y) log p(y) −∑x ,y
p(x , y) log p(x/y)
= H(Y ) + H(X/Y ).
![Page 8: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/8.jpg)
Mutual Information
The mutual information between X and Y is given by:
I (X ;Y ) := H(X )− H(X/Y )
= −∑x ,y
p(x , y) logp(x)
p(x/y)
= −∑x ,y
p(x , y) logp(x)p(y)
p(x , y)
= H(X ) + H(Y )− H(X ,Y ).
![Page 9: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/9.jpg)
Mutual Information is Non-Negative
Since the function − log(·) is convex, we have that :
I (X ;Y ) := H(X )− H(X/Y )
=∑x ,y
p(x , y){− logp(x)p(y)
p(x , y)}
≥ − log
(∑x ,y
p(x , y)p(x)p(y)
p(x , y)
)
= − log
(∑x ,y
p(x)p(y)
)= 0.
![Page 10: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/10.jpg)
Conditional Mutual Information
The conditional mutual information between X and Y , conditioned on Z , isgiven by:
I (X ;Y /Z ) := H(X/Z )− H(X/Y ,Z )
= −∑z
p(z)
{∑x ,y
p(x , y/z) logp(x/z)
p(x/y , z)
}
= −∑z
p(z)
{∑x ,y
p(x , y/z) logp(x/z)p(y/z)
p(x , y/z)
}
= −∑x ,y
p(x , y , z) logp(x/z)p(y/z)
p(x , y/z).
Clearly
I (X ;Y /Z ) ≥ 0.
![Page 11: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/11.jpg)
Outline
1 Basic Definitions
2 Sets of Random Variables
3 Entropic Vectors
4 Backup Slides
![Page 12: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/12.jpg)
Sets of Random Variables
Let
[n] = {1, 2, · · · , n}X = {X1,X2, · · · ,Xn}
Then if
A = {i1, i2, · · · , i`} ⊆ [n]
we set XA = {xi1 , xi2 , · · · , xi`} ⊆ X .
![Page 13: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/13.jpg)
Basic Inequalities
Can show just as we have shown above that
H(XA) ≥ 0
H(XA/XB) ≥ 0
I (XA/XB) ≥ 0
I (XA;XB/Xc) ≥ 0.
Note that
H(XA/XB) = H(XA∩B ,XA\B/XB)
= H(XA\B/XB),
etc.
![Page 14: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/14.jpg)
Polymatroidal Axioms
These state the following:
H(XA) ≥ 0
H(XA) ≤ H(XB) if A ⊆ B
H(XA) + H(XB) ≥ H(XA∪B) + H(XA∩B).
The first statement, we have already established.
![Page 15: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/15.jpg)
Proof of Monotonicity
To see that
H(XA) ≤ H(XB) if A ⊆ B ,
note that
H(XB)− H(XA) = H(XB\A,XA)− H(XA)
= H(XB\A/XA)
≥ 0.
![Page 16: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/16.jpg)
Proof of Submodularity
To see that
H(XA) + H(XB) ≥ H(XA∪B) + H(XA∩B),
note that
H(XA) + H(XB)− H(XA∪B)− H(XA∩B) ≥ 0
⇔ (H(XA)− H(XA∩B))− (H(XA∪B)− H(XB)) ≥ 0
⇔ H(XA\B/XA∩B)− H(XA\B/XB) ≥ 0
⇔ I (XA\B ;XB\A/XA∩B) ≥ 0
which we know to be true.In turns out that the basic inequalities and the polymatroidal axioms can bederived from each other.
![Page 17: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/17.jpg)
Outline
1 Basic Definitions
2 Sets of Random Variables
3 Entropic Vectors
4 Backup Slides
![Page 18: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/18.jpg)
Entropic Vectors
Let n = 3. Then
X = {X1,X2,X3}
The vector
h(X ) :=
(H(X1), H(X2),H(X3),H(X1,X2),H(X1,X3),H(X2,X3),H(X1,X2,X3))
is called the entropy vector associated to X . Note that in general
h(·) : X → <2n−1.
![Page 19: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/19.jpg)
The Entropic Vector Region
A vector is called entropic if it is the entropy vector associated to someset of RV X
The entropic vector region Γ∗n is the region containing precisely the setof all entropic vectors h(X ) for all possible X .
Both the basic inequalities (or equivalently, the polymatroidal axioms)can be re-expressed in terms of joint entropies
Thus every entropic vector must satisfy the basic inequalities
Does an entropic vector necessarily have to satisfy any otherinequalities ?
![Page 20: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/20.jpg)
Aside: Inequalities that “Always Hold”f(h) � 0 Always holds
!n
f(h) > 0
*
Figure taken from:
Raymond Yeung, “Facets of Entropy’ http://www.inc.cuhk.edu.hk/EII2013/entropy.pdf
![Page 21: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/21.jpg)
Aside: Inequalities that do not “Always Hold”f(h) � 0 Does Not Always holds
f(h) > 0
*!n
. h0
Figure taken from:
Raymond Yeung, “Facets of Entropy’ http://www.inc.cuhk.edu.hk/EII2013/entropy.pdf
![Page 22: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/22.jpg)
The Region Γn
Let Γn denote the region in <2n−1 of vectors that satisfy the basicinequalities. The big question, is:
Γ∗n = Γn?
Turns out that:
Γ∗2 = Γ2
Γ∗3 6= Γ3
but Γ̄∗3 = Γ3
In general,
Γ∗n 6= Γn
![Page 23: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/23.jpg)
The Zhang-Yeung Counter Example when n = 4
Turns out 2
2I (X3;X4) ≤I (X1;X2) + I (X1;X3,X4) + 3I (X3;X4/X1) + I (X3;X4/X2),
is an example of an inequality that is satisfied by all vectors in Γ∗4, but thatthis inequality is not derivable from the inequalities defining Γ4.
2Zhang and Yeung, “On characterization of entropy function via informationinequalities,” T-IT, 1998.
![Page 24: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/24.jpg)
The Consequent Picture for n = 4
!!
!!
"*
ZY98
An Illustration of ZY98
Figure taken from:
Raymond Yeung, “Facets of Entropy’ http://www.inc.cuhk.edu.hk/EII2013/entropy.pdf
![Page 25: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/25.jpg)
Pipenger (1986)
![Page 26: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/26.jpg)
Raymond Yeung
![Page 27: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/27.jpg)
Zhen Zhang and Others
Zhen ZhangUniversity of Southern California
Terence ChanUniversity of South Australia
Imre CsiszárHungarian Academy of Sciences
![Page 28: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/28.jpg)
Outline
1 Basic Definitions
2 Sets of Random Variables
3 Entropic Vectors
4 Backup Slides
![Page 29: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/29.jpg)
Polyhedra
a closed half space:
H+ = {x ∈ V | λ(x) + a0 ≥ 0}.
a subset P ⊆ V is called a polyhedron if it is the intersection of finitelymany closed half spaces.
a polytope is a bounded polyhedron
a subset P ⊆ V is called a cone if it is the intersection of finitely manylinear closed half spaces (i.e., half spaces where a0 = 0)
![Page 30: Shannon Inequalities in Distributed Storage - Part Ivijay/pdfs/2016_Shannon...Shannon Inequalities in Distributed Storage - Part I Birenjith Sasidharan and P. Vijay Kumar (joint work](https://reader031.vdocuments.net/reader031/viewer/2022030717/5b022ce77f8b9a952f8f8252/html5/thumbnails/30.jpg)
Other Results
hZ = hX + hY ∈ Γ∗n; proof use Zi = (Xi ,Yi ), X ,Y independent
Hence for any integer k, kh ∈ Γ∗n
0 ∈ Γ∗n (use the constant random variable)
Γ∗n is convex by appropriate construction of random variables