foundations of probabilities

Foundations of Probabilities

and Information Theory

for Machine Learning

Random Variables

Some proofs

E[X + Y ] = E[X] + E[Y ]where X and Y are random variables of the same type (i.e. either discrete or cont.)

The discrete case:

E[X + Y ] =∑

ω∈Ω

(X(ω) + Y (ω)) · P (ω)

X(ω) · P (ω) +∑

Y (ω) · P (ω) = E[X ] + E[Y ]

The continuous case:

E[X + Y ] =

(x+ y)pXY (x, y)dydx

xpXY (x, y)dydx+

ypXY (x, y)dydx

pXY (x, y)dydx+

pXY (x, y)dxdy

xpX(x)dx +

ypY (y)dy = E[X ] + E[Y ]

X and Y are independent ⇒ E[XY ] = E[X] · E[Y ],X and Y being random variables of the same type (i.e. either discrete or continuous)

The discrete case:

E[XY ] =∑

x∈V al(X)

y∈V al(Y )

xyP (X = x, Y = y) =∑

x∈V al(X)

y∈V al(Y )

xyP (X = x) · P (Y = y)

x∈V al(X)

xP (X = x)∑

y∈V al(Y )

yP (Y = y)

x∈V al(X)

xP (X = x)E[Y ] = E[X ] · E[Y ]

The continuous case:

E[XY ] =

xy p(X = x, Y = y)dydx =

xy p(X = x) · p(Y = y)dydx

x p(X = x)

y p(Y = y)dy

x p(X = x)E[Y ]dx

= E[Y ] ·∫

x p(X = x)dx = E[X ] ·E[Y ]

Binomial distribution: b(r;n, p)def.= Cr

n pr(1− p)n−r

Significance: b(r;n, p) is the probability of drawing r heads in n independentflips of a coin having the head probability p.

b(r;n, p) indeed represents a probability distribution:

• b(r;n, p) = Crn p

r(1− p)n−r ≥ 0 for all p ∈ [0, 1], n ∈ N and r ∈ 0, 1, . . . , n,•

r=0 b(r;n, p) = 1:

(1− p)n + C1np(1− p)n−1 + · · ·+ Cn−1

n pn−1(1− p) + pn = [p + (1− p)]n = 1

Binomial distribution: calculating the mean

E[b(r;n, p)]def.=

r · b(r;n, p) =

= 1 · C1np(1− p)n−1 + 2 · C2

np2(1− p)n−2 + · · ·+ (n− 1) · Cn−1

n pn−1(1− p) + n · pn= p

n(1− p)n−1 + 2 · C2np(1− p)n−2 + · · ·+ (n− 1) · Cn−1

n pn−2(1− p) + n · pn−1]

= np[(1− p)n−1 + C1

n−1p(1− p)n−2 + · · ·+ Cn−2n−1p

n−2(1− p) + Cn−1n−1p

n−1]

= np[p + (1− p)]n−1 = np

For the (1) equality we used the following property:

k Ckn = k

k! (n− k)! =n!

(k − 1)! (n− k)! =n (n− 1)!

(k − 1)! (n− 1− (k − 1))!

= n Ck−1n−1, ∀k = 1, . . . , n.

Binomial distribution: calculating the variancefollowing www.proofwiki.org/wiki/Variance of Binomial Distribution, which cites

“Probability: An Introduction”, by Geoffrey Grimmett and Dominic Welsh,

Oxford Science Publications, 1986

We will make use of the formula Var[X ] = E[X2]−E2[X ].By denoting q = 1− p, it follows:

E[b2(r;n, p)]def.=

r2Crnp

rqn−r =

r2n(n− 1) . . . (n− r + 1)

r!prqn−r

rn(n− 1) . . . (n− r + 1)

(r − 1)!prqn−r =

rnCr−1n−1 p

rqn−r

= npn∑

r Cr−1n−1 p

r−1q(n−1)−(r−1)

Binomial distribution: calculating the variance (cont’d)

By denoting j = r − 1 and m = n− 1, we’ll get:

E[b2(r;n, p)] = npm∑

(j + 1)Cjm pjqm−j

j Cjm pjqm−j +

Cjm pjqm−j

jm · . . . · (m− j + 1)

j!pjqm−j + (p+ q

︸︷︷︸1

mCj−1m−1 p

jqm−j + 1

Cj−1m−1 p

j−1q(m−1)−(j−1) + 1

= np[(n− 1)p(p+ q︸︷︷︸

)m−1 + 1] = np[(n− 1)p+ 1] = n2p2 − np2 + np

Finally,

Var[X ] = E[b2(r;n, p)] − (E[b(r;n, p)])2 = n2p2 − np2 + np− n2p2 = np(1− p)

Binomial distribution: calculating the variance

Another solution

• se demonstreaza relativ usor ca orice variabila aleatoare urmanddistributia binomiala b(r;n, p) poate fi vazuta ca o suma de n vari-abile independente care urmeaza distributia Bernoulli de parametrup;a

• stim (sau, se poate dovedi imediat) ca varianta distributiei Bernoullide parametru p este p(1− p);

• tinand cont de proprietatea de liniaritate a variantelor — Var[X1 +X2 . . . + Xn] = Var[X1] + Var[X2] . . . + Var[Xn], daca X1, X2, . . . , Xn suntvariabile independente —, rezulta ca Var[X ] = np(1− p).

aVezi www.proofwiki.org/wiki/Bernoulli Process as Binomial Distribution, care citeaza de asemenea ca sursa “Proba-bility: An Introduction” de Geoffrey Grimmett si Dominic Welsh, Oxford Science Publications, 1986.

The Gaussian distribution: p(X = x) = 1√2πσe− (x− µ)2

Calculating the mean: E[Nµ,σ(x)]def.=∫∞−∞ xp(x)dx = 1√

∫∞−∞ x · e

−(x− µ)2

2σ2 dx

Using the variable transformation v =x− µσ

will imply x = σv + µ and dx = σdv, so:

E[X ] =1√2πσ

∫ ∞

−∞(σv + µ)e

2 (σdv) =σ√2πσ

∫ ∞

−∞ve

2 dv + µ

∫ ∞

−∞e−v2

=1√2π

∫ ∞

−∞(−v)e

2 dv + µ

∫ ∞

−∞e−v2

1√2π

−σ e−v2

∣∣∣∣∣∣∣

−∞︸︷︷︸

∫ ∞

−∞e−v2

=µ√2π

∫ ∞

−∞e−v2

2 dv (see the next slide for the computation on this last integral)

=µ√2π

√2π = µ

The Gaussian distribution: calculating the mean (Cont’d)

v=−∞

x=−∞

y=−∞

x=−∞

y=−∞

x2 + y2

2 dydx

∫∫

x2 + y2

2 dydx

By switching from x, y to polar coordinates r, θ (see the Note below), it follows:

v=−∞

2 (rdrdθ) =

2 θ|2π0 dr

2 dr = 2π (−e−

= 2π(0− (−1)) = 2π ⇒∫

v=−∞

2 dv =√2π.

Note: x = r cos θ and y = r sin θ, with r ≥ 0 and θ ∈ [0, 2π). Therefore, x2 + y2 = r2, and theJacobian matrix is

∂(x, y)

∂(r, θ)=

cos θ −r sin θ

sin θ r cos θ

= r cos2 θ + r sin2 θ = r ≥ 0. So, dxdy = rdrdθ.

Bivariate Gaussian

The Gaussian distribution: calculating the variance

We will make use of the formula Var[X ] = E[X2]− E2[X ].

E[X2] =

∫ ∞

−∞x2p(x)dx =

1√2πσ

∫ ∞

−∞x2 · e

−(x− µ)2

2σ2 dx

Again, using the transformation v =x− µσ

will imply x = σv+µ and dx = σdv. Therefore,

E[X2] =1√2πσ

∫ ∞

−∞(σv + µ)2 e

2 (σdv)

=σ√2πσ

∫ ∞

−∞(σ2v2 + 2σµv + µ2) e

=1√2π

∫ ∞

−∞v2 e

2 dv + 2σµ

∫ ∞

−∞v e

2 dv + µ2

∫ ∞

−∞e−v2

Note that we have already computed∫∞−∞ ve

2 dv = 0 and∫∞−∞ e

2 dv =√2π.

The Gaussian distribution: calculating the variance (Cont’d)

Therefore, we only need to compute

∫ ∞

−∞v2e

2 dv =

∫ ∞

−∞(−v)

∫ ∞

−∞(−v)

= (−v) e−v2

∣∣∣∣∣∣∣

−∞

−∫ ∞

−∞(−1)e

2 dv = 0 +

∫ ∞

−∞e−v2

2 dv =√2π.

Here above we used the fact that

limv→∞

v e−v2

2 = limv→∞

l’Hopital= lim

v→∞1

= 0 = limv→−∞

v e−v2

So, E[X2] = 1√2π

(σ2√2π + 2σµ · 0 + µ2

√2π)= σ2 + µ2.

And, finally, Var[X ] = E[X2]− (E[X ])2 = (σ2 + µ2)− µ2 = σ2.

Vectors of random variables.

A property:

The covariance matrix Σ corresponding to such a vector issymmetric and positive semi-definite

Chuong Do, Stanford University, 2008

[adapted by Liviu Ciortuz]

Fie variabilele aleatoare X1, . . . , Xn, cu Xi : Ω → R pentrui = 1, . . . , n. Matricea de covarianta a vectorului de variabilealeatoare X = (X1, . . . , Xn) este o matrice patratica de dimensiune

n×n, ale carei elemente se definesc astfel: [Cov(X)]ijdef.= Cov(Xi, Xj),

pentru orice i, j ∈ 1, . . . , n.

Aratati ca Σnot.= Cov(X) este matrice simetrica si pozitiv semi-

definita, cea de-a doua proprietate ınsemnand ca pentru oricevector z ∈ Rn are loc inegalitatea z⊤Σz ≥ 0. (Vectorii z ∈ Rn suntconsiderati vectori-coloana, iar simbolul ⊤ reprezinta operatia detranspunere de matrice.)

Cov(X)i,jdef.= Cov(Xi, Xj), for all i, j ∈ 1, . . . , n, and

Cov(Xi, Xj)def.= E[(Xi − E[Xi])(Xj − E[Xj ])] = E[(Xj − E[Xj ])(Xi − E[Xi])] = Cov(Xj , Xi),

therefore Cov(X) is a symmetric matrix.

We will show that zTΣz ≥ 0 for any z ∈ Rn (seen as a column-vector):

zTΣz =

Σijzj) =

(ziΣijzj) =

(ziCov[Xi, Xj] zj)

(zi E[(Xi − E[Xi])(Xj − E[Xj])] zj) = E

zi (Xi − E[Xi])(Xj − E[Xj ]) zj

zi (Xi − E[Xi])

(Xj − E[Xj ]) zj

(Xi − E[Xi]) zi

(Xj − E[Xj ]) zj

= E[((X − E[X ])T · z)2] ≥ 0

Multi-variate Gaussian distributions:

A property:

When the covariance matrix of a multi-variate (d-dimensional) Gaussian

distribution is diagonal, then the p.d.f. (probability density function) of

the respective multi-variate Gaussian is equal to the product of d

independent uni-variate Gaussian densities.

Chuong Do, Stanford University, 2008

Let’s consider X = [X1 . . . Xd]T , µ ∈ Rd and Σ ∈ Sd+, where Sd+ is the set of symmetric

positive definite matrices (which implies |Σ| 6= 0 and (x − µ)TΣ−1(x − µ) > 0, therefore− 1

2 (x− µ)TΣ−1(x− µ) < 0, for any x ∈ Sd, x 6= µ).

The probability density function of a multi-variate Gaussian distribution of parametersµ and Σ is:

p(x;µ,Σ) =1

(2π)d/2|Σ|1/2 exp

2(x − µ)TΣ−1(x − µ)

Notation: X ∼ N (µ,Σ).Show that when the covariance matrice Σ is diagonal, then the p.d.f. (probability den-sity function) of the respective multi-variate Gaussian is equal to the product of dindependent uni-variate Gaussian densities.

We will make the proof for d = 2(generalization to d > 2 will be easy):

[σ21 0

0 σ22

Note: It is easy to show that if Σ ∈ Sd+ is diagonal, the

elements on the principal diagonal Σ are indeed strictly

positive. (It is enough to consider z = (1, 0) and respec-

tively z = (0, 1) in formula for pozitive-definiteness of Σ.)

This is why we wrote these elements of σ as σ21 and σ2

p(x;µ,Σ) =1

∣∣∣∣

σ21 0

0 σ22

∣∣∣∣

[x1 − µ1

x2 − µ2

]T [σ21 0

0 σ22

]−1 [x1 − µ1

x2 − µ2

2π σ1σ2exp

[x1 − µ1

x2 − µ2

[x1 − µ1

x2 − µ2

2π σ1σ2exp

[x1 − µ1

x2 − µ2

(x1 − µ1)

(x2 − µ2)

2π σ1σ2exp

(x1 − µ1)2 − 1

(x2 − µ2)2

= p(x1;µ1, σ21) p(x2;µ2, σ

Bi-variate Gaussian distributions. A property:

The conditional distributions X1|X2 and X2|X1 are alsoGaussians.

The calculation of their parameters

Duda, Hart and Stork, Pattern Classification, 2001,Appendix A.5.2

Fie X o variabila aleatoare care urmeaza o distributie gaussiana bi-variatade parametri µ (vectorul de medii) si Σ (matricea de covarianta). Asadar,µ = (µ1, µ2) ∈ R2, iar Σ ∈M2×2(R).

Prin definitie, Σ = Cov(X,X), unde Xnot.= (X1, X2), asadar Σij = Cov(Xi, Xj)

pentru i, j ∈ 1, 2. De asemenea, Cov(Xi, Xi) = Var[Xi]not.= σ2

i ≥ 0 pentru

i ∈ 1, 2, ın vreme ce pentru i 6= j avem Cov(Xi, Xj) = Cov(Xj , Xi)not.= σij.

In sfarsit, daca introducem ,,coeficientul de corelare“ ρdef.=

σ12σ1σ2

, rezulta ca

putem scrie astfel matricea de covarianta:

[σ21 ρσ1σ2

ρσ1σ2 σ22

Demonstrati ca ipoteza X ∼ N (µ,Σ), im-plica faptul ca distributia conditionalaX2|X1 este de tip gaussian, si anume

X2|X1 = x1 ∼ N (µ2|1, σ22|1),

cu µ2|1 = µ2+ρσ2σ1

(x1−µ1) si σ22|1 = σ2

2(1−ρ2).

Observatie: Pentru X1|X2, rezultatuleste similar: X1|X2 = x2 ∼ N (µ1|2, σ

21|2), cu

µ1|2 = µ1 + ρσ1σ2

(x2 − µ2) si σ21|2 = σ2

1(1− ρ2).

Source:

Pattern Classification, Appendix A.5.2,

Duda, Hart and Stork, 2001

Answer

pX2|X1(x2|x1) def.

=pX1,X2

(x1, x2)

pX1(x1)

pX1,X2(x1, x2) =

(√2π)2

|Σ|exp

2(x− µ)⊤Σ−1(x− µ)

pX1(x1) =

1√2πσ1

(x1 − µ1)2

From (2) it follows that |Σ| = σ21σ

22(1 − ρ2). In order that

|Σ| and Σ−1 be defined, it

follows that ρ ∈ (−1, 1). Moreover, since σ1, σ2 > 0, we will have√

|Σ| = σ1σ2√

1− ρ2.

Σ−1 =1

σ21σ

22(1− ρ2)

Σ∗ =1

σ21σ

22(1 − ρ2)

[σ22 −ρσ1σ2

−ρσ1σ2 σ21

(1 − ρ2)

− ρ

σ1σ2

− ρ

σ1σ2

pX1,X2(x1, x2) =

2πσ1σ2√

1− ρ2exp

2(1− ρ)2 (x1 − µ1, x2 − µ2)

− ρ

σ1σ2

− ρ

σ1σ2

(x1 − µ1

x2 − µ2

2πσ1σ2√

1− ρ2·

2(1− ρ2)

[(x1 − µ1

− 2ρ

(x1 − µ1

)(x2 − µ2

(x2 − µ2

By substitution (4) and (5) in the definition (3), we will get:

p(x2|x1) =pX1,X2

(x1, x2)

pX1(x1)

2πσ1σ2√

1− ρ2· exp

2(1− ρ2)

[(x1 − µ1

− 2ρ

(x1 − µ1

)(x2 − µ2

(x2 − µ2

·√2πσ1 exp

(x1 − µ1

2πσ2√

1− ρ2exp

2(1− ρ2)

(x2 − µ2

σ2− ρx1 − µ1

2πσ2√

1− ρ2exp

x2 − [µ2 + ρσ2σ1

(x1 − µ1)]

σ2√

1− ρ2

Therefore,

X2|X1 = x1 ∼ N (µ2|1, σ22|1) with µ2|1

not.= µ2 + ρ

σ2σ1

(x1 − µ1) and σ22|1

not.= σ2

2(1− ρ2).

Using the Central Limit Theorem (the i.i.d. version)

to compute the real error of a classifier

CMU, 2008 fall, Eric Xing, HW3, pr. 3.3

Chris recently adopts a new (binary) classifier to filter emailspams. He wants to quantitively evaluate how good the classi-fier is.

He has a small dataset of 100 emails on hand which, you canassume, are randomly drawn from all emails.

He tests the classifier on the 100 emails and gets 83 classifiedcorrectly, so the error rate on the small dataset is 17%.

However, the number on 100 samples could be either higher orlower than the real error rate just by chance.

With a confidence level of 95%, what is likely to be the range ofthe real error rate? Please write down all important steps.

(Hint: You need some approximation in this problem.)

Notations:

Let Xi, i = 1, . . . , n = 100 be defined as:Xi = 1 if the email i was incorrectly classified, and 0 otherwise;

E[Xi]not.= µ

not.= ereal ; Var(Xi)

not.= σ2

esamplenot.=

X1 + . . .+Xn

n= 0.17

Zn =X1 + . . .+Xn − nµ√

n σ(the standardized form of X1 + . . .+Xn)

Key insight:

Calculating the real error of the classifier (more exactly, a symmetric interval

around the real error pnot.= µ) with a “confidence” of 95% amounts to finding a > 0

sunch that P (|Zn| ≤ a) ≥ 0.95.

Calculus:

| Zn |≤ a ⇔∣∣∣∣

X1 + . . .+Xn − nµ√n σ

∣∣∣∣≤ a ⇔

∣∣∣∣

X1 + . . .+Xn − nµnσ

∣∣∣∣≤ a√

⇔∣∣∣∣

X1 + . . .+Xn − nµn

∣∣∣∣≤ aσ√

n⇔∣∣∣∣

X1 + . . .+Xn

n− µ

∣∣∣∣≤ aσ√

⇔ |esample − ereal| ≤aσ√n⇔ |ereal − esample| ≤

aσ√n

⇔ − aσ√n≤ ereal − esample ≤

aσ√n

⇔ esample −aσ√n≤ ereal ≤ esample +

aσ√n

⇔ ereal ∈[

esample −aσ√n, esample +

aσ√n

Important facts:

The Central Limit Theorem: Zn → N (0; 1)

Therefore, P (|Zn| ≤ a) ≈ P (|X | ≤ a) = Φ(a)− Φ(−a), where X ∼ N (0; 1)

and Φ is the cumulative function distribution of N (0; 1).

Calculus:

Φ(−a) + Φ(a) = 1⇒ P (| Zn |≤ a) = Φ(a)− Φ(−a) = 2Φ(a)− 1

P (| Zn |≤ a)=0.95⇔2Φ(a)− 1=0.95⇔Φ(a)=0.975⇔ a ∼= 1.97 (see Φ table)

σ2 not.= Varreal = ereal(1− ereal) because Xi are Bernoulli variables.

Futhermore, we can approximate ereal with esemple, because

E[esample] = ereal and Varsample =1

nVarreal → 0 for n→ +∞,

cf. CMU, 2011 fall, T. Mitchell, A. Singh, HW2, pr. 1.ab.

Finally:

⇒ aσ√n≈ 1.97 ·

0.17(1− 0.17)√100

∼= 0.07

|ereal − esample| ≤ 0.07 ⇔ |ereal − 0.17| ≤ 0.07 ⇔ −0.07 ≤ ereal − 0.17 ≤ 0.07

⇔ ereal ∈ [0.10, 0.24]

Exemplifying

a mixture of categorical distributions;

how to compute its expectation and variance

CMU, 2010 fall, Aarti Singh, HW1, pr. 2.2.1-2

Suppose that I have two six-sided dice, one is fair and the other one isloaded – having:

P (x) =

2x = 6

10x ∈ 1, 2, 3, 4, 5

I will toss a coin to decide which die to roll. If the coin flip is heads I willroll the fair die, otherwise the loaded one. The probability that the coinflip is heads is p ∈ (0, 1).

a. What is the expectation of the die roll (in terms of p).

b. What is the variation of the die roll (in terms of p).

Solution:

E[X ] =6∑

i · [P (i|fair) · p+ P (i|loaded) · (1− p)]

i · P (i|fair)]

i · P (i|loaded)]

(1− p)

2(1− p) = 9

2− p

b. Recall that we may write Var (X) = E[X2]− (E[X ])2, therefore:

E[X2] =

i2 · [P (i|fair) · p+ P (i|loaded) · (1− p)]

i2 · P (i|fair)]

i2 · P (i|loaded)]

(1− p)

10)(1− p)

2− 25

Combining this with the result of the previous question yields:

Var (X) = E[X2]− (E[X ])2 =141

6− 50

6p− (

2− p)2

6− 50

6p− (

4− 9p+ p2)

= (141

6− 81

4)− (

6− 9)p− p2

3p− p2

Elements of Information Theory:Some examples and then some useful proofs

Computing entropies and specific conditional entropies

for discrete random variables

CMU, 2012 spring, R. Rosenfeld, HW2, pr. 2

On the roll of two six-sided fair dice,

a. Calculate the distribution of the sum (S) of the total.

b. The amount of information (or surprise) when seeing the outcome x

for a random variable X is defined as log21

P (X = x)= − log2 P (X = x). How

surprised are you (in bits) to observe S = 2, S = 11, S = 5, S = 7?

c. Calculate the entropy of S [as the expected value of the random variable− log2 P (X = x)].

d. Let’s say you throw the die one by one, and the first die shows 4. Whatis the entropy of S after this observation? Was any information gained / lostin the process? If so, calculate how much information (in bits) was lost orgained.

S 2 3 4 5 6 7 8 9 10 11 12P (S) 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36

Information(S = 2) = − log2(1/36) = log2 36 = 2 log2 6 = 2(1 + log2 3)

= 5.169925001 bits

Information(S = 11) = − log2 2/36 = log2 18 = 1 + 2 log2 3 = 4.169925001 bits

Information(S = 5) = − log2 4/36 = log2 9 = 2 log2 3 = 3.169925001 bits

Information(S = 7) = − log2 6/36 = log2 6 = 1 + log2 3 = 2.584962501 bits

H(S) = −n∑

pi log pi

= −(

2 · 136

36+ 2 · 2

36+ 2 · 3

36+ 2 · 4

2 · 536

36(2 log2 36 + 4 log2 18 + 6 log2 12 + 8 log2 9 + 10 log2

5+ 6 log2 6)

36(2 log2 6

2 + 4 log2 6 · 3 + 6 log2 6 · 2 + 8 log2 32 + 10 log2

5+ 6 log2 6)

36(40 log2 6 + 20 log2 3 + 6− 10 log2 5)

36(60 log2 3 + 46− 10 log2 5) = 3.274401919 bits.

S 2 3 4 5 6 7 8 9 10 11 12P (S|...) 0 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0

H(S|First-die-shows-4) = −6 · 16log2

6= log2 6 = 2.58 bits,

IG(S;First-die-shows-4) = H(S)−H(S|First-die-shows-4) = 3.27− 2.58 = 0.69 bits.

Computing entropies and mean conditional entropies

for discrete random variables

CMU, 2012 spring, R. Rosenfeld, HW2, pr. 3

A doctor needs to diagnose a person having cold (C). The primary factorhe considers in his diagnosis is the outside temperature (T ). The randomvariable C takes two values, yes / no, and the random variable T takes 3values, sunny, rainy, snowy. The joint distribution of the two variables isgiven in following table.

T = sunny T = rainy T = snowy

C = no 0.30 0.20 0.10C = yes 0.05 0.15 0.20

a. Calculate the marginal probabilities P (C), P (T ).

Hint: Use P (X = x) =∑

Y P (X = x;Y = y). For example,

P (C = no) = P (C = no, T = sunny) + P (C = no, T = rainy) + P (C = no, T = snowy).

b. Calculate the entropies H(C), H(T ).

c. Calculate the mean conditional entropies H(C|T ), H(T |C).

a. PC = (0.6, 0.4) si PT = (0.35, 0.35, 0.30).

H(C) = 0.6 log5

3+ 0.4 log

2= log 5− 0.6 log 3− 0.4 = 0.971 bits

H(T ) = 2 · 0.35 log 20

7+ 0.3 log

3= 0.7(2 + log 5− log 7) + 0.3(1 + log 5− log 3)

= 1.7 + log 5− 0.7 log 7− 0.3 log 3 = 1.581 bits.

H(C|T ) def.=

t∈Val (T )

P (T = t) ·H(C|T = t)

= P (T = sunny) ·H(C|T = sunny) + P (T = rainy) ·H(C|T = rainy) +

P (T = snowy) ·H(C|T = snowy)

= 0.35 ·H(

0.30 + 0.05,

0.30 + 0.05

+ 0.35 ·H(

0.20 + 0.15,

0.20 + 0.15

0.30 ·H(

0.10 + 0.20,

0.20 + 0.10

20·H(6

20·H(4

10·H(1

20·(6

7log 7

20·(4

10·(1

3log 3 +

log 7− 6

7− 6

7log 3

log 7− 8

7− 3

7log 3

log 3− 2

10log 7−

20− 3

· log 3 =7

10log 7− 3

20log 3− 9

10= 0.82715 bits.

H(T |C) def.=

c∈Val (C)

P (C = c) ·H(T |C = c)

= P (C = no) ·H(T |C = no) + P (C = yes) ·H(T |C = yes)

= 0.60 ·H(

0.30 + 0.20 + 0.10,

0.30 + 0.20 + 0.10

0.40 ·H(

0.05 + 0.15 + 0.20,

0.05 + 0.15 + 0.20

5·H(1

3log 3 +

6(1 + log 3)

8· 3 + 3

8(3− log 3) +

2log 3

2− 3

8log 3

20log 3 = 1.43774 bits.

Computing the entropy of theexponential distribution

CMU, 2011 spring, R. Rosenfeld,

HW2, pr. 2.c

xxxxxx Exponential p.d.f.

0 1 2 3 4 5

θ = 0.5θ = 1θ = 1.5

Pentru o distributie de probabilitate continua P , entropia se defineste ast-fel:

H(P ) =

∫ +∞

−∞

P (x) log21

P (x)dx

Calculati entropia distributiei continue exponentiale de parametru λ > 0.Definitia acestei distributii este urmatoarea:

P (x) =

λe−λx, daca x ≥ 0;0, daca x < 0.

Indicatie: Daca P (x) = 0, veti presupune ca −P (x) log2 P (x) = 0.

Answer

H(P ) =

−∞P (x) log2

P (x)dx+

∫ ∞

P (x) log21

P (x)dx

def. P=

−∞0 log2 0 dx

︸︷︷︸0

∫ ∞

λe−λx log21

λe−λxdx =

∫ ∞

λe−λx log21

λe−λxdx

⇒ H(P ) =1

∫ ∞

λe−λx ln1

λe−λxdx =

∫ ∞

λe−λx

λ+ ln

e−λx

∫ ∞

λe−λx(− lnλ+ ln eλx

∫ ∞

λe−λx (− lnλ+ λx) dx

∫ ∞

λe−λx(− lnλ) dx+1

∫ ∞

λe−λxλxdx

=− lnλ

∫ ∞

λe−λx dx+λ

∫ ∞

λe−λxx dx

∫ ∞

(e−λx

)′dx− λ

∫ ∞

(e−λx

)′x dx

Prima integrala se rezolva foarte usor:

∫ ∞

(e−λx

)′dx = e−λx

∣∣∞0

= e−∞ − e0 = 0− 1 = −1

Pentru a rezolva cea de-a doua integrala se poate folosi formula de integrare prin parti:

∫ ∞

(e−λx

)′x dx = e−λxx

∣∣∞0−∫ ∞

e−λxx′ dx = e−λxx∣∣∞0−∫ ∞

e−λx dx

Integrala definita e−λxx∣∣∞0

nu se poate calcula direct (din cauza conflictului 0 · ∞ carese produce atunci cand lui x i se atribuie valoarea-limita ∞), ci se calculeaza folosindregula lui l’Hopital:

limx→∞

xe−λx = limx→∞

eλx= lim

x→∞x′

(eλx)′ = lim

x→∞1

λeλx=

λlimx→∞

e−λx = e−∞ = 0,

decie−λxx

∣∣∞0

= 0− 0 = 0.

Integrala∫∞0 e−λx dx se calculeaza usor:

∫ ∞

e−λx dx = − 1

∫ ∞

(e−λx

)′dx = − 1

λe−λx

∣∣∞0

= − 1

λ(0− 1) =

Prin urmare,∫ ∞

(e−λx

)′x dx = 0− 1

λ= − 1

ceea ce conduce la rezultatul final:

H(P ) =ln λ

ln 2(−1)− λ

= − lnλ

1− lnλ

Derivation of entropy definition,

starting from a set of desirable properties

CMU, 2005 fall, T. Mitchell, A. Moore, HW1, pr. 2.2

Remark: The definition we gave for entropy−∑ni=1 pi log pi is not very intuitive.

Theorem:

If ψn(p1, . . . , pn) satisfies the following axioms

A0. [LC:] ψn(p1, . . . , pn) ≥ 0 for any n ∈ N∗ and p1, . . . , pn, since we view ψn is a

measure of disorder ; also, ψ1(1) = 0 because in this case there is no disorder;

A1. ψn should be continuous in pi and symmetric in its arguments;

A2. if pi = 1/n then ψn should be a monotonically increasing function of n;(If all events are equally likely, then having more events means beingmore uncertain.)

A3. if a choice among N events is broken down into successive choices, thenthe entropy should be the weighted sum of the entropy at each stage;

then ψn(p1, . . . , pn) = −K∑

i pi log pi where K is a positive constant.

(As we’ll see, K depends however on ψs

s, . . . ,

for a certain s ∈ N∗).

Remark: We will prove the theorem firstly for uniform distributions (pi = 1/n) and secondly

for the case pi ∈ Q (only!).

Example for the axiom A3:

Encoding 1

(a,b,c)

1/2 1/3 1/6

Encoding 2

(a,b,c)

1/2 1/2

2/3 1/3

2log 2 +

3log 3 +

6log 6 =

log 2 +

log 3 =2

2log 3

= 1 +1

3log 3

= 1 +1

log 3− 2

2log 3

The next 3 slides:

Case 1: pi = 1/n for i = 1, . . . , n; proof steps:

a. A(n)not.= ψ(1/n, 1/n, . . . , 1/n) implies

A(sm) = mA(s) for any s,m ∈ N∗. (1)

b. If s,m ∈ N⋆ (fixed), s 6= 1, and t, n ∈ N⋆ such that sm ≤ tn ≤ sm+1, then

| mn −log tlog s |≤ 1

n . (2)

c. For sm ≤ tn ≤ sm+1 as above, due to A2 it follows (immediately)

sm, . . . ,

≤ ψtn

tn, . . . ,

≤ ψsm+1

sm+1, . . . ,

i.e. A(sm) ≤ A(tn) ≤ A(sm+1)c. Show that

| mn −A(t)A(s) |≤ 1

n for s 6= 1. (3)

d. Combining (2) + (3) immediately gives

| A(t)A(s) −log tlog s |≤ 2

n pentru s 6= 1 (4)

d. Show that this inequation implies

A(t) = K log t with K > 0 (due to A2). (5)

Proofa.

. . . . . .. . .

1/s 1/s 1/s

nivel 2

nivel 1

1/s 1/s 1/s

nivel m

de s ori

. . .de s ori

de s ori

de s orim

Applying the axion A3 on the right encoding from above gives:

A(sm) = A(s) + s · 1sA(s) + s2 · 1

s2A(s) + . . .+ sm−1 · 1

sm−1A(s)

= A(s) +A(s) +A(s) + . . .+A(s)︸︷︷︸

m times

= mA(s)

Proof (cont’d)

sm ≤ tn ≤ sm+1 ⇒ m log s ≤ n log t ≤ (m+ 1) log s⇒m

n≤ log t

log s≤ m

n⇒ 0 ≤ log t

log s− m

n≤ 1

n⇒∣∣∣∣

log s− m

∣∣∣∣≤ 1

A(sm) ≤ A(tn) ≤ A(sm+1)(1)⇒ mA(s) ≤ n A(t) ≤ (m+ 1) A(s)

s 6=1⇒m

n≤ A(t)

A(s)≤ m

n⇒ 0 ≤ A(t)

A(s)− m

n≤ 1

n⇒∣∣∣∣

A(s)− m

∣∣∣∣≤ 1

d. Consider again sm ≤ tn ≤ sm+1 with s, t fixed. If m→∞ then n→∞ and

∣∣∣∣

A(s)− log t

∣∣∣∣≤ 2

nit follows that

∣∣∣∣

A(s)− log t

∣∣∣∣→ 0.

Therefore

∣∣∣∣

A(s)− log t

∣∣∣∣= 0 and so

log s.

Finally, A(t) =A(s)

log slog t = K log t, where K =

log s> 0 (if s 6= 1).

Case 2: pi ∈ Q for i = 1, . . . , n

Let’s consider a set of N ≥ 2equiprobable random events, and P =(S1, S2, . . . , Sk) a partition of this set.Let’s denote pi =| Si | /N .

A “natural” two-step ecoding (asshown in the nearby figure) leads toA(N) = ψk(p1, . . . , pk) +

i piA(| Si |),based on the axiom A3.

Finally, using the result A(t) = K log t,gives:

K logN = ψk(p1, . . . , pk) +K∑

i pi log | Si |

1/|S |i1/|S |i 1/|S |i

|S |/N2 |S |/Ni

S 1 S 2 S k

|S |/Nk|S |/N1

level 1

level 2

. . .. . .

. . . . . . . . .

⇒ ψk(p1, . . . , pk) = K[ logN −∑

pi log | Si | ]

= K[ logN∑

pi −∑

pi log | Si | ] = −K∑

pi log| Si |N

= −K∑

pi log pi

Entropie, entropie corelata,

entropie conditionala, castig de informatie:

definitii si proprietati imediate

CMU, 2005 fall, T. Mitchell, A. Moore, HW1, pr. 2

Definitii

• Entropia variabilei X:

H(X)def.= −∑i P (X = xi) logP (X = xi)

not.= EX [− logP (X)].

• Entropia conditionala specifica a variabilei Y ın raport cu valoarea xk a variabilei X:

H(Y | X = xk)def.= −∑j P (Y = yj | X = xk) logP (Y = yj | X = xk)

not.= EY |X=xk

[− logP (Y | X = xk)].

• Entropia conditionala medie a variabilei Y ın raport cu variabila X:

H(Y | X)def.=∑

k P (X = xk)H(Y | X = xk)not.= EX [H(Y | X)].

• Entropia corelata a variabilelor X si Y :

H(X,Y )def.= −∑i

j P (X = xi, Y = yj) logP (X = xi, Y = yj)not.= EX,Y [− logP (X,Y )].

• Informatia mutuala a variabilelor X si Y , numita de asemenea castigul de informatieal variabilei X ın raport cu variabila Y (sau invers):

MI(X,Y )not.= IG(X,Y )

def.= H(X)−H(X | Y ) = H(Y )−H(Y | X)

(Observatie: ultima egalitate de mai sus are loc datorita rezultatului de la punctulc de mai jos.)

a.H(X) ≥ 0.

H(X) = −∑

P (X = xi) logP (X = xi) =∑

P (X = xi)︸︷︷︸

Mai mult, H(X) = 0 daca si numai daca variabila X este constanta:

,,⇒“ Presupunem ca H(X) = 0, adica∑

i P (X = xi) log1

P (X = xi)= 0. Datorita faptului

ca fiecare termen din aceasta suma este mai mare sau egal cu 0, rezulta ca H(X) = 0

doar daca pentru ∀i, P (X = xi) = 0 sau log1

P (X = xi)= 0, adica daca pentru ∀i,

P (X = xi) = 0 sau P (X = xi) = 1. Cum ınsa∑

i P (X = xi) = 1 rezulta ca exista osingura valoare x1 pentru X astfel ıncat P (X = x1) = 1, iar P (X = x) = 0 pentruorice x 6= x1. Altfel spus, variabila aleatoare discreta X este constanta.

,,⇐“ Presupunem ca variabila X este constanta, ceea ce ınseamna ca X ia o singuravaloare x1, cu probabilitatea P (X = x1) = 1. Prin urmare, H(X) = −1 · log 1 = 0.

H(Y | X) = −∑

P (X = xi, Y = yj) logP (Y = yj | X = xi)

H(Y | X) =∑

P (X = xi)H(Y | X = xi)

P (X = xi)

−∑

P (Y = yj | X = xi) logP (Y = yj | X = xi)

= −∑

P (X = xi)P (Y = yj | X = xi)︸︷︷︸

=P (X=xi,Y=yj)

logP (Y = yj | X = xi)

= −∑

P (X = xi, Y = yj) logP (Y = yj | X = xi)

c.H(X, Y ) = H(X) +H(Y | X) = H(Y ) +H(X | Y )

H(X, Y ) = −∑

p(xi, yj) log p(xi, yj)

= −∑

p(xi) · p(yj | xi) log[p(xi) · p(yj | xi)]

= −∑

p(xi) · p(yj | xi)[log p(xi) + log p(yj | xi)]

= −∑

p(xi) · p(yj | xi) log p(xi)−∑

p(xi) · p(yj | xi) log p(yj | xi)

= −∑

p(xi) log p(xi) ·∑

p(yj | xi)︸︷︷︸

−∑

p(xi)∑

p(yj | xi) log p(yj | xi)

= H(X) +∑

p(xi)H(Y | X = xi) = H(X) +H(Y | X)

Mai general (regula de ınlantuire):

H(X1, . . . , Xn) = H(X1) +H(X2 | X1) + . . .+H(Xn | X1, . . . , Xn−1)

H(X1, . . . , Xn) =E

p(x1, . . . , xn)

=− Ep(x1,...,xn)[log p(x1, . . . , xn)]

=− Ep(x1,...,xn)[log p(x1) + log p(x2 | x1) + . . .+ log p(xn | x1, . . . , xn−1)]

=− Ep(x1)[log p(x1)]−Ep(x1,x2)[log p(x2 | x1)]− . . .− Ep(x1,...,xn)[log p(xn | x1, . . . , xn−1)]

=H(X1) +H(X2 | X1) + . . .+H(Xn | X1, . . . , Xn−1)

An upper bound for the entropy of a discrete distribution

CMU, 2003 fall, T. Mitchell, A. Moore, HW1, pr. 1.1

Fie X o variabila aleatoare discreta care ia n valori si urmeaza distributiaprobabilista P . Conform definitiei, entropia lui X este

H(X) = −n∑

P (X = xi) log2 P (X = xi).

Aratati ca H(X) ≤ log2 n.

Sugestie: Puteti folosi inegalitatea lnx ≤ x − 1 care are loc pentru oricex > 0.

Answer

H(X) =1

−n∑

P (X = xi) lnP (X = xi)

Asadar,

H(X) ≤ log2 n ⇔ 1

−n∑

P (X = xi) lnP (X = xi)

≤ log2 n

⇔ −n∑

P (xi) lnP (xi) ≤ lnn

⇔n∑

P (xi) ln1

P (xi)−(

P (xi)

︸︷︷︸

lnn ≤ 0

⇔n∑

P (xi) ln1

P (xi)−

P (xi) lnn ≤ 0

⇔n∑

P (xi)

P (xi)− lnn

⇔n∑

P (xi) ln1

nP (xi)≤ 0

Aplicand inegalitatea lnx ≤ x− 1 pentru x =1

nP (xi), vom avea:

P (xi) ln1

nP (xi)≤

P (xi)

nP (xi)− 1

P (xi)

︸︷︷︸1

= 1− 1 = 0

Observatie: Aceasta margine superioara chiar este ,,atinsa“. De exemplu, ın cazul ıncare o variabila aleatoare discreta X avand n valori urmeaza distributia uniforma, sepoate verifica imediat ca H(X) = log2 n.

Relative entropy a.k.a. the Kulback-Leibler divergence,

and the [relationship to] information gain;

some basic properties

CMU, 2007 fall, C. Guestrin, HW1, pr. 1.2[adapted by Liviu Ciortuz]

The relative entropy — also known as the Kullback-Leibler (KL) diver-gence — from a distribution p to a distribution q is defined as

KL(p||q) def.= −

p(x) logq(x)

From an information theory perspective, the KL-divergence specifies thenumber of additional bits required on average to transmit values of X ifthe values are distributed with respect to p but we encode them assumingthe distribution q.

1. KL is not a distance measure, since it is not symmetric (i.e., in generalKL(p||q) 6= KL(q||p)).

Another measure, which is defined as JSD(p||q) = 1

2(KL(p||q)+KL(q||p)), and

is called the Jensen-Shannon divergence is symmetric.

2. The quantity

d(X, Y )def.= H(X, Y )− IG (X, Y ) = H(X) +H(Y )− 2IG (X, Y )

= H(X | Y ) +H(Y | X)

known as variation of information, is a distance metric, i.e., it is non-negative, symmetric, implies indiscernability, and satisfies the triangle in-equality.

a. Show that KL(p||q) ≥ 0, and KL(p||q) = 0 iff p(x) = q(x) for all x.(More generally, the smaller the KL-divergence, the more similar the two distributions.)

Indicatie:

Pentru a demonstra punctul acesta puteti folosi inegalitatea lui Jensen:

Daca ϕ : R → R este o functie convexa, atunci pentru orice t ∈ [0, 1] si orice x1, x2 ∈ R

urmeaza ϕ(tx1 + (1− t)x2) ≤ tϕ(x1) + (1− t)ϕ(x2).Daca ϕ este functie strict convexa, atunci egalitatea are loc doar daca x1 = x2.

Mai general, pentru orice ai ≥ 0, i = 1, . . . , n cu∑

i ai 6= 0 si orice xi ∈ R, i = 1, . . . , n, avem

i aixi∑

≤∑

i aiϕ(xi)∑

Daca ϕ este strict convexa, atunci egalitatea are loc doar daca x1 = . . . = xn.

Evident, rezultate similare pot fi formulate si pentru functii concave.

Answer

Vom dovedi inegalitatea KL(p||q) ≥ 0 folosind inegalitatea lui Jensen, ın expresia careia

vom ınlocui ϕ cu functia convexa − log2, pe ai cu p(xi) si pe xi cuq(xi)

p(xi).

(Pentru convenienta, ın cele ce urmeaza vor renunta la indicele variabilei x.)Vom avea:

KL (p || q) def.= −

p(x) logq(x)

Jensen

≥ − log(∑

p(x)q(x)

p(x)) = − log(

︸︷︷︸1

) = − log 1 = 0

Asadar, KL (p || q) ≥ 0, oricare ar fi distributiile (discrete) p si q.

Vom demonstra acum ca KL(p||q) = 0⇔ p = q.

⇐Egalitatea p(x) = q(x) implica

p(x)= 1, deci log

p(x)= 0 pentru orice x, de unde

rezulta imediat KL(p||q) = 0.

⇒Stim ca ın inegalitatea lui Jensen are loc egalitatea doar ın cazul ın care xi = xjpentru orice i si j.

In cazul de fata, aceasta conditie se traduce prin faptul ca raportulq(x)

p(x)este acelasi

pentru orice valoare a lui x.

Tinand cont ca∑

x p(x) = 1 si∑

x p(x)q(x)

p(x)=∑

x q(x) = 1, rezulta caq(x)

p(x)= 1 sau,

altfel spus, p(x) = q(x) pentru orice x, ceea ce ınseamna ca distributiile p si q suntidentice.

b. We can define the information gain as the KL-divergence from the observed jointdistribution of X and Y to the product of their observed marginals:

IG(X,Y )def.= KL(pX,Y || (pX pY )) = −

pX,Y (x, y) log

(pX(x)p(y)

pY (x, y)

not.= −

p(x, y) log

(p(x)p(y)

p(x, y)

Prove that this definition of information gain is equivalent to the one given in problemCMU, 2005 fall, T. Mitchell, A. Moore, HW1, pr. 2. That is, show that IG(X,Y ) =H [X ]−H [X |Y ] = H [Y ]−H [Y |X ], starting from the definition in terms of KL-divergence.

Remark:

It follows that

IG(X,Y ) =∑

p(y)∑

p(x | y) log p(x | y)p(x)

p(y)KL(pX|Y || pX)

= EY [KL(pX|Y || pX)]

Answer

By making use of the multiplication rule, namely p(x, y) = p(x | y)p(y), we will have:

KL(pXY || (pX pY ))

def. KL= −

p(x, y) log

(p(x)p(y)

p(x, y)

= −∑

p(x, y) log

(p(x)p(y)

p(x | y)p(y)

= −∑

p(x, y)[log p(x)− log p(x | y)]

= −∑

p(x, y) log p(x)−(

−∑

p(x, y) log p(x | y))

= −∑

log p(x)∑

p(x, y)

︸︷︷︸

−H [X | Y ]

= H [X ]−H [X | Y ] = IG(X,Y )

c.A direct consequence of parts a. and b. is that IG(X, Y ) ≥ 0 (and thereforeH(X) ≥ H(X|Y ) and H(Y ) ≥ H(Y |X)) for any discrete random variables Xand Y .

Prove that IG(X, Y ) = 0 iff X and Y are independent.

Answer:

This is also an immediate consequence of parts a. and b. already proven:

IG (X, Y ) = 0(b)⇔ KL (pXY ||pX pY ) = 0

(a)⇔ X and Y are independent.

Remark

Putem demonstra inegalitatea IG(X,Y ) ≥ 0 si ın maniera directa, folosind rezultatul dela punctul b. si aplicand inegalitatea lui Jensen ın forma generalizata, cu urmatoarele,,amendamente“:

− ın locul unui singur indice, se vor considera doi indici (asadar ın loc de ai si xi vomavea aij si respectiv xij);

− vom lua ϕ = − log2 iar aij ← p(xi, yj) si xij ←p(xi)p(xj)

p(xi, xj);

− ın fine, vom tine cont ca∑

j p(xi, yj) = 1.

Prin urmare,

IG(X,Y ) =∑

p(xi, yj) logp(xi, yj)

p(xi) · p(yj)=∑

p(xi, yj)

− logp(xi) · p(yj)p(xi, yj)

≥ − log

p(xi, yj)p(xi) · p(yj)p(xi, yj)

= − log(∑

p(xi) · p(yj))

= − log(∑

︸︷︷︸1

) = − log 1 = 0

In concluzie, IG(X,Y ) ≥ 0.

Remark (cont’d)

Daca X si Y sunt variabilele independente,atunci p(xi, yj) = p(xi)p(yj) pentru orice i si j.

In consecinta, toti logaritmii din partea dreapta a primei egalitati din calculul de maisus sunt 0 si rezulta IG(X,Y ) = 0.

Invers, presupunand ca IG(X,Y ) = 0, vom tine cont de faptul ca putem exprima castigulde informatie cu ajutorul divergentei KL si vom aplica un rationament similar cu celde la punctul a.

Rezulta cap(xi)p(yj)

p(xi, yj)= 1 si deci p(xi)p(yj) = p(xi, yj) pentru orice i si j.

Aceasta echivaleaza cu a spune ca variabilele X si Y sunt independente.

Proving [in a direct manner] thatthe Information Gain is always positive or 0

(an indirect proof was made at CMU, 2007 fall, Carlos Guestrin, HW1, pr. 1.2)

Liviu Ciortuz, 2017

Definitia castigului de informatie (sau: a informatiei mutuale) al unei vari-abile aleatoare X ın raport cu o alta variabila aleatoare Y este

IG (X,Y ) = H(X)−H(X | Y ) = H(Y )−H(Y | X).

La CMU, 2007 fall, Carlos Guestrin, HW1, pr. 1.2 s-a demonstrat — pentru cazul ın care X

si Y sunt discrete — ca IG(X,Y ) = KL(PX,Y ||PX PY ), unde KL desemneaza entropia relativa (sau:

divergenta Kullback-Leibler), PX si PY sunt distributiile variabilelor X si, respectiv, Y , iar PX,Y este

distributia corelata a acestor variabile. Tot la CMU, 2007 fall, Carlos Guestrin, HW1, pr. 1.2 s-a

aratat ca divergenta KL este ıntotdeauna ne-negativa. In consecinta, IG(X,Y ) ≥ 0 pentru orice X

si Y .

La acest exercitiu va cerem sa demonstrati inegalitatea IG(X,Y ) ≥ 0 ınmaniera directa, plecand de la prima definitie data mai sus, fara a [mai] apelala divergenta Kullback-Leibler.

Sugestie: Puteti folosi urmatoarea forma a inegalitatii lui Jensen:

ai log xi ≤ log

unde baza logaritmului se considera supraunitara, ai ≥ 0 pentru i = 1, . . . , n si∑n

i=1 ai = 1.

Observatie: Avantajul la aceasta problema, comparativ cu CMU, 2007 fall, Carlos Guestrin,

HW1, pr. 1.2.a, este ca aici se lucreaza cu o singura distributie (p), nu cu doua distributii (p si q).

Totusi, demonstratia de aici va fi mai laborioasa.

Answer (in Romanian)

Presupunem ca valorile variabilei X sunt x1, x2, . . . , xn, iar valorile variabilei Ysunt y1, y2, . . . , ym. Avem:

IG(X,Y )def.= H(X)−H(X |Y )

−P (xi) log2 P (xi)−m∑

P (yj)

(−P (xi|yj) log2 P (xi|yj))

−IG(X,Y ) =

P (xi) log2 P (xi)−m∑

P (yj)

P (xi|yj) log2 P (xi|yj)

prob. marg.

P (xi, yj)

log2 P (xi)−m∑

P (yj)

P (xi|yj) log2 P (xi|yj)

distrib.·,+=

P (xi, yj) log2 P (xi)−m∑

P (yj)P (xi|yj) log2 P (xi|yj)

prob. cond.

P (xi, yj) log2 P (xi)−m∑

P (xi, yj) log2 P (xi|yj)

distrib.·,+=

P (xi, yj)(log2 P (xi)− log2 P (xi|yj))

prop.=log.

P (xi, yj) log2P (xi)

P (xi|yj)

reg. de=

multipl.

P (xi|yj)P (yj) log2P (xi)

P (xi|yj)

distrib.·,+=

P (yj)

P (xi|yj)︸︷︷︸

log2P (xi)

P (xi|yj)

Intrucat pe de o parte P (xi|yj) ≥ 0 si pe de alta parte∑n

i=1 P (xi|yj) = 1 pentrufiecare valoare yj a lui Y ın parte, putem aplica inegalitatea lui Jensen pentrucea de-a doua suma din ultima expresie de mai sus — mai exact, pentrufiecare valoare a indicelui j ın parte — si obtinem:

−IG(X,Y ) ≤m∑

P (yj) log2

P (xi|yj)P (xi)

P (xi|yj)

P (yj) log2

P (xi)

︸︷︷︸1

Prin urmare, IG(X,Y ) ≥ 0.

foundations of probabilities

Documents

1 slide slide probability is conditional. theorems of...

table of poisson probabilities

13.4 probabilities of compound events.notebook

separation probabilities for products of permutations

on computing probabilities of dismissal of 10b-5...

probabilities of independent events

elementary manipulations of probabilities

measuring probabilities of vessel infrastructure rawson...

estimation of rating class transition probabilities with ......

world of probabilities

estimating historical probabilities of natural and

foundations of bayesian theoryadres.ens.fr › img › pdf...

the quality of experts' probabilities techniques3.2.2...

probabilities of dependent events. determining probabilities...

fixation probabilities of evolutionary coordination games...

6.3 assignment of probabilities

probabilities of conditionals

default probabilities of privately held firms

4. probabilities in epidemiology 2017 -...

probabilities of disjoint and overlapping events