a vc-dimension-based outer bound on the zero-error ...ordent/slides/bac_isit15.pdfa...

A VC-D IMENSION-BASED OUTER BOUND ON

THE ZERO-ERRORCAPACITY OF THE BINARY

ADDER CHANNEL

Or OrdentlichJoint work with Ofer Shayevitz

ISIT 2015, Hong Kong

June 19, 2015

THE BINARY ADDER CHANNEL

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}

CodebooksC1, C2 ⊆ {0, 1}n with |C1| = 2nR1 and|C2| = 2nR2

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


Error ⇔ c1 + c2 = c′1 + c

′2 for c1, c′1 ∈ C1, c2, c

′2 ∈ C2

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


Error ⇔ c1 + c2 = c′1 + c

′2 for c1, c′1 ∈ C1, c2, c

′2 ∈ C2

Shannon Capacity Region (vanishing error)

Simple MAC channel... the region is

R1 ≤ 1

R2 ≤ 1

R1 +R2 ≤ 1.5

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


Error ⇔ c1 + c2 = c′1 + c

′2 for c1, c′1 ∈ C1, c2, c

′2 ∈ C2

Zero-Error Capacity Region

No errors allowed (no collisions)

All sums are different

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


Error ⇔ c1 + c2 = c′1 + c

′2 for c1, c′1 ∈ C1, c2, c

′2 ∈ C2




Capacity region unknown

Large gap between best inner and outer bounds

1 / 16


M1 ∈[

2nR1]

:

M2 ∈[

2nR2]

:

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


Error ⇔ c1 + c2 = c′1 + c

′2 for c1, c′1 ∈ C1, c2, c

′2 ∈ C2




Capacity region unknown

Large gap between best inner and outer bounds

In this talk: Outer bound improved

1 / 16

KNOWN BOUNDS

R2

0 0.2 0.4 0.6 0.8 1

R1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Best inner bound

[Lindström ’69], [Kasami & Lin ’76], [Weldon ’78], [Peterson & Costello ’79], [Khachatrian

’82], [van Tilborg ’83], [Kasami et al ’83], [van der Braak & van Tilborg ’85], [Guo &

Watanabe ’91, ’92], [Blake ’94], [Ahlswede & Balakirsky ’99], [Mattas & Ostergard ’05]

2 / 16

KNOWN BOUNDS

R2

0 0.2 0.4 0.6 0.8 1

R1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Best inner boundShannon capacity region

2 / 16

KNOWN BOUNDS

R2

0 0.2 0.4 0.6 0.8 1

R1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Best inner boundShannon capacity regionUrbanke and Li

[Urbanke & Li 1998]

ForR1 = 1: 0.25 ≤ R2 < 0.49216 [Kasami et al ’83], [Urbanke & Li ’98]

2 / 16

KNOWN BOUNDS

R2

0 0.2 0.4 0.6 0.8 1

R1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Best inner boundShannon capacity regionUrbanke and Li UBNew UB

ForR1 = 1: 0.25 ≤ R2 <✭✭✭✭0.49216 0.4798 (this talk)

2 / 16

KNOWN BOUNDS

R2

0.48 0.485 0.49 0.495 0.5 0.505 0.51 0.515 0.52

R1

0.98

0.985

0.99

0.995

1

1.005


ForR1 = 1: 0.25 ≤ R2 <✭✭✭✭0.49216 0.4798 (this talk)

2 / 16

BACKGROUND & M OTIVATION

Projection and Shattering

Let c ∈ {0, 1}n

Let S ⊆ [n]def= {1, . . . , n} be some subset of coordinates

c(S) ∈ {0, 1}|S| is theprojectionof c ontoS

3 / 16



Let c ∈ {0, 1}n



c = 011101, S = {2, 3, 5, 6}

3 / 16



Let c ∈ {0, 1}n



c = 011101, S = {2, 3, 5, 6} ⇒ c(S) = 1101

3 / 16



Let c ∈ {0, 1}n



The projection of a codebookC ⊆ {0, 1}n ontoS is themultiset

C(S)def= {c(S) ∈ {0, 1}|S| : c ∈ C} with multiplicities

3 / 16



Let c ∈ {0, 1}n





S is shattered byC if C(S) contains all2|S| possible vectors

3 / 16



Let c ∈ {0, 1}n






C = {101100, 011101, 100110, 000111}

3 / 16



Let c ∈ {0, 1}n






C = {101100, 011101, 100110, 000111} ⇒ S = {5, 6} shattered

3 / 16



Let c ∈ {0, 1}n






C = {101100, 011101, 100110, 000111}

3 / 16



Let c ∈ {0, 1}n






C = {101100, 011101, 100110, 000111} ⇒ S = {1, 2} not shattered

3 / 16


Theorem[Weldon 78]

Let (C1, C2) be a pair of zero-error codebooks for the BAC with cardi-nalities(2nR1 , 2nR2). If C1 shatters a set of coordinatesS with cardi-nality |S| = nα, then

R2 ≤ (1− α) log 3

4 / 16


Proof:

Choose anyc2 ∈ C2

By shatterdness, there existsc1 ∈ C1 such thatc1(S) = c2(S), i.e.,

(c1 + c2)(S) = (1 . . . , 1)

|C2| = 2nR2 =⇒ 2nR2 suchS-complement pairsexist (at least)

4 / 16


Proof:

Choose anyc2 ∈ C2


(c1 + c2)(S) = (1 . . . , 1)


Their sums should be all distinct on̄Sdef= [n] \ S, hence

2nR2 ≤ 3|S̄|

4 / 16


Proof:

Choose anyc2 ∈ C2


(c1 + c2)(S) = (1 . . . , 1)



2nR2 ≤ 3n(1−α)

4 / 16


Proof:

Choose anyc2 ∈ C2


(c1 + c2)(S) = (1 . . . , 1)



2nR2 ≤ 3n(1−α) ⇒ R2 ≤ (1− α) log 3

4 / 16


Proof:

Choose anyc2 ∈ C2


(c1 + c2)(S) = (1 . . . , 1)



2nR2 ≤ 3n(1−α) ⇒ R2 ≤ (1− α) log 3

Special case: IfC1 is systematic(e.g., linear) it shatters a setS ofcardinalitynR1. Thus in this caseR2 < (1−R1) log 3

4 / 16


Proof:

Choose anyc2 ∈ C2


(c1 + c2)(S) = (1 . . . , 1)



2nR2 ≤ 3n(1−α) ⇒ R2 ≤ (1− α) log 3

Special case: IfC1 is systematic(e.g., linear) it shatters a setS ofcardinalitynR1. Thus in this caseR2 < (1−R1) log 3

Weldon stopped here, but what can be said in general?

4 / 16


VC Dimension[Vapnik-Chervonenkis 1971]

TheVC dimensionof a codebookC ⊆ {0, 1}n is the cardinality of thelargest set shattered byC

5 / 16




Lemma[Sauer-Perles-Shelah 1972]

If the VC dimension ofC is d, then

|C| ≤d

∑

k=0

(

n

k

)

5 / 16






|C| ≤d

∑

k=0

(

n

k

)

Remark: The Lemma is tight for a Hamming Ball of radiusd

5 / 16






|C| ≤d

∑

k=0

(

n

k

)

≈ 2nh(dn)


h(p) = −p log p− (1− p) log (1− p)

5 / 16






|C| ≤d

∑

k=0

(

n

k

)

≈ 2nh(dn)


h(p) = −p log p− (1− p) log (1− p)

Corollary

If |C| = 2n(R+ε) then there is a shattered setS with |S| ≥ nh−1(R).

5 / 16


Plugging this into Weldon’s theorem gives

6 / 16



Corollary

If (R1, R2) is achievable then

R2 ≤ (1− h−1(R1)) log 3

6 / 16



Corollary


R2 ≤ (1− h−1(R1)) log 3

Unfortunately, for anyR1 ∈ [0, 1]

R1 + (1− h−1(R1)) log 3 > 1.5

6 / 16



Corollary


R2 ≤ (1− h−1(R1)) log 3

Unfortunately, for anyR1 ∈ [0, 1]

R1 + (1− h−1(R1)) log 3 > 1.5

Weaknesses:We assumed only oneS-complement for eachc2 ∈ C2Weak lower bound on# of S-complement pairs (2nR2)We disregarded the sumset structure outsideSWeak upper bound on# of S-complement pairs (3|S̄|)

6 / 16

HIGHER ORDER VC DIMENSION

k-shattering

S ⊆ [n] isk-shattered byC ⊆ {0, 1}n, if C(S) contains all2|S| possiblevectors, each with multiplicity at leastk

7 / 16


k-shattering


C = {101100, 011101, 100110, 000111}

7 / 16


k-shattering


C = {101100, 011101, 100110, 000111} , S = {6} 2-shattered

7 / 16


k-shattering


C = {101100, 011101, 100110, 000111}

7 / 16


k-shattering


C = {101100, 011101, 100110, 000111} , S = {2} 1-shattered

7 / 16


k-shattering


C = {101100, 011101, 100110, 000111} , S = {2} 1-shattered

kth-order VC dimension

Thekth-orderVC dimensionof a codebookC ⊆ {0, 1}n is the cardi-nality of the largest subsetk-shattered byC

7 / 16


Lemma (“soft” Sauer-Perles-Shelah lemma)

If the kth-order VC dimension ofC ⊆ {0, 1}n is d− 1, then

|C| ≤t∗∑

t=1

(

n

t

)

+

(

n

t∗

) n∑

t=t∗+1

(

t∗

d

)

(

td

)

wheret∗ is the smallest integert satisfying(

n−dt−d

)

≥ k if such an inte-ger exists, andt∗ = n otherwise.

8 / 16




|C| ≤t∗∑

t=1

(

n

t

)

+

(

n

t∗

) n∑

t=t∗+1

(

t∗

d

)

(

td

)


n−dt−d

)


O(n/d)-tight for a Hamming Ball of radiust∗

8 / 16




|C| ≤t∗∑

t=1

(

n

t

)

+

(

n

t∗

) n∑

t=t∗+1

(

t∗

d

)

(

td

)


n−dt−d

)


O(n/d)-tight for a Hamming Ball of radiust∗

Corollary

If |C| = 2n(R+ε) then for any0 ≤ α ≤ h−1(R) there existsS with|S| ≥ nα that is2nβ-shattered byC, where

β = (1− α) · h

(

h−1(R)− α

1− α

)

8 / 16

NUMBER OFS-COMPLEMENT PAIRS

Assume(C1, C2) are zero-error

9 / 16



By the soft lemma, there exists a subsetS with cardinality|S| = nα that is2nβ-shattered byC1

9 / 16



By the soft lemma, there exists a subsetS with cardinality|S| = nα that is2nβ-shattered byC12n(R2+β)≤ # of S-complement pairs

9 / 16



By the soft lemma, there exists a subsetS with cardinality|S| = nα that is2nβ-shattered byC12n(R2+β)≤ # of S-complement pairs≤ 3n(1−α)

9 / 16



By the soft lemma, there exists a subsetS with cardinality|S| = nα that is2nβ-shattered byC12n(R2+β)≤ # of S-complement pairs≤ 3n(1−α)

R2 < (1− α) log 3− β

9 / 16



By the soft lemma, there exists a subsetS with cardinality|S| = nα that is2nβ-shattered byC12n(R2+β)≤ # of S-complement pairs≤ ?

9 / 16




Partition (C1, C2) to disjoint pairs{C1,i, C2,i}2nα

i=1 of subcodes

9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C2:1000110011110110010101111101010011010111010000000

...9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C1,a

C2:1000110011110110010101111101010011010111010000000

...

C2,a

9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C1,a

C2:1000110011110110010101111101010011010111010000000

...

C2,a

∀c1,a ∈ C1,a, c2,a ∈ C2,a:

00*****+11*****11*****

In addition: |C1,a| ≥ 2nβ

9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C1,b

C2:1000110011110110010101111101010011010111010000000

...

C2,b

9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C1,b

C2:1000110011110110010101111101010011010111010000000

...

C2,b

∀c1,b ∈ C1,b, c2,b ∈ C2,b:

01*****+10*****11*****

In addition: |C1,b| ≥ 2nβ

9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C1,c

C2:1000110011110110010101111101010011010111010000000

...

C2,c

9 / 16





i=1 of subcodes

Example:S = {1, 2}

C1:0010110111100111010110110111000010110100100111000

...

C1,c

C2:1000110011110110010101111101010011010111010000000

...

C2,b

∀c1,c ∈ C1,c, c2,c ∈ C2,c:

11*****+00*****11*****

In addition: |C1,c| ≥ 2nβ

9 / 16


We get{C1,i, C2,i}2nα

i=1 with |C1,i| = 2nβ and|C2,i| ≈ 2n(R2−α) foreveryi

10 / 16



i=1 with |C1,i| = 2nβ and|C2,i| ≈ 2n(R2−α) foreveryiInduces a zero-error scheme for BAC with common message

M1 ∈ [2nr1 ]

M2 ∈ [2nr2 ]

M0 ∈ [2nr0 ]

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}

SendC1,M0(M1) andC2,M0(M2)

10 / 16




M1 ∈ [2nr1 ]

M2 ∈ [2nr2 ]

M0 ∈ [2nr0 ]

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


r0 = α r1 = β r2 = R2 − α

10 / 16




M1 ∈ [2nr1 ]

M2 ∈ [2nr2 ]

M0 ∈ [2nr0 ]

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


r0 = α r1 = β r2 = R2 − α

For eachc1,i ∈ C1,i, c2,i ∈ C2,i we have

(c1,i + c2,i)(S) = (1, . . . , 1)

Thenα coordinates inS can be discarded!10 / 16




M1 ∈ [2nr1 ]

M2 ∈ [2nr2 ]

M0 ∈ [2nr0 ]

X1 ∈ {0, 1}

X2 ∈ {0, 1}

+ X1 +X2 ∈ {0, 1, 2}


r0 =α

1− αr1 =

β

1− αr2 =

R2 − α

1− α

For eachc1,i ∈ C1,i, c2,i ∈ C2,i we have

(c1,i + c2,i)(S) = (1, . . . , 1)

Thenα coordinates inS can be discarded!10 / 16

THE BAC WITH A COMMON MESSAGE

A reduction lemma

If (R1, R2) are in the BAC zero-error capacity region, then for any0 < α < h−1(R1)

r0 =α

1− α, r1 =

β(α,R1)

1− α, r2 =

R2 − α

1− α

is in the zero-error capacity region of the BAC with common message.

11 / 16


A reduction lemma


r0 =α

1− α, r1 =

β(α,R1)

1− α, rΣ =

R2 + β(α,R1)

1− α


11 / 16


A reduction lemma


r0 =α

1− α, r1 =

β(α,R1)

1− α, rΣ =

R2 + β(α,R1)

1− α


R2 < (1− α)rΣ − β(α,R1)

11 / 16


A reduction lemma


r0 =α

1− α, r1 =

β(α,R1)

1− α, rΣ =

R2 + β(α,R1)

1− α


R2 < (1− α)rΣ − β(α,R1)

New goal: Upper boundrΣ under the above constraints onr0, r1

11 / 16


A reduction lemma


r0 =α

1− α, r1 =

β(α,R1)

1− α, rΣ =

R2 + β(α,R1)

1− α


R2 < (1− α)rΣ − β(α,R1)

New goal: Upper boundrΣ under the above constraints onr0, r1Trivial bound:rΣ ≤ log 3

11 / 16


A reduction lemma


r0 =α

1− α, r1 =

β(α,R1)

1− α, rΣ =

R2 + β(α,R1)

1− α


R2 < (1− α) log 3− β(α,R1)


11 / 16


A reduction lemma


r0 =α

1− α, r1 =

β(α,R1)

1− α, rΣ =

R2 + β(α,R1)

1− α

is in the✭✭✭✭✭zero-error capacity region of the BAC with common message.

R2 < (1− α)rΣ − β(α,R1)


11 / 16




r1 ≤ H(X1|U)

✭✭✭✭✭✭✭

r2 ≤ H(X2|U)

r1 + r2 ≤ H(X1 +X2|U)

rΣ = r0 + r1 + r2 ≤ H(X1 +X2)


12 / 16




r1 ≤ H(X1|U)

✭✭✭✭✭✭✭

r2 ≤ H(X2|U)

r1 + r2 ≤ H(X1 +X2|U)

rΣ = r0 + r1 + r2 ≤ H(X1 +X2)


Still difficult - 7 parameters to optimize...

Need to upper boundrΣ(r0, r1) analytically!

12 / 16


Lemma (Sum-Rate Bound)

If (r0, r1, r2) is achievable then there is someη ∈ [0, 12 ] s.t.

rΣ ≤ maxh−1(r1)≤η≤ 1

2

min(

L(η), J(h−1(r1), η) + r0)

where

L(η)def= h(η) + 1− η

J(p, η)def=

2h(

12

(

1−√1− 2η

))

− η η ≥ p ⋆ p

2h

(

12

(

1− 1−η−p⋆p√1−2(p⋆p)

))

− 12

(

1− (1−η−p⋆p)2

1−2(p⋆p)

)

η < p ⋆ p

13 / 16


r0

0 0.1 0.2 0.3 0.4 0.5

r Σ

1.5

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

r1= 0.8

r1= 0.95

r1= 0.99

14 / 16

TYING THE LOOSEENDS

Theorem (Outer bound for the Zero-error Capacity Region)

Let

rΣ(r0, r1) , maxh−1(r1)≤η≤ 1

2

min{L(η), J(h−1(r1), η) + r0}

Then any zero-error achievable rate pair(R1, R2) satisfies

R2 < min0≤α≤h−1(R1)

(1− α)

(

rΣ

(

α

1− α, Γ(R1, α)

)

− Γ(R1, α)

)

where

Γ(R1, α) , h

(

h−1(R1)− α

1− α

)

15 / 16

TYING THE LOOSEENDS

R2

0.48 0.485 0.49 0.495 0.5 0.505 0.51 0.515 0.52

R1

0.98

0.985

0.99

0.995

1

1.005


15 / 16

SUMMARY AND DISCUSSION

New outer bound on BAC zero-error capacity region

We introduced the notion ofkth order VC-dimension and provedan analog of Sauer’s Lemma

Our bounding technique combined this combinatorial notionwithnetwork information theoretic arguments

Weaknesses of our boundThe lower bound on# of S-complement pairs is valid for any pair(C1, C2) (not just zero-error pairs)We lower bounded the number ofS-complement pairs foranyk-shattered set inC1, but there aremany such sets.

Our technique may be applicable to other zero-error problems

16 / 16

a vc-dimension-based outer bound on the zero-error ...ordent/slides/bac_isit15.pdfa...

Documents