Outline:

1 “Simplest” unsolved IT problem.

2 Information-theoretic attacks.

3 Finite dimensional problem.

4 Summary.

Pulkit Grover, Anant Sahai and Se Yong Park

The Finite-Dimensional Witsenhausen CounterexampleInternational Symposium on Information Theory (ISIT), 2009.

Slides available: www.eecs.berkeley.edu/∼pulkit/ConCom09Slides.pdfPoster available: www.eecs.berkeley.edu/∼pulkit/ISIT09Poster.pdfThis Handout: . . . /∼pulkit/ISIT09Handout.P.pdf

1 Vector Witsenhausen counterexample: “Simplest” unsolved IT problem

Encoder ++ Decoderx1u1

x0 ∼ N (0, σ2

0I) w ∼ N (0, I)

M M

Dirty paper coding

[Ho, Kastner and Wong '78]

Even in the 70’s, IT and Witsenhausen’s counterexample wereconsidered related. In the 80’s, the new framework of Dirty-Paper Coding (DPC) [Costa ’83] brought forward a model ofan idealized kind of interference. Some recent extensions stateamplification ( communicating x0) [Kim, Sutivong, Cover ’08],state masking ( hiding x0) [Merhav, Shamai, ’06] have beensolved completely.

state-dependent MAC with a common message.[Somekh-Baruch, Shamai, Verdu '08]

Distributed DPC is more interesting. If the informed encoderknows all the messages, [Somekh-Baruch, Shamai, Verdu ’08]solve the problem.

Multiaccess channelp(y|x1, x2, s)

DecoderYn

Sn

State generator

(W1, W2)

Xn1

Xn2

Informed encoder

Uninformed encoder

W1

W2

[Kotagiri, Laneman '08]

But if the informed encoder does not know the other message[Kotagiri, Laneman ’08], the problem is unsolved even if theinformed encoder has no message of its own.

Receiver

Rx

Helper

H

InterfererI Transmitter

Tx

Assisted Interference Suppression

u1x0

x0

w

Encoder ++ Decoder x1

x1u1

x0 ∼ N (0, σ2

0I) w ∼ N (0, I)

The vector Witsenhausen counterexample

1

mE[‖u1‖

2] ≤ P min1

mE[‖x1 − x1‖

2] ≤ Px1 − x1

The above point-to-point problem (with the objective minimizing the ex-pected cost, E[C]) is a natural simplification of the [Kotagiri, Laneman ’08]problem. Using a convex duality argument, this problem is equivalent tominimizing MMSE in x1 estimation for an average constraint P on u1.Here, the Helper H attempts to “clean” the “noisy” channel by makingthe net interference x1 = x0 + u1 better estimable, in turn improving theestimate of the uncoded transmissions of the transmitter Tx. We call thisproblem the Assisted Interference Suppression.

Amazingly, this is exactly the Witsenhausen counterexample!

2 Information-theoretic attacks on the counterexample

2.1 Witsenhausen’s lower bound

Witsenhausen derived the following lower bound to the total cost for the scalar problem.

Cscalarmin ≥ 1

σ0

∫ +∞

−∞φ

(

ξ

σ0

)

Vk(ξ)dξ,

where φ(t) = 1√2π

exp(− t2

2 ), Vk(ξ) := mina[k2(a − ξ)2 + h(a)], and h(a) :=

√2πa2φ(a)

∫ +∞−∞

φ(y)cosh(ay)dy.

The bound is based on the idea of giving the decoder the side-information of absolute value of x1. The boundturns out to be quite loose, and does not extend easily to the vector case.

2.2 The core tension of Gaussianity, and a lossy source coding based lower bound

!" #" $" %" &"

!#'!

!#

!!'(

!!'&

!!'$

!!'!

!!

!)'(

!)'&

!)'$*+,-./0

1234-563-5.4

75-8+49328+4:8

0;.<+/0=.24>

.2/0;.<+/0=.24>

log10(C

)

σ0 (on log scale)4 9 16 25 36

Witsenhausen's scalar

lower bound

The key observation is that the transmitter modifies x0

and communicates the modified source across the channel.Therein lies the core tension — while x1 being Gaussianmaximizes the channel capacity, thus helping in communi-cating x1, a Gaussian source is also the hardest to estimatethrough an AWGN channel. The decoder would prefer anon-Gaussian source, preferably even discrete!

We ignore this tension, and provide a lower bound based onlossy-source coding and channel coding arguments,

Cmin ≥ infP≥0

k2P +(

(√

κ(P ) −√

P )+)2

,

where κ(P ) =σ2

0σ2

w

σ2

0+2σ0

√P+P+σ2

w

, where σ2w = 1 is the ob-

servation noise variance. Despite ignoring the tension, thebound outperforms Witsenhausen’s bound in many cases.The problem of a tight lower bound on costs for the vectorWitsenhausen problem is still open.

2.3 A DPC-based achievability, and approximate optimality for infinite length

u1

x0

αx0

x0

u1

x1

αx0

√

m(P + α2σ

2

0)

An interpretation of the "slopey"

quantization strategies of [Baglietto et al]

The exact same sequence of steps yields

the DPC-based strategy!

We also provide two achievable strategies. Thefirst is based on vector quantization ideas.The more interesting dirty-paper coding basedstrategy performs better. Some “slopey”-quantization strategies are obtained in [Lee,Lau, Ho ’01] [Baglietto, Parisini, Zoppoli ’01]by a search over the entire solution space (andhence are believed to be optimal). These strate-gies can be interpreted as scalar version of ourDPC-based strategy!

!!

!"

#

"

!"!$

#$

"$

$%"

$%!

$%&

$%'

"

log10(k)

log10(!0

2)

Ratio

()*+,-,.-/0012-)34-5,612-7,/348-/8+39*:1-;,<7+3)*+,3-=>?@-A-5+31)2B-8*2)*19C

!0

2 = 0.618

Ratio of costs attained by combination (DPC+linear) strategy

and our lower bound

The ratio of the upper bound on costs attained by DPC-based strategy and our vector lower bound turns out to besmaller than 2 for all parameter values.

This result is in the spirit of recent approximation results ininformation-theory (e.g.[Etkin, Tse and Wang ’07]) whereapproximate rate-regions are found for longstanding prob-lems.

3 The finite-dimensional problem: sphere-packing arguments

3.1 The scalar case

−2−1.5

−1−0.5

0

0

0.5

1

1.50

10

20

30

40

log10

(k)log10

(σ0)

ratio

of u

pper

bou

nd (

quan

tizat

ion/

linea

r)an

d ve

ctor

low

er b

ound

Information-theorists have recently been satisfied with approxi-mate optimality that holds only in the limit of large dimensions.The control-theoretic history of this problem demands answers forfinite dimensions as well.

For the scalar case, the ratio of the cost attained by thequantization-based strategies and our vector lower bound runs offto infinity in the low k regime. As we shall see, this is because ofthe looseness in the lower bound, rather than a mediocre perfor-mance of quantization-based strategies.

1 1.5 2 2.5 3

10−6

10−4

10−2

Power P

MM

SE

Scalar lower bound

σG

= 1

σG

= 1.25σ

G = 2.1 σ

G = 3.0

σG

= 3.9

A “Platonic” sphere-packing approach that treats the finite-dimensional world as shadows of the infinite-dimensional worldallows us to tighten bounds for finite dimensions. The observa-tion noise can behave as if its variance is much larger, therebyincreasing the lower bound. For finite dimensions, there is a non-zero probability (that decays exponentially with the number ofdimensions) that this atypical behavior happens.

This style of obtaining “sphere-packing” bounds follows the workof [Blahut ’72] and the ensuing extensions to delay [Sahai ’06] andneighborhood sizes with bit-error probability [Sahai, Grover ’07].The main technical contribution is extension to an extension ofthese techniques to a source with unbounded distortion measure.

−2

−1

0

00.20.40.60.812

4

6

8

log10

(k)log10

(σ0)

ratio

of u

pper

and

low

er b

ound

s

It turns out that the ratio of costs achieved by the optimal com-bination of linear + quantization strategies and that of the newbound is bounded by 8 uniformly over all k and σ0 in the scalarcase.

3.2 More than one dimensional counterexample

x0

rc

rp

ζ =

rc

rp

rc

Since the initial state can fall outside theshell of typical realizations with non-zeroprobability, we need to tile the space withquantization points. Lattices provide anatural set of such quantization points.Lattices that are good for quantization(and hence require low power at the firststage) as well as error resilience have asmall ζ = rc

rp, where rc is the covering ra-

dius and rp is the packing radius of thelattice.

−2

−1

0

00.5

11.5

2

4

6

8

10

12

14

16

log10

(k)log

10(σ

0)

ratio

of u

pper

and

low

er b

ound

s

The 2−D problem

For the two-dimensional Witsenhausen counterexample, thehexagonal lattice (of ζ = 2√

3≈ 1.155) is better than the

square lattice (extension of scalar-quantization, ζ =√

2 ≈1.414). The hexagonal lattice attains within a ratio of 15 ofthe optimal cost.

More generally, we show that lattices can attain within afactor of 300ζ2 of the optimal costs for any finite dimen-sion. Since ζ ≤ 4 for all dimensions, lattices attain within aconstant factor uniformly over all k, σ0, and number of di-mensions. The constant can be improved by a more carefulanalysis.

4 Summary

This poster intends to bring out the following ideas:

• Information-theoretic techniques help solve the long-standing (40+ year old) Witsenhausen counterexampleapproximately. It seems to be one amongst many similar information-theory problems, and the infinite-lengthcase does not seem impossible to solve completely.

• Lattices attain within a constant factor of the optimal costs for the vector Witsenhausen counterexampleuniformly over the number of dimensions, k, and σ0.

• We observed that solving infinite length Witsenhausen counterexample approximately does not guaranteean approximate solution at finite lengths as well, and substantial work may be required in doing so. Doesthe same conclusion hold for information-theory problems as well?