3864 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 8, AUGUST 2009

Joint Source–Channel Coding for TransmittingCorrelated Sources Over Broadcast Networks

Todd P. Coleman, Member, IEEE, Emin Martinian, Member, IEEE, and Erik Ordentlich, Senior Member, IEEE

Abstract—We consider a set of � independent encoders thatmust transmit a set of correlated sources through a network ofnoisy, independent, broadcast channels to � receivers, with nointerference at the receivers. For the general problem of sendingcorrelated sources through broadcast networks, it is known thatthe source–channel separation theorem breaks down and theachievable rate region as well as the proper method of coding areunknown. For our scenario, however, we establish the optimalrate region using a form of joint source–channel coding. When theoptimal channel input distribution from transmitter � to receiver �is independent of �, our result has a max-flow/min-cut interpreta-tion. Specifically, in this case, our result implies that if it is possibleto send the sources to each receiver separately while ignoring theothers, then it is possible to send to all receivers simultaneously.

Index Terms—Joint source–channel coding, network informa-tion theory, separation theorem.

I. INTRODUCTION

W HILE many point-to-point communication problemsare relatively well understood, the more general prob-

lem of how to transmit correlated sources to multiple receiversis much harder to analyze. Two of the key features that com-plicate network information theory problems include the lackof source–channel separation and the conflicting goals thatarise when multiple destinations receive the same transmission(broadcast) or when multiple transmitters send to the samedestination (multiple access).

In order to understand how these issues affect communi-cation system design, we consider the problem of sendingcorrelated sources through a noisy broadcast network with

receivers as illustrated in Fig. 1. Specifically, we imaginethat transmitter observes a source which consists of a setof independent and identically distributed (i.i.d.) samples,

. The sources are correlated (across thetransmitters) in the sense that the joint distribution (across ) is

(1)

Manuscript received January 05, 2007; revised March 19, 2009. Current ver-sion published July 15, 2009. The work of T. P. Coleman was supported by theDARPA ITMANET program via US Army RDECOM Contract W911NF-07-1-0029. The material in this paper was presented in part at the IEEE InternationalSymposium on Information Theory (ISIT), Seattle, WA, July 2006.

T. P. Coleman is with the Coordinated Science Laboratory, University ofIllinois at Urbana-Champaign, Urbana, Il 61801 USA (e-mail: colemant@illinois.edu).

E. Martinian is with Bain Capital, Boston, MA 02199 USA (e-mail:emin@alum.mit.edu).

E. Ordentlich is with Hewlett–Packard Laboratories, Palo Alto, CA 94304USA (e-mail: eord@hpl.hp.com).

Communicated by Y. Steinberg, Associate Editor for Shannon Theory.Digital Object Identifier 10.1109/TIT.2009.2023722

Fig. 1. Transmitting correlated sources over a broadcast network: Two corre-lated sources � and � are observed by independent transmitters and bothsources must be communicated to each receiver.

The channels are independent broadcast channels in the sensethat if each transmitter chooses a block of channel inputs

then each receiver observes thechannel output sequences withoutinterference. The aggregate channel law is thus assumed to be

(2)for some set of discrete memoryless channels . Ofinterest are communication schemes, possibly involving jointsource–channel coding, that allow each receiver to decode allof the source sequences with high probability.

Ho et al. considered Slepian–Wolf [1] coding over net-works within the context of network coding [2], with implicituse of joint source–channel coding schemes. They do not,however, consider noisy channels within their framework. Ra-mamoorthy et al. considered separating Slepian–Wolf codingfrom network coding and developed a scenario where separatesource–channel coding is insufficient [3]. Other work thatillustrates max-flow/min-cut interpretations of communicationacross networks of noisy channels includes [4]. Cover, ElGamal, and Salehi [5] considered the problem of transmittingcorrelated sources over multiple-access networks. In theirmodel, there is only a single receiver (i.e., ) and themultiple-access interference does not factor as in (2). As aresult, [5] shows that the source–channel separation theoremfor point-to-point communication [6, Sec. 14.10] breaks downand a joint source–channel coding strategy is required.

In contrast, as in the case of [7], a key feature of our networkmodel is that each receiver observes the output of the channelfrom each transmitter independently and so there is no mul-tiple-access interference. Consequently, effects such as coherentor incoherent interference, beamforming, or transmitter coop-eration are not possible. Note, however, that there is a type of“broadcast interference” in the sense that transmitter cannotchoose to send independent signals to each receiver. Instead,transmitter must choose a channel input that simultaneouslyconveys the desired message to all receivers. Similarly, there is

0018-9448/$25.00 © 2009 IEEE

COLEMAN et al.: JOINT SOURCE–CHANNEL CODING FOR TRANSMITTING CORRELATED SOURCES OVER BROADCAST NETWORKS 3865

also a kind of “distribution interference” in the sense that trans-mitter cannot choose a different input distribution for eachchannel. For example, if is the result of passing througha binary-symmetric channel (BSC) while is the result ofpassing through a -channel, then it is not possible to si-multaneously transmit at the point-to-point capacity of the indi-vidual channels.

Another closely related paper is that of Tuncel [8], whichconsiders the problem of transmitting a single source over abroadcast channel to receivers that also observe side informa-tion correlated with the source. A single-letter characterizationof the source–channel pairs for which reliable communicationis possible is derived. In contrast, our setting involves multiplesources and requires that each source be correctly decodedby each receiver. We assume no receiver side information, al-though this could be easily incorporated, as can requiring eachreceiver to correctly decode only a prespecified, receiver-de-pendent subset of the sources. Either generalization would theninclude the setting of [8]. The achievability proof of [8] relieson a joint source–channel coding scheme using a kind of virtualbinning in which a bin consists of all source sequences whosecorresponding channel codewords are jointly typical with thereceived signal. Our achievability proof is based on a similarapproach.

II. DEFINITIONS

• We denote a length- sequence as.

• We use the notation to denote the set andto denote the set .

• Source , the channel input for transmitter , and thechannel output from transmitter to transmitter shall beassumed to be over the respective finite alphabets , ,and . Sequences over these alphabets shall respectivelybe denoted as , , and .

• For any , we define and as

We denote by the jointly typical set associatedwith the discrete probability distribution on and

s.t.

• For any we define , the set of all sequencessuch that differs from for each and that,

when combined with , are jointly typical, as

(3)

We note the following properties [6] for a collection of i.i.d.random variables with probability distribution

.• We denote a pair of random variables sequences

as jointly typical with respect to the distribution if.

Fig. 2. Strict suboptimality of source–channel separation at both encoders anddecoders.

• For any and for large enough

(4)

• If are i.i.d. according to and andagree with the marginals of , then

(5)

The following is also a well-known consequence of jointtypicality.

• If , then

(6)

III. STATEMENT OF RESULTS

Let denote the point-to-point capacity of the channelconnecting source to receiver

(7)

Specifically, is the maximum rate of information that canbe sent from transmitter to receiver , if all other receiversare ignored. Thus, if we only cared about receiver it is trivialto determine whether all the sources could be successfullycommunicated to the receiver with no bandwidth expansion/compression.1

Proposition 1: If we only consider receiver , the sources canbe reliably communicated to receiver if for some

(8)

where is the Slepian–Wolf [1] rate re-gion for the sources . Conversely, if the sourcescan be reliably communicated to receiver then

(9)

While (8) is sufficient if we only want to communicate to re-ceiver , it is not sufficient if we wish to communicate to all re-ceivers simultaneously, in a source–channel separated fashion.Specifically, even if (8) is satisfied for each , it may be im-possible to successfully communicate the sources to all the re-ceivers with separate source and channel decoding.

For example, as illustrated in Fig. 2, suppose each channel isa BSC where , , , , andsuppose we would like to communicate the correlated sources

1With no bandwidth expansion/compression, if � source samples are ob-served for each � , then exactly � channel samples are used. We consider themore general case of differing numbers of samples in Section V.

3866 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 8, AUGUST 2009

and where , .Then from the cut-set bounds [6], if we were to perform com-plete source–channel separation at the encoders and decoders,then message 1 can be decoded by receiver 2 only if

. Similarly, message 2 can be decoded by receiver 1 only if. This implies that . So

performing Slepian–Wolf encoding and decoding in a modularfashion with channel coding does not suffice even though (8) issatisfied for all .

Any scheme based on Slepian–Wolf source coding followedby channel encoding for the above example would require trans-mitting across some of the BSCs at rates greater than their ca-pacities. Nevertheless, the following theorem, which is our mainresult, implies that the use of a certain joint source–channelcoding procedure does permit the reliable transmission of thesources to each of the receivers.

Theorem 1: The sources can be reliably commu-nicated to all receivers simultaneously if there exist random vari-ables satisfying

(10)

for all and . Conversely, if the sourcescan be reliably communicated to all receivers simultaneouslythen there exist random variables satisfying

(11)

for all and .

Consider the case where for each , there is an optimal inputdistribution that simultaneously achieves the point-to-point capacity from transmitter to receiver

such that (12)

When this condition is satisfied (e.g., when each channel fromto is a BSC), Theorem 1 has a max-flow/min-cut interpre-

tation. Specifically, if we consider a cut-set separating receiverfrom all the transmitters we can easily determine whether re-

ceiver can decode all the sources using capacity arguments andthe Slepian–Wolf theorem via Proposition 1. Evidently, when(12) is true, Theorem 1 says that we can determine whether agiven communication problem is feasible (i.e., whether all thereceivers can successfully decode all the sources) simply byconsidering the minimum cut. It seems that in this scenario,information acts like a fluid in the sense that analyzing flowsis sufficient and worrying about “broadcast interference” is notrequired.

IV. PROOF OF THEOREM 1

A. Achievability

1) Encoder: Select at random, independently for eachtransmitter , a source-sequence-to-channel-codeword encoder

by letting the symbols of for all

be i.i.d. (within and across codewords) according to.2

2) Decoder: At the th decoder, decode to a source tuplein also satisfyingfor all .

We shall use the upper case symbols , , andto denote the random values of the source sequence, channelcodewords, and channel outputs, respectively, with a joint dis-tribution induced by the source distribution, the above randomencoder selection, and the above broadcast network channelmodel.

3) Error Probability Analysis: To be concise, we shall letdenote

The error probability can be bounded by

(13)

where•

The probability of this event, averaged over the randomencoding, tends to by (4).

•

We now bound which is averaged overthe random encoding.

For each , and any

(14)

which follows from the independence of andand (5).

Letting

it follows that

(15)

2The rate of each implicit channel code portion of the encoding is thus��� �� �. These “codes,” however, will not be analyzed in isolation from thesource and their rates will not be of direct relevance to the rest of the proof.

COLEMAN et al.: JOINT SOURCE–CHANNEL CODING FOR TRANSMITTING CORRELATED SOURCES OVER BROADCAST NETWORKS 3867

(16)

(17)

where (15) follows from the union bound, (3) and (6); (16) fol-lows from (14) and the independence of the encoder selectionacross transmitters; and (17) follows by (10).

B. Converse

Let us denote the error probability at receiver as .For each , define

Without loss of generality, assume that .Fano’s inequality tells us that if as , then

where (18)

Thus

(19)

(20)

(21)

(22)

where (19) is due to (18), (20) follows because the channels areindependent, (21) follows because the channels are memorylessand because conditioning reduces entropy, and (22) follows bydefining

and noting the concavity of mutual information.

V. BANDWIDTH EXPANSION AND COMPRESSION

Theorem 1 assumes that source symbols are encoded intochannel symbols and characterizes the source–channel pairs

for which this can be done reliably. Additional flexibility can beintroduced by allowing encoder to map source symbols to

channel symbols, where the bandwidth factor can beinterpreted as a suitably normalized bandwidth of the channel atsender . By slightly modifying the proof of Theorem 1, we canobtain the following generalization that characterizes the set ofreliably achievable bandwidth factors for a givencollection of sources and channels.

Theorem 2: The sources can be reliably commu-nicated to all receivers simultaneously, using bandwidth fac-tors , if there exist random variablessatisfying

(23)

for all and . Conversely, if the sourcescan be reliably communicated to all receivers simultaneously,using bandwidth factors , then there exist randomvariables satisfying

(24)

for all and .

VI. CONCLUDING REMARKS

To summarize, we illustrated that for broadcast networks,performing source–channel separation for encoding and de-coding is in general suboptimal. Next, we characterized theoptimal performance for this setting via Theorem 1, the proofof which shows the existence of a joint source–channel codingscheme attaining optimal performance. In the (random) con-struction of this scheme (Section IV-A.1), channel codewordsare assigned to source sequences in an i.i.d. fashion. The proofcan thus be interpreted as demonstrating the existence of a setof channel codewords such that almost all assignments of thesecodewords to source sequences yields a good source–channelcode, when decoded using the joint source–channel decoderof Section IV-A.2. This is in contrast to multiple-access net-works exhibiting multiple-access interference, where a carefulassignment of channel codewords to source sequences appearsto be needed to optimally match the correlation in the sourcesto the structure of the interference. Thus, in a sense, a kindof source–channel separation exists on the encoding side ofour problem. An in-depth discussion of various degrees ofsource–channel separation can be found in the related work [8]mentioned above.

Our result may have practical relevance as a robust architec-ture—for example, in sensor networks with fusion centers, pro-cessing power is usually not nearly as constrained of a resourcefor the fusion center as is the case for the individual sensors.Consequently, making the signaling methods for the individualsensor nodes to require the least possible cooperation and com-putation is generally desirable. Note that these properties fit well

3868 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 8, AUGUST 2009

with the achievability methods we proposed here, and they stillattain asymptotically optimal performance.

REFERENCES

[1] D. Slepian and J. K. Wolf, “Noiseless coding of correlated informationsources,” IEEE Trans. Inf. Theory, vol. IT-19, no. 4, pp. 471–480, Jul.1973.

[2] T. Ho, M. Médard, M. Effros, and R. Koetter, “Network coding forcorrelated sources,” in Proc. Conf. Information Sciences and Systems(CISS), Princeton, NJ, Mar. 2004.

[3] A. Ramamoorthy, K. Jain, P. A. Chou, and M. Effros, “Separating dis-tributed source coding from network coding,” IEEE Trans. Inf. Theory,vol. 52, no. 6, pp. 2785–2795, Jun. 2006.

[4] A. F. Dana, R. Gowaikar, R. Palanki, B. Hassibi, and M. Effros, “Ca-pacity of wireless erasure networks,” IEEE Trans. Inf. Theory, vol. 52,no. 3, pp. 789–804, Mar. 2006.

[5] T. M. Cover, A. El Gamal, and M. Salehi, “Multiple access channelswith arbitrarily correlated sources,” IEEE Trans. Inf. Theory, vol.IT-26, no. 6, pp. 648–657, Nov. 1980.

[6] T. M. Cover and J. Thomas, Elements of Information Theory. NewYork: Wiley, 1991.

[7] J. N. Laneman, E. Martinian, G. W. Wornell, and J. G. Apostolopoulos,“Source-channel diversity for parallel channels,” IEEE Trans. Inf.Theory, vol. 51, no. 10, pp. 3518–3539, Oct. 2005.

[8] E. Tuncel, “Slepian–Wolf coding over broadcast networks,” IEEETrans. Inf. Theory, vol. 52, no. 4, pp. 1469–1482, Apr. 2006.

Todd P. Coleman (S’01–M’05) received the B.S. degrees in electrical engi-neering (summa cum laude), as well as computer engineering (summa cumlaude) from the University of Michigan, Ann Arbor, in 2000, along withthe M.S. and Ph.D. degrees in electrical engineering from the MassachusettsInstitute of Technology (MIT), Cambridge, in 2002, and 2005.

During the 2005–2006 academic year, he was a Postdoctoral Scholar at MITand Massachusetts General Hospital in computational neuroscience. Since thefall of 2006, he has been on the faculty in the Electrical and Computer Depart-ment and Neuroscience Program at University of Illinois at Urbana-Champaign(UIUC), Urbana, IL. His research interests include information theory, opera-tions research, and computational neuroscience.

Dr. Coleman, a National Science Foundation Graduate Research Fellowshiprecipient, was awarded the University of Michigan College of Engineering’sHugh Rumler Senior Class Prize in 1999 and was awarded the MIT Electricalengineering and Computer Science Department’s Morris J. Levin Award forBest Masterworks Oral Thesis Presentation in 2002.

Emin Martinian (S’00–M’04) received the B.S. degree from the University ofCalifornia, Berkeley in 1997, and the S.M. and Ph.D. degrees from the Mass-achusetts Institute of Technology (MIT), Cambridge, in 2000 and 2004 (all inelectrical engineering and computer science).

He worked at various technology startups (OPC Technologies, PinPoint) andresearch labs (Bell Labs, Mitsubishi Electric Research Labs) before joining BainCapital, Boston, MA, in 2006. He currently works on research and strategy de-velopment at Bain’s global macro hedge fund.

Dr. Martinian has been awarded a National Science Foundation Fellowship,the Capocelli Award for Best Paper, second place for best Computer SciencePh.D., and over 10 patents.

Erik Ordentlich (S’92–M’96–SM’06) received the S.B. and S.M. degreefrom the Massachusetts Institute of Technology, Cambridge, in 1990 and thePh.D. degree from Stanford University, Stanford, CA, in 1996, all in electricalengineering.

He has been with Hewlett–Packard Labs, Palo Alto, CA, since 1996, withthe exception of a period in 1999–2002 when he was with iCompression, Inc.,Santa Clara, CA. His work has addressed multiple topics in information theory,signal processing, and financial math. He is also a coinventor on numerous U.S.Patents and contributed technology to the ISO/IEC JPEG 2000 image compres-sion standard.

Dr. Ordentlich was a corecipient of the 2006 IEEE Joint CommunicationsSociety and Information Theory Society Paper Award and was a 2007 ShannonMemorial Endowed Co-Lecturer at the Center for Magnetic Recording Re-search, University of California, San Diego. He is also a member of Tau Beta Piand Phi Beta Kappa. He has been serving as Associate Editor for Source Codingfor the IEEE TRANSACTIONS ON INFORMATION THEORY since September 2007.