Coding Strategies for

Sensor Networks:

Clustering, Compression and

Network Coding

Gerhard Maierbacher

Departamento de Ciência de Computadores

Faculdade de Ciências da Universidade do Porto

July 2011

Coding Strategies for

Sensor Networks:

Clustering, Compression and

Network Coding

Gerhard Maierbacher

Dissertação submetida à Faculdade de Ciências da Universidade do Porto para

obtenção do grau de Doutor em Ciência de Computadores

Orientador: Professor Doutor João Barros, Universidade do Porto

Departamento de Ciência de Computadores

Faculdade de Ciências da Universidade do Porto

July 2011

Abstra tWireless sensor networks, onsisting of a large number of sensor nodes that are apableof sensing the physi al world and ommuni ating in ooperative fashion, open up awhole new range of possible appli ations and potential opportunities to improve oureveryday life. Seeking pra ti al oding solutions for su h systems, we onsider a datagathering s enario in whi h orrelated data from several sour es has to be sent to oneor more sinks. The goal is to en ode, transmit, and re over the original sour e datasubje t to some delity riterion of interest. This thesis aims at providing essentialtheory and design on epts that are pra ti able even for a large number of sensors.In order to a hieve this goal we adopt the following three strategies: (a) sour e-optimized lustering for distributed ompression; (b) sour e-optimized ode designbased on Diophantine analysis; and ( ) s alable joint sour e-network oding. Ourwork shows how these strategies an be put into pra ti e by providing pra ti al design on epts and de oding algorithms. To underline the ee tiveness and versatility ofour approa hes, we provide analyti al results and numeri al examples.

3

ResumoRedes de sensores, ompostas por um grande número de nós apazes de medir e de omuni ar de forma ooperativa, ofere em novas apli ações e novas oportunidades demelhorar a nossa vida quotidiana. Em bus a de soluções práti as de odi ação, apazes de fun ionar em redes deste tipo, analisamos sistemas em que diferentesfontes enviam dados orrela ionados para vários destinos. O obje tivo deste sistemaé odi ar, transmitir e re uperar os dados originais, quando estes são submetidos a ritérios rigorosos de delidade. Desta forma, o obje tivo desta tese é forne er soluçõeses aláveis para grandes números de nós da rede. As nossas estratégias onsistem em:(a) organização dos nós em lusters optimizados tendo em onta as ara terísti asdas fontes; (b) desenho de ódigos baseados em equações diofantinas; ( ) odi açãoem rede e ompressão dos dados onjunta. Assim, a prin ipal ontribuição desta teseé forne er uma metodologia de odi ação, bem omo algoritmos de des odi açãoque fun ionem em enários práti os. Finalmente, in luímos resultados analíti os eresultados obtidos por simulação, que realçam a e á ia e versatilidade das soluçõespropostas.

5

A knowledgmentsDoing my PhD here in Porto, Portugal was a magni ent and hallenging, alwaysrewarding, experien e to me. It ertainly shaped me as a person and helped me tosee the world with dierent eyes; not only on the professional level. During all thoseyears many people were instrumental, dire tly or indire tly involved in the pursuit ofmy resear h problemsnot an easy quest from time to timeeventually leading tothese lines.I would like to start with my supervisor, Prof. João Barros, sin e he is the onewho made it possible for me to ome to Porto and work on sensor network relatedproblemsrst working on a resear h proje t and later as a PhD student. I rstmet João as a le turer at the University of Te hnology Muni h (TUM), Germanywhere he kindled in me the uriosity for problems related to sour e oding. Being aDiploma Student at this time, I therefore de ided to work with João on my Diplomathesis in Muni h, a de ision, whi h started our ollaboration and eventually pavedthe way to ome to Porto and start my PhD. Parti ularly I would like to thank Joãofor his onden e in me largely allowing me to dene and follow my own resear hpath as well as giving me the possibility and the responsibility to supervise studentsduring their master theses. Furthermore, it was due to João's skills in getting fundingthrough resear h proje ts whi h eventually nan ed part of my resear h and allowedme to present my work at international onferen es giving me the han e to sharemy experien e with other resear hers in the eld; for this I am very grateful. WhenI arrived here to Porto I was the among the rst international studentsmaybe Iwas even the rst oneat the Department of Computer S ien e (DCC), Fa ulty ofS ien e at the University of Porto (FCUP), Portugal and I started in a small butprosperous group with another PhD student and one MS student. In was thanks toJoão's vision and his hard work that grew and shaped this group giving it the diverseand international setting it has now, opening up the possibility to share and dis ussour work together, and, thus, making us all better in what we do and how we work.I wish to express my gratitude to Prof. Muriel Médard at the Massa husettsInstitute of Te hnology (MIT), Cambridge, MA, USA who invited me for an extendedresear h stay at her resear h group from May to July 2008. Besides giving me theopportunity to indulge in the inspiring environment at MIT, I had the han e to meetProf. Médard for several brainstorming sessions where, among others, the idea for thejoint sour e-network de oder presented later in this thesis was born.My spe ial thanks go to Prof. Christina Fragouli, É ole Polyte hnique Fédéralede Lausanne (EPFL), Switzerland who gave me the opportunity to visit her prosperingresear h group from Mar h to May 2009. Although my resear h stay at EPFL is not7

dire tly ree ted in this thesis, I proted immensely from Prof. Fragouli's experien eas a s ientistamong other things, Prof. Fragouli's is the one who made me a tuallyrealize that sometimes it might be better (that it might be time wisely spend) to rstlook for a ounter example to a given problem than attempting to solve it right away.Parti ularly, I wish to thank Prof. Christina Fragouli for a epting to be a member ofthe do toral ommittee evaluating my thesis.There is another person I wish to mention here: Prof. Sergio D. Servetto, ba kthen at Cornell University, Itha a, NY, USA, who kindly invited me to work with himon a resear h problem related to this thesis. Unfortunately, Prof. Servetto passedaway in a tragi a ident; he surely was a person I would have liked to know betterboth on the personal as well as the professional level.My spe ial thanks to go to Prof. Paulo Ferreira at the University of Aveiro (UA),Portugal, and Prof. Antonio Porto as well as Prof. Luís Lopes both at FCUP forlending their expertise in evaluating this thesis as part of the do toral ommittee.My sin ere thanks also go to the the Fundação para a Ciên ia e Te nologia(Portuguese Foundation for S ien e and Te hnology) for their support under grantSFRH/BD/29918/2006 and SFRH/BD/70748/2010.The time I spent in Porto, Cambridge, and Lausanne would not have been thatwonderful without the onstant support of many people. These people range fromthe guys in the lab and my at mates, to the people I met here in Porto or whiletraveling; many of whom I now all my friends. Without you guys, without yourhelp and parti ularly without your ompany I would not have made it that far. Youhelped me in all kind of situations, with all kind of problems, be it bureau rati andorganizational issues like lling out formsheets to get a monthly bus ti ket, logisti operations like moving my whole inventory from A to B, resear h related problems likeanalyzing the properties of geometri obje ts in N−dimensional spa e, or hallengingreal-world problems like nding the best pla e to eat a Fran esinha (an intriguinglytasty, lo al dish) in Porto. You invited me to your homes, to meet your families, youtook me on holidays, you heered me up when everything seemed lost. There were(and still are) so many in redibly ni e people involved than I annot possibly mentionthem all here; some of them are (in alphabeti al order):Andrea C., Andrea F., Andreas (Andy) and Vera and Family, Andreas (Andi)and Suse and Family, Anisia, Anne, Annette, Antonio, Antti, Arnaldo, Biswajit (Biz),Bri e, Bruno, Carla, Carmen, Carolina, Carolin, Charlotte, Chee, Christian (Nulli),Crista, Danail, David F., David (Pikuz) P. and his Parents, Dejan, Despo, Diogo,Ela, Eleni, Elisa, Emre, Euri o, Federi a, Filipe B., Filipe S., Fran es a (Fran y) C.,Fran es a L., Fran es a P., Fran is o (Toni), Frank and Katja, Fulvio, Geo, Gisele,8

Giulia, Guilherme and Ana Maria and Family, Gustavo, Hana, Hanna, Hannes, Ian,Iiro, Inês, Ivo, Izabela, Ja obo, Jean-Baptiste (JB), Jennifer (Jenny), João, João Paulo(JP), Jorge and Sónia and Family, Karoll, Kassandra (Kassie), Katrin, Klaus (Klausi),Krisztina, Laura F., Laura V., Lav, Lea, Liliana, Linda, Lorenzo, Lu a, Ludovi o,Luisa, Luís C., Luís (Ciupman) P., Luís N., Maria (aka Kal Maria), Mari, Marlin,Marta, Marten, Marthe, Martin and Alex and Family, Mate and Sanja, Matteo, Maxi,Mi hael, Mikko, Minji, Mirjana, Nadia, Nausi a, Ni olau, Nikolas, Nuno, Olivía, Ota,Parviz and Family, Patrizia, Paulo F., Paulo M., Paulo O., Pavlina, Pedro C., PedroSa., Pedro Si., Philippe, Rayan, Roberta, Roland and Verena, Romeo, Rosa, Rui,Rumi, Sabina, Sarah, Saurabh, Sérgio, Sílvia, Simone B., Simone P., Sive, Sonia,Sree hakra (Sree), Stavroula, Stefania, Susana, Tamara, Tânia, Tania, Teresa, Thomas(Rossi) and Monika (Moni) and Family, Tiago, Titti, Tobias, Uli, Vinay, Virginie,Vítor, Waldir, William, Zenaida.Last but not least there are my parents, my sister, my brothers, and espe iallyalso my grandparents who supported me all the time. In the years I spend abroadmany things hanged, some of them happy, some of them sad, but all of them wereimportant to me. Sometimesseparated by the physi al distan e between Portugaland GermanyI ould not be there as mu h as I would, as mu h as I should, but Iknow that you understand and I am very grateful for it.Gerhard Maierba herPorto, June 2011

9

To my friends, old and new,but be ause of youI wouldn't have made it.To my grandparents,for your onstant support,in every aspe t of life.To my parents, my sister, and my brothers,be ause I know you are there, always,even if I wander far sometimes.To my nephew, Sebastian,now the time has ome!

11

ContentsAbstra t 3Resumo 5List of Tables 17List of Figures 191 Introdu tion 211.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.2 Prior State-of-the-Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231.3 Main Contributions and Outline . . . . . . . . . . . . . . . . . . . . . . 252 Sour e-Optimized Clustering 282.1 Motivation and Ba kground . . . . . . . . . . . . . . . . . . . . . . . . 282.2 System Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.2.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.2.2 Our Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.3 Index Assignment Design . . . . . . . . . . . . . . . . . . . . . . . . . . 352.3.1 Optimization Criterion . . . . . . . . . . . . . . . . . . . . . . . 352.3.2 Index-Reuse Algorithm . . . . . . . . . . . . . . . . . . . . . . . 362.4 Sour e-Optimized Clustering . . . . . . . . . . . . . . . . . . . . . . . . 392.4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.4.2 Clustering Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 402.4.3 Sour e-Optimized Fa torization . . . . . . . . . . . . . . . . . . 442.5 Results and Dis ussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.5.1 Randomly Pla ed Sensors . . . . . . . . . . . . . . . . . . . . . 482.5.2 The CEO Problem . . . . . . . . . . . . . . . . . . . . . . . . . 502.6 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5213

3 Diophantine Distributed Sour e Coding 533.1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.1.1 Stru tured Distributed Sour e Coding . . . . . . . . . . . . . . 533.1.2 Key Idea and Contributions . . . . . . . . . . . . . . . . . . . . 573.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.3 Preliminaries and Problem Statement . . . . . . . . . . . . . . . . . . . 613.3.1 System Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613.3.2 Sour e and Correlation Model . . . . . . . . . . . . . . . . . . . 623.3.3 Diophantine Index Assignments . . . . . . . . . . . . . . . . . . 643.3.4 De odability and Admissibility . . . . . . . . . . . . . . . . . . 653.3.5 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . 673.4 Diophantine Code Analysis and Combinatorial Design for the S alar Case 673.4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673.4.2 Diophantine Code Analysis . . . . . . . . . . . . . . . . . . . . 693.4.3 Combinatorial Sear h . . . . . . . . . . . . . . . . . . . . . . . . 753.5 Diophantine Code Design and the Fundamental Theorem of Arithmeti 773.5.1 Symmetri Design for N = 2 . . . . . . . . . . . . . . . . . . . . 783.5.2 Generalization for N > 2 . . . . . . . . . . . . . . . . . . . . . . 813.5.3 Maximum Supported Radius and Minimum Required SegmentNumber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843.5.4 The Fundamental Theorem of Arithmeti and its Impli ations . 853.6 Coding without Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 893.6.1 S alar Quantization . . . . . . . . . . . . . . . . . . . . . . . . . 893.6.2 Memoryless Binning . . . . . . . . . . . . . . . . . . . . . . . . 903.6.3 De oding Con epts and Complexity . . . . . . . . . . . . . . . . 913.6.4 S alable Joint De oding . . . . . . . . . . . . . . . . . . . . . . 923.7 Binning with Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . 953.7.1 Trellis Code Constru tion . . . . . . . . . . . . . . . . . . . . . 953.7.2 En oder Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 973.7.3 De oder Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 983.8 Quantization with Memory . . . . . . . . . . . . . . . . . . . . . . . . . 993.8.1 Preliminaries - Trellis Coded Quantization . . . . . . . . . . . . 1003.8.2 System Integration . . . . . . . . . . . . . . . . . . . . . . . . . 1013.9 System Analysis and Performan e Evaluation . . . . . . . . . . . . . . 1023.9.1 Dis rete-valued Sour es . . . . . . . . . . . . . . . . . . . . . . . 1023.9.2 Multivariate Gaussian Case . . . . . . . . . . . . . . . . . . . . 1053.10 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11214

4 Joint Sour e-Network Coding 1134.1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1144.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1154.3 System Setup and Problem Formulation . . . . . . . . . . . . . . . . . 1164.3.1 Network Topology and Sour e-Terminal Conguration . . . . . 1164.3.2 Sour e Model and Fa torization . . . . . . . . . . . . . . . . . . 1164.3.3 System Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1174.3.4 Pa ket Subgraphs . . . . . . . . . . . . . . . . . . . . . . . . . . 1184.3.5 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . 1194.4 Joint Sour e-Network De oder Design . . . . . . . . . . . . . . . . . . . 1194.4.1 Graphi al De oding Model . . . . . . . . . . . . . . . . . . . . . 1194.4.2 Iterative De oding Algorithm . . . . . . . . . . . . . . . . . . . 1224.5 Code Design for Sour e-Network Coding . . . . . . . . . . . . . . . . . 1244.5.1 Coding Instan es . . . . . . . . . . . . . . . . . . . . . . . . . . 1244.5.2 Joint Sour e-Network Code Design . . . . . . . . . . . . . . . . 1274.5.3 Open Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 1304.6 Proof-of-Con ept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1314.6.1 De oder Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 1314.6.2 Proof-Of-S alability . . . . . . . . . . . . . . . . . . . . . . . . . 1344.6.3 Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1374.7 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1385 Con lusions 1395.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1395.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140A Sour e-Optimized Clustering 143A.1 Optimal De oding Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 143A.2 E ient Marginalization and its Complexity . . . . . . . . . . . . . . . 144A.3 Complexity of Optimal De oding . . . . . . . . . . . . . . . . . . . . . 146A.4 Distortion Cal ulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 147A.5 Complexity of the Index-Reuse Optimization . . . . . . . . . . . . . . . 148A.6 E ient Sub-Optimal De oding and its Complexity . . . . . . . . . . . 149A.7 Complexity of Sour e-Optimized Clustering . . . . . . . . . . . . . . . 152A.8 Complexity of Sour e-Optimized Linking . . . . . . . . . . . . . . . . . 152B Diophantine Distributed Sour e Coding 154B.1 De odability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15415

B.2 Main Diagonal Distan e d(u, δ) . . . . . . . . . . . . . . . . . . . . . . 155B.3 The Segment Distan e d(c) and its Properties . . . . . . . . . . . . . . 156Referen es 157

16

List of Tables2.1 Simulation results - Low- omplexity distributed sour e oding, N = 100 492.2 Simulation results - Low- omplexity CEO oding, N = 100 . . . . . . . 514.1 Simulation results - Counterexample where separation does not hold ingeneral [RJCE06 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1324.2 Simulation results - System performan e . . . . . . . . . . . . . . . . . 1364.3 Simulation results - System performan e . . . . . . . . . . . . . . . . . 136

17

List of Figures1.1 Wireless sensor network s enario . . . . . . . . . . . . . . . . . . . . . . 222.1 System setup - Distributed sour e oding s enario . . . . . . . . . . . . 312.2 S alar quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.3 Clustering example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432.4 Linking example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.5 Simulation s enario - Graphi al representation of the sour e fa toriza-tion, N = 100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.1 System setup - Diophantine distributed sour e oding . . . . . . . . . . 543.2 Gaussian sour e model . . . . . . . . . . . . . . . . . . . . . . . . . . . 553.3 Additive sour e model . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.4 Typewriter sour e model . . . . . . . . . . . . . . . . . . . . . . . . . . 583.5 Diophantine ode omparison . . . . . . . . . . . . . . . . . . . . . . . 593.6 Geometri denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643.7 Admissible and de oded set . . . . . . . . . . . . . . . . . . . . . . . . 663.8 Diagonal segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683.9 Graphi al representation of the main diagonal distan e for N = 2, 3 and 4 693.10 Hierar hi al onstru tion method . . . . . . . . . . . . . . . . . . . . . 733.11 Interval test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753.12 Codeword position on destination diagonal . . . . . . . . . . . . . . . . 773.13 Symmetri design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793.14 Permutation of diagonal segments . . . . . . . . . . . . . . . . . . . . . 813.15 Supported set, maximum supported radius, and maximum requiredsegment number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 833.16 Systemati ode onstru tion based on prime fa tors . . . . . . . . . . 873.17 S alable de oding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933.18 Trellis-based binning - Considered trellises and label . . . . . . . . . . . 963.19 System setup - Trellis-based binning and index de oding . . . . . . . . 983.20 System Setup - TCQ en oder and de oder . . . . . . . . . . . . . . . . 10018

3.21 TCQ - Considered trellises and label . . . . . . . . . . . . . . . . . . . 1003.22 Simulation results - Rate-distortion performan e, N = 2 . . . . . . . . 1073.23 Simulation results - System performan e, N = 2, onguration (a) . . . 1093.24 Simulation results - System performan e, N = 2, onguration (b) . . . 1103.25 Simulation results - System performan e for N = 2, 3, 4, 8 . . . . . . . . 1114.1 Example network - Counterexample where separation does not hold ingeneral [RJCE06 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1174.2 System setup - Sour e-network oding s enario . . . . . . . . . . . . . . 1184.3 Probabilisti modelling - System omponents . . . . . . . . . . . . . . . 1204.4 Probabilisti modelling - Pa ket model . . . . . . . . . . . . . . . . . . 1214.5 Probabilisti modelling - De oding model . . . . . . . . . . . . . . . . . 1224.6 Sour e-network oding instan es . . . . . . . . . . . . . . . . . . . . . . 1254.7 Linear network oding example . . . . . . . . . . . . . . . . . . . . . . 1284.8 Systemati sour e-network oding . . . . . . . . . . . . . . . . . . . . . 1294.9 Diophantine sour e-network oding example . . . . . . . . . . . . . . . 1304.10 Considered s enario - Original and simplied network . . . . . . . . . . 135

19

Chapter 1Introdu tion1.1 MotivationSensor networks, onsiting of a (potentially) large number of distributed sensing de-vi es apable of sensing the physi al world and ommuni ating in multi-hop fash-ion [ASSC02, CK03, open up a whole new range of possible appli ations and op-portunities in areas su h as environmental and industrial monitoring, health are,and home automatization [CK03. The nodes in su h networks are multifun tional:they are equipped to dete t the environmental onditions or measure the physi alpro esses relevant for the onsidered appli ation (e.g., temperature, pressure, pH-level, me hani al stress); they are apable of performing basi data pro essing and oding operations; and they have a ommuni ation unit (e.g., a radio trans eiver) thatallows them to ommuni ate with other nodes in the network [CK03. Common tomost appli ation s enarios is that the network nodes are generally resour e onstraint(e.g., the nodes might only have limited energy, ommuni ation bandwith, or om-putational resour es to perform their tasks [CK03). Another important aspe t isthat the entire sensor network (generally) serves a ommon goal su h that the nodesgenerally have to jointly pro ess their data to a hieve a good trade-o between theoverall system performan e (e.g., the delity of the re overed sensor measurements)and the invested ost (e.g., the total number of required transmission to ommuni atethe measurements to their orresponding destinations). Joint data pro essing mightbe performed at dierent levels, e.g., the nodes might pro ess their own data subje tto some global goal, while forwarding the data of the other nodes, or the nodes mightpro ess their data ooperatively, i.e., they might pro ess jointly their own data togetherwith the re eived data from other nodes when produ ing the their output to be sentalong the network. 21

CHAPTER 1. INTRODUCTION 22

Figure 1.1: Data gathering in a wireless sensor network. The sour e nodes pi k upmeasurements about some spatially distributed pro ess. The goal is to en ode, transmittand re over the observed measurements at the sink nodes. The network nodes generally ommuni ate in a multi-hop fashion ooperatively forwarding data through the network.We will parti ularly onsider a data gathering s enario (e.g., see Figure 1.1),where spatially orrelated data from several sour e nodes has to be transmitted, in amulti-hop fashion to one sink node or more. Considering the ase where a ommontransmission medium has to be shared, or the power onsumption of the sensor nodes ishighly ae ted by the time of transmitter utilization, or both whi h is surely true forenergy- onstraint wireless sensor networks e ient, but simple, data ompressionte hniques are one of the key-fa tors in enabling a high spatial node density anda long network life-time. Assuming that both fa tors are ae ted by the number of ommuni ation operations required, any redu tion in the amount of data that needs tobe transmitted over the medium will have a de isive impa t on the power onsumptionof the individual sensor nodes and, thus, the longevity of the overall sensor network.The goal therefore is to en ode, transmitt and re onstru t the original sour e data(subje t to some delity riterion of interest) while minimizing the total number ofrequired transmissions in the network.Sin e the measurements pi ked up by the sensor nodes are (generally) orrelated,it is only natural to onsider data ompression s hemes that remove the inherentredundan y [XLC04. One way to a hieve this goal is to exploit the (joint) statisti s ofthe observed physi al pro ess and to allow the sensors to en ode their measurements ina distributed fashion, i.e., without any additional node ooperation or data ex hange.This method is alled distributed sour e oding [SW73, XLC04. Alternatively, thesensor nodes might ompress their data in a ooperative way, whi h an provide an

CHAPTER 1. INTRODUCTION 23ee tive solution when data is ommuni ated within a multi-hop framework, i.e, whereadditional information from forwarded pa kets may be retrieved and used within the oding pro ess.On the transmission side, network oding [ACLY00 has proven to be a key en-abler towards a hieving the maximum throughput in networks with one or more (gen-erally un orrelated) data sour es and several sinks. The key idea is to repla e routingoperations, that have to be performed when forwarding data pa kets at intermediatenetwork nodes, by oding operations. Those oding operations have the advantagethat (as opposed to routing) pa kets an be ombined as they traverse the networkallowing for more exibility when distributing the data pa kets within the network.This additional exibility an eventually be exploited to maximize the throughput or,alternatively, to minimize the total number of required transmissions that are requiredto ommuni ate the data pa kets from the sour es to the orresponding sinks.In s enarios where orrelated data is ommuni ated over a network to (possibly)more than one sink, omponents of both oding problems of distributed ompressionand of network oding an be found. Although distributed sour e oding andnetwork oding an be arried out separately, in a modular ar hite ture, we will seelater in this thesis that this modular approa h is (generally) suboptimal. The obvioussolution from an information-theoreti point of view is to opt for a joint sour e-network oding solution [SY01, HMEK04. Whether sear hing for a joint solution, or for aseparate solution, there are three major problems/ three open questions that need tobe addressed.• Design omplexity: How to design odes, in a reasonable amount of time, fora large number of sour es?• Sour e-optimized ode design: How to identify odes that are apable ofexploiting the sour e orrelations?• De oder omplexity: How to implement a (joint) de oder with tra table omplexity?This thesis aims at providing answers to these questions. We start with a brief reviewof prior ontributions to the eld while emphasizing their pra ti ability in s enarioswith a large number of sensor nodes.1.2 Prior State-of-the-ArtSin e Slepian and Wolf's Landmark paper [SW73, hara terizing the fundamentallimits of separate en oding of orrelated sour es, several authors have ontributed

CHAPTER 1. INTRODUCTION 24with distributed pra ti al sour e oding solutions (see, e.g., [XLC04 and referen estherein). Chara teristi to the proposed approa hes is that they either systemati allyexploit the joint properties of the sour es in short blo k length s enarios [FG87, CVA02,RMZG03, ZE03, or that they build on powerful hannel odes, operating ee tively atlong blo k lengths [GF01, BM01, LGS04, LXG02, without an expli it need to a ountfor the sour e properties. One possible ex eption is the oding s heme presentedin [PR99, PR03 whi h, in the broader sense, is sour e-optimized but works in short aswell as long blo k length s enarios. Whether those approa hes are sour e-optimized,or not, they are (usually) only feasible for a small numbers of sour es sin e their design omplexity, the resulting de oder omplexity, or both, be ome prohibitive when morethan two or three en oders are onsidered.Ahlwede et al. [ACLY00 onsidered the problem of ommuni ating (un orre-lated) sour es over a network to more than one sink and showed that the maximumnetwork throughput an only be a hieved by performing network oding [ACLY00. Apra ti al, algebrai framework for network oding based on linear odes was presentedin [KM03. Sin e the network oding fun tions an be hosen randomly [HKM+03,that is, based on a random hoi e of oding oe ients, the en oder design doesnot pose a serious problem even in s enarios with a large number of nodes. Similarstatements hold true for the de oder implementation sin e the de oding pro edureredu es to a simple matrix inversion whi h is omputationally tra table even for alarge number of network nodes.For s enarios where orrelated sour es have to be ommuni ated over a network,the a hievable rates were derived in [SY01. In [HMEK04 it was shown that linear odes are su ient to a hieve those rates when optimal de oders are used at the sink.Although some attempts have been made to redu e the omplexity of optimal de oders,e.g., [CME05, the de oding omplexity remained omputationally intra table for alarge number of en oders. Therefore, [RJCE06 asked the question on whether thejoint sour e-network oding problem an be separated and showed that this, in general,is not the ase.An important rst step towards providing a pra ti able de oding solution forlarge-s ale sensor networks (i.e., for s enarios with hundreds of sensor nodes) was madeby Barros and Tü hler in [BT06. In their work, whi h also will play an important rolein this thesis, the authors addressed a sour e- hannel oding problem in s enarios witha large number of independently operating and low- omplexity en oders (performings alar quantization alone) and a single joint de oder. In order to keep the de oding omplexity tra table, the authors proposed a s alable de oder whi h runs the sum-produ t algorithm [KFL01 on a arefully hosen fa tor graph [KFL01 approximation

CHAPTER 1. INTRODUCTION 25of the sour e orrelation.Despite these important ontributions, the question on how to pra ti ably per-form distributed ompression and joint sour e network oding in large-s ale sensornetworks remained largely open.1.3 Main Contributions and OutlineThe goal of this thesis is to provide pra ti al oding solutions for data gathering inlarge-s ale sensor networks. Sin e in the most general setting the pi ked-up sensormeasurements are (usually) orrelated and the data is sent over a multi-hop networkto (generally) more than one sink, we onsider both the ase of distributed ompressionand the ase of a joint sour e-network oding s enario. In order to provide sour e-optimized but s alable oding solutions, and in order to over ome omplexity problemsarising at the de oder, we propose the following three strategies.• Sour e-optimized lustering for distributed ompression: We onsider adistributed sour e oding s enario in whi h orrelated data from a large numberof independently operating sensors has to be en oded, transmitted, and re overedat a ommon re eiver subje t to a rate-distortion onstraint. We present a s al-able solution based on the following key elements: distortion-optimized index-reuse for low- omplexity distributed quantization, sour e-optimized hierar hi al lustering, and s alable de oding based on graphi al models.• Sour e-optimized ode design based on Diophantine analysis: We ad-dress the design of sour e-optimized distributed sour e odes apable of exploit-ing the symmetries ommon to many relevant sour e models. Using basi toolsfrom number theory, spe i ally Diophantine analysis, we propose a onstru tiveframework for designing memory-less index assignments subje t to a zero-error riterion; whi h we then apply to s enarios that onsider the error probabilityor the (mean squared error) distortion as the primary performan e riterion.Short as well as long blo k length s enarios are onsidered to allow for a exibletrade-o between the system's performan e and its operational omplexity.• S alable joint sour e-network oding: We onsider a s enario where or-related data has to be ommuni ated over a network to more than one sinkwhile onsidering the error probability or the (mean squared error) distortion asdelity riterion. Given the network topology and the orrelation stru ture ofthe data, we show how to probabilisti ally represent the overall system (in ludingthe pa kets' paths) by a graphi al model. The obtained graph is then used for an

CHAPTER 1. INTRODUCTION 26e ient de oder implementation. Furthermore, we briey address the problemof possible ode onstru tions for joint sour e-network oding.In a ordan e with these strategies, the rest of this thesis is organized as follows. InChapter 2 we show how sour e-optimized lustering an be used to enable distributed ompression in large-s ale sensor networks. We begin with the design of low- omplexitydistributed sour e odes. Inspired by [FG87, we formulate a generalized index-reuse optimization algorithm for the design of distributed s alar quantizers. Sin ethis optimization is only feasible for small numbers of sensors, we devise a sour e-optimized hierar hi al lustering algorithm using the Kullba k-Leibler distan e [CT91as optimization riterion. The algorithm partitions the set of all sensors into orrelationpreserving lusters of a maximum predened size. This eventually allows us to applythe quantizer optimization within ea h of the lusters and, thus, to all the sensors inthe system redu ing the overall design omplexity signi antly. In the se ond partwe show how the obtained lusters an be in orporated in a sour e-optimized fa torgraph [KFL01 using the Kullba k-Leibler distan e as optimization riterion. Theobtained fa tor graph is then used for an e ient de oder implementation, based onthe iterative sum-produ t algorithm [KFL01, extending the work in [BT06. Finally,we show how our te hniques an be applied to general sensor network s enarios as wellas the so- alled CEO problem [BZV96 and provide numeri al results for setups with100 en oders.Chapter 3 presents the design of distributed sour e odes based on so- alledDiophantine index assignments a simple lass of memory-less index assignmentsthat an be des ribed by linear Diophantine equations.1 After reviewing some sour emodels that are parti ularly relevant for distributed sensing s enarios, and after pro-viding some formal denitions, we show how the overall oding system (subje t tothe zero-error riterion) an be analyzed using the number-theoreti al properties ofthe Diophantine index assignments. Using the obtained insights from the Diophantine ode analysis, we formulate an algorithm for the systemati ode design using insightsobtained by the fundamental theorem of arithmeti . We show how to integratethe Diophantine index assignments into our system and show how de oding an beperformed and optimized for large numbers of en oders assuming a s alar quantizationapproa h. Based on the obtained (memory-less) index assignments, we then show howto onstru t sour e-optimized trellis odes. We on lude with a detailed analyti aland numeri al performan e evaluation.1Named after the Greek mathemati ian Diophantus of Alexandria who worked on problems on erning integer solutions to algebrai equations.

CHAPTER 1. INTRODUCTION 27The main goal in Chapter 4 is to show how lose-to-optimal de oders for jointsour e-network oding problems an be implemented while remaining omputationallytra table even for a (potentially) large number of en oders. The main idea is to exploitknowledge about the network topology as well as the orrelation stru ture of the datafor an e ient de oder implementation based on fa tor graphs [KFL01. In parti ular,we show how to step-by-step onstru t a (probabilisti ) fa tor graph de oding modeland show how the iterative sum-produ t algorithm [KFL01 an be employed for lose-to-optimal de oding. We show that, depending on the a tual network onguration,the de oding omplexity is mostly governed by the maximum node degree (withexponential dependen y) and not by the number of network nodes (approximatelyquadrati dependen y). Besides this main goal, we also briey address the problemof possible ode onstru tions for joint sour e-network oding. To underline theee tiveness and versatility of our approa h a proof-of- on ept in an form of a workingde oder implementation onsidering the ounter example for separation in [RJCE06as well as a sensor network s enario with 30 network nodes.Chapter 5 on ludes this thesis with some on luding remarks and a brief surveyabout possible extentions and dire tions for future work.The work ondu ted in the s ope of this thesis was published in the following in-ternational journals [MB09; it is under review in the following international jour-nals [CMB11a, MB11; it was presented at the following international onferen es andworkshops [MBM09, AMB09, MB07b, MB07a, MB06b, MB06a, MB05; and it wasa epted for the following international onferen e [CMB11b.

Chapter 2Sour e-Optimized ClusteringIn this hapter we take a rst step towards the main goal of providing pra ti al odingsolutions for large-s ale sensor networks (i.e., for s enarios with hundreds of sensornodes). In parti ular, we onsider a distributed sour e oding s enario in whi h orrelated data from a large number of independently operating sensors has to been oded, transmitted, and re overed at a ommon re eiver subje t to a rate-distortion onstraint. This goal shall be approa hed by ombining a low- omplexity but (atthe same time) sour e-optimized en oding s heme with an equally sour e-optimized lustering approa h, eventually giving rise to a omputationally tra table ode designeven in large-s ale s enarios. We furthermore show how the obtained lusters an beused for a sour e-optimized, and hen e e ient, implementation of a lose-to-optimaljoint de oder. Although only low- omplexity en oders are onsidered in this hapter,we will see that the ombination of the proposed strategies sour e-optimized odedesign, sour e-optimized lustering, and sour e-optimized joint de oding allows usto ee tively apply distributed sour e oding prin iples to large-s ale sensor networks.2.1 Motivation and Ba kgroundIn distributed sensing s enarios, where orrelated data has to be gathered by a largenumber of low- omplexity, power-restri ted sensors, e ient sour e oding and datagathering te hniques are key towards redu ing the required number of transmissionsand enabling extended network life-time. Inspired by the seminal work of Slepian andWolf [SW73, hara terizing the fundamental limits of separate en oding of orrelatedsour es, several authors have ontributed with distributed sour e oding solutions(e.g., see [XLC04 and referen es therein). Fo using on s alar quantization, Flynn andGray [FG87 provided one of the rst pra ti al approa hes to onstru t distributedsour e odes for two ontinuous-valued sour es. The basi idea behind this approa h28

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 29 whi h will also play an important role in our work is to reuse the indi es ofa high-resolution quantizer su h that the overall end-to-end distortion (after jointde oding) is minimized. Pradhan and Ram handran presented in [PR99 a method alled distributed sour e oding using syndromes (DISCUS), whi h is based on hannel odes with good distan e properties partitioning the set of all possible odewordsinto o-sets and transmitting only the o-set's syndrome and not the a tual odewordto the de oder. This method, originally onsidered for an asymmetri s enario whereinformation about one sour e is available as side information at the de oder, wasre ently extended to the symmetri ase [PR05 where all sour es are to be en odedand side information is not available at the de oder. An alternative approa h for theasymmetri s enario was provided by Zamir et al. [ZSE02 and by Servetto in [Ser00where a onstru tive approa h for Gaussian sour es based on linear odes and nestedlatti es was presented. Cardinal and Van Ass he [CVA02 as well as Rebollo-Monederoet al. [RMZG03 fo used on the optimization of the quantization stage and proposeddesign algorithms for multiterminal quantizers. Beyond these ontributions, highlyevolved iterative hannel oding te hniques su h as low density parity he k (LDPC)and turbo odes have been applied to the distributed sour e oding problem [XLC04,rea hing the fundamental limits of Slepian and Wolf [SW73.Despite these important ontributions, very little is known on how to performdistributed ompression in large-s ale sensor networks (i.e., in s enarios with hundredsof sensor nodes). The main reason is that most approa hes be ome infeasible whenthe omplexity of joint de oding or the omplexity of a joint design of separateen oders is onsidered for a large number of orrelated sour es. Previous work towardsthis goal produ ed a s alable solution for the de oding side by running the sum-produ t algorithm on a arefully hosen fa tor graph approximation of the sour e orrelation [BT06. In this paper, we present a s alable solution whi h in ludesthe en oding side. The main idea is to redu e the number of quantization bitsin a systemati way, exploiting orrelation preserving lusters, whi h minimize theKullba k-Leibler Distan e (KLD) between the given sour e statisti s and a fa torgraph approximation. Our main ontributions are as follows.• Design of Low-Complexity Distributed Sour e Codes: We propose amethodology to design quantizers for a large number of sensors whi h exploitsthe spatial orrelation between sensor measurements. Inspired by [FG87 weformulate a generalized index-reuse optimization algorithm whi h allows us toredu e the number of bits for data transmission by adding to our system a oarsequantization stage.• Sour e-Optimized Clustering: We devise a hierar hi al lustering algorithm

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 30that uses the joint probability density fun tion (PDF) of the sensor measure-ments to partition the set of all sensors into lusters and prove that the om-plexity of quantizer design an be redu ed signi antly.• Combination with Fa tor Graph De oding: We show how sour e-optimized lusters used for distributed sour e oding an be in orporated in a KLD opti-mized fa tor graph whi h, in turn, is used at the de oder to exploit sour e orrelations in a omputationally tra table way.• Simulation Results: We show how our te hniques an be applied to generalsensor network s enarios as well as the so- alled CEO problem [BZV96 andprovide numeri al results for setups with 100 en oders.The rest of the paper is organized as follows. In Se tion 2.2 we give a pre ise formu-lation of the problem setup and des ribe the underlying system model. In Se tion 2.3we present a te hnique to optimize quantizers exploiting orrelations in the sour eobservations. Se tion 2.4 des ribes our s alable solution based on sour e-optimizedhierar hi al lustering in sensor networks. The results of numeri al experiments aredis ussed in Se tion 2.5. We on lude with a brief summary in Se tion 2.6.2.2 System SetupWe start by introdu ing our notation. Random variables are denoted by apital letters,e.g., U , where its realizations are denoted by the orresponding lower ase letters, e.g.,

u. Ve tors are denoted by bold letters and, if not stated dierently, assumed to be olumn ve tors, e.g., u = (u1, u2, . . . , uN)T and U = (U1, U2, . . . , UN)T . The length−Nall-zero ( olumn) ve tor is denoted as 0TN = (0, 0, . . . , 0)T . Matri es are denoted bybold apital letters, e.g., A. Verti al bars |·| are used to denote the matrix determinant,e.g., |A|. The expression IN is the N × N identity matrix. It is always lear fromthe ontext, or stated expli itly, if a bold apital letter refers to a ve tor of randomvariables or to a matrix. Index sets are denoted by apital alligraphi letters, e.g., N ,unless otherwise noted. The set's ardinality is referred to by the usage of verti al bars

| · |, e.g., |N |. We follow the onvention that variables indexed by a set denote a set ofvariables, e.g., if N = 1, 2, 3 then uN = u1, u2, u3, and we use the same on eptto dene ve tors of variables, e.g., uN = (u1, u2, u3)T . Furthermore, the entries ofa ve tor are referred to by spe ifying its index within paretheses, for example, u(0)refers to the rst and u(N − 1) to the last entry of the length−N ve tor u.The ovarian e is dened by Cova,b = EabT−EaEbT , where E· isthe expe tation operator.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 31

. . .

. . .

u1

u2

i1

i2

u1

u2

R1

R2

w1

w2

..

.

..

.

..

.

Encoder e2

Encoder e1

Sou

rce

q1

q2

qN

m1

m2

mN

p(u

1,u

2,.

..,u

N)

uN iN wN RNuN

Encoder eN

Join

tD

ecod

erΦ

Figure 2.1: System model. N orrelated sour es are en oded independently andde oded jointly. At ea h en oder en, n ∈ 1, 2, . . . , N, the observed sour e symbol unis en oded onto the odeword wn and ommuni ated to the joint de oder Φ at rateRn. In the rst stage of en oding, the dis rete sour e index in is obtained from un bythe s alar quantizer qn and, subsequently, wn is obtained by the index assignment mnsu h that en = mn qn. After perfe t transmission the joint de oder uses the ve torof re eived odewords w = (w1, w2, . . . , wN)T and its knowledge about the sour estatisti s p(u1, u2, . . . , uN) to jointly form the estimates u = (u1, u2, . . . , uN)T .An N−dimensional random variable with realizations u = (u1 u2, · · · , uN)T ∈

RN is Gaussian distributed with meanµµµ = Eu and ovarian e matrixΣΣΣ = Covu,uwhen p(u) is given by

p(u) =1

√

(2π)N |Σ|exp

(

−1

2(u− µ)TΣ−1(u− µ)

)

. (2.1)Su h a PDF is simply denoted as N (µ,Σ).2.2.1 System ModelWe onsider a setup of N independently operating sensors. In this setup ea h sensorindexed by n ∈ N , N = 1, 2, · · · , N, observes a ontinuous-valued sour e sampleun(t) at time instant t. For simpli ity, only spatial orrelations between measure-ments and not their temporal dependen e is onsidered su h that the time index tis dropped and only one time instant is onsidered. The ve tor of sour e samplesu = (u1, u2, · · · , uN)T , u ∈ R

N , at ea h time instant t is assumed to be one realizationof a N-dimensional Gaussian random variable distributed a ording to N (µ,Σ) with

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 32the ve tor of mean values µ = 0TN and the ovarian e matrix Σ set equal to the orrelation matrix

R =

1 ρ1,2 · · · ρ1,N

ρ2,1 1 · · · ρ2,N... ... . . . ...ρN,1 ρN,2 · · · 1

,su h that the individual sour e samples un, n ∈ N , have zero mean EUn = 0, unitvarian e CovUn, Un = 1 and are orrelated with um, m ∈ N , m 6= n, a ording tothe orrelation oe ient ρn,m = CovUn, Um. Gaussian models for apturing thespatial orrelation between sensors at dierent lo ations are dis ussed in [SS02 andmodels for the orrelation oe ients of physi al pro esses unfolding in a eld an befound in [DN97.We assume that the sensors are low- omplexity devi es onsisting only of a s alarquantizer followed by an index assignment stage (see Figure 2.1). Spe i ally, we onsider the following en oding pro edure for ea h sensor n ∈ N :In the rst step, the observed sour e samples un ∈ R are mapped onto quantiza-tion indi es in ∈ In, In = 0, 1, . . . , |In|−1, by the quantization fun tion qn : R→ Insu h that in = qn(un). During quantization, an input value un is mapped onto theindex in if it falls into the interval Bn(in) ⊆ R between the de ision levels bn(in)and bn(in + 1) su h that bn(in) < un ≤ bn(in + 1) (see Figure 2.2). The obtainedquantization index in is then asso iated with the re onstru tion level un,in ∈ Un,Un = un,0, un,1, . . . , un,|In|−1, representing all sour e samples un falling into thequantization region Bn(in). We onsider PDF optimized quantizers su h that themean squared error (MSE) E||Un − Un||2 = E(Un − Un)2 =

∫∞

un=−∞(un,qn(un) −

un)2 · p(un) dun within the observations is minimized, e.g., see [JN84), whi h impliesthat the re onstru tion levels un,in are hosen to be the entroid ( onditional expe tedvalue) of the quantization region Bn(in), i.e., un,in = EUn|in for all in ∈ In.

bn(in − 1) bn(in) bn(in + 1)

Bn(in)

un,in−1 un,in

Figure 2.2: S alar quantization. The sour e samples un ∈ R are mapped onto theindex in ∈ In if they fall into the quantization region Bn(in) su h that bn(in) < un ≤

bn(in + 1). Those samples, i.e., all samples un ∈ Bn(in), are then represented by theirre onstru tion level un,in ∈ Un.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 33In the se ond step of en oding, the obtained quantization index in ∈ In is mappedonto the odeword wn ∈ Wn, Wn =0, 1, . . . , |Wn| − 1, by the mapping fun tion, also alled the index assignment, mn : In → Wn su h that wn = mn(in). We dene themapping fun tion to be surje tive, i.e., for any wn ∈ Wn there exists at least onein ∈ In su h that wn = mn(in), for n = 1, 2, . . . , N . This property shall be importantlater on.In summary, the en oder of ea h sensor operates in a sequential way and theoverall en oding fun tion an be expressed as en = mn qn su h that wn = en(un) =

mn(qn(un)). The data rate at whi h the odewords wn are transmitted to the de oderis dened as Rn = ⌈log2(|Wn|)⌉ [bit.Assuming data transmission over an array of N ideal hannels, the de oder usesve tor of odewords w = (w1, w2, . . . , wN)T ∈ W , W =∏N

n=1Wn, and availableknowledge of the sour e orrelation R to form the estimate u = (u1, u2, . . . , uN)T , u ∈R

N , of the originally observed sour e samples u ∈ RN . The de oding fun tion is denedas Φ : W → R

N su h that u = Φ(w). Assuming that the MSE E||U−U||2 betweenthe estimates U = (U1, U2, . . . , UN )T and sour e samples U = (U1, U2, . . . , UN)T is thedelity riterion to be minimized by the de oder, we observe thatE||U−U||2 = E(U−U)T · (U−U) = E

N∑

n=1

(Un − Un)2 =

N∑

n=1

E(Un − Un)2,(2.2)whi h shows us that E||U−U||2 an be minimized globally by lo al minimizationof the terms E(Un − Un)2 for n = 1, 2, . . . , N . The optimal estimate un(w) for agiven odeword ve tor w, i.e., su h that E(Un − Un)2 is minimized globally, an beobtained by onditional mean estimation (CME), e.g., see [Poo94, su h thatun(w) = EUn|w

(a)=

|In|−1∑

in=0

EUn|in · p(in|w)(b)=

|In|−1∑

in=0

un,in · p(in|w) (2.3)where equality (a), as derived in Appendix A.1, allows us to express the estimateun(w) as a fun tion of EUn|in and, thus, as a fun tion of the re onstru tion levelsassuming that un,in = EUn|in as onsidered in (b).The required posterior probabilities p(in = l|w) an be derived by

p(in = l|w)(a)= γ · p(in = l,w)

(b)= γ ·

∑

∀i∈I:in=l

p(w, i), (2.4)where the Bayes rule was applied in (a) using the onstant γ=1/p(w) for normalizingthe sum over all probabilities to one and in (b) we al ulate p(in = l,w) from p(w, i) bymarginalizing over all possible realizations of i = (i1, i2, . . . , iN)T ∈ I, I =∏

∀n∈N In.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 34It is possible to express p(w, i) in terms of the probability p(i) known a-priori fromthe sour e statisti s and the transition probabilities p(wn|in) known from the indexassignments mn for n = 1, 2, . . . , N su h thatp(w, i)

(a)= p(w|i) · p(i)

(b)= p(i) ·

∏

∀n∈N

p(wn|in), (2.5)where the Bayes rule was applied in (a), and (b) takes into a ount that the index as-signment operation performed at ea h en oder is independent from the other en oders.The probability mass fun tion (PMF) p(i) of the index ve tors i an be obtained bynumeri ally integrating the sour e PDF p(u) over the quantization region dened byBn(in) for all en oders n = 1, 2, . . . , N . Alternatively, one an resort to Monte Carlosimulation or approximate p(i) by other means. Considering implementation issuesit is worth pointing out that the transition probabilities p(wn|in) are either zero orunity sin e the mapping mn from the indi es in to the odewords wn is a fun tion (i.e.,knowledge of the index in implies knowledge of the odeword wn). Thus, the produ t ofthe transition probabilities in (2.5) is also zero or unity, a fa t, whi h an be exploitedfor an e ient implementation of the marginalization as shown in Appendix A.2.The omplexity of optimal de oding is analyzed in Appendix A.3 and using thederived result we are able to state that the omputational omplexity of al ulating allestimates a ording to (2.3) is of O(NF N ) where F ≤ L−K + 1 is a system spe i parameter depending on the hara teristi s of the mapping fun tions.12.2.2 Our GoalsUnder the system model des ribed above, our rst goal is to nd distributed sour e oding algorithms that, by joint design of the index assignments, oer a suitablesolution for large numbers of en oders. Inspired by the work in [FG87, we formulatea generalized index-reuse algorithm to onstru t, for subsets of en oders, distortion-optimized index assignments suitable for distributed sour e oding.Sin e optimal de oding a ording to (2.3) is not feasible for large number ofsour es, part of this work shall be devoted to sub-optimal, yet feasible, de oders basedon the prin iples presented in [BT06.1It is worth pointing out that de oding a ording to (2.3) has only to be performed, in prin iple,only on e for ea h realization w ∈W and that the al ulated estimate ould be then stored in theform of a de oding table for n = 1, 2, . . . , N . Thus, the de oding operation would redu e to a meretable look-up, i.e., de oding of N sour es would be of omputational omplexity O(N). However,su h a de oding table itself has a spa e omplexity of O(NKN ) and the omputational omplexityfor reating it would be NKN times the omplexity of de oding a single sour e, i.e., it would be ofO(NKNFN ). Therefore, the on ept of using a de oding table shall be dis arded throughout thiswork.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 35We shall show that sour e-optimized lustering algorithms an be a key enablertowards the goal of obtaining both a s alable en oding and de oding solution feasiblefor large-s ale sensor networks.2.3 Index Assignment DesignThe distributed sour e oding on ept followed throughout this work is hara terizedby the property that it an be represented by a simple index assignment stage, i.e., by aone-to-one mapping from the quantization indi es in ∈ In to the odewords wn ∈ Wnsu h that wn = mn(in) for all n ∈ N . Considering this low- omplexity approa h,distributed ompression an be a hieved by hoosing |Wn| < |In|, i.e., whenever wehave fewer odewords than quantization levels.2 Thus, the data rate an be redu edfrom R′n = ⌈log2 |In|⌉ to Rn = ⌈log2 |Wn|⌉ [bit for several en oders n ∈ N . The goalis to jointly design su h index assignments su h that the end-to-end distortion d(Ψ)for an arbitrarily hosen subset of sour es Ψ ⊆ N is minimized. The design pro edurepresented in the following was inspired by [FG87 where a oding solution for two orrelated observations was presented. In this work, we generalize the orrespondingdesign algorithm to onstru t distortion optimized index assignments for an arbitrarysubset of en oders Ω ⊆ N . It is worth mentioning that, in prin iple, several othermethods an be used to onstru t suitable index assignments, e.g., those based onsyndromes [PR99 or even on random index assignments. However, the presentedmethod has the advantage that it is very versatile and that the distortion itself servesas an optimization riterion within the design.2.3.1 Optimization CriterionThe gure of merit for our optimization pro edure is the minimization of the end-to-end distortion d(Ψ) for an arbitrarily hosen subset of sour es Ψ ⊆ N whi h, for the ase of a MSE distortion metri , an be expressed as follows:

d(Ψ) = E||UΨ −UΨ||2 = E(UΨ −UΨ)T · (UΨ −UΨ)

= E∑

∀n∈Ψ

(Un − Un)2 =∑

∀n∈Ψ

E(Un − Un)2, (2.6)2Su h index assignments generally in rease the distortion of the system, be ause information islost during the mapping pro ess, i.e., more than one quantization index in might lead to one and thesame odeword wn. However, sin e the rate an be redu ed onsiderably, this method oers a wayto a hieve a wider range of rate-distortion trade-os.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 36where UΨ denotes the ve tor of onsidered sour e variables Un and, equally, UΨdenotes the ve tor of estimates Un onsidered within the al ulation, n ∈ Ψ. InAppendix A.4 it is shown that the distortion asso iated with ea h sour e d(n) =

E(Un − Un)2, n ∈ Ψ, an be expressed as followsd(n) = E(Un − Un)2+ E(Un − Un)2, (2.7)where the re onstru tion levels of the quantizers are assumed to be the entroid of thequantization ells su h that un,in = EUn|in for all in ∈ In. We see that the distortion

d(n) onsists of two omponents: the omponent dq(n) = E(Un − Un)2 dire tlyresults from the (nite) granularity of the (s alar) quantizers, and the omponentdd(n) = E(Un − Un)2 =

∑

∀iΨ∈IΨ

p(iΨ) · (un(wΩ)− un,in)2 (2.8)is mainly ae ted by the hoi e of the estimate un(wΩ) for the given the odewordve tor wΩ ∈ WΩ, WΩ =∏

∀l∈ΩWl, Ω ⊆ N (whi h an be al ulated by summingover all possible index ve tors iΨ ∈ IΨ, IΨ =∏

∀n∈Ψ In, Ψ ⊆ N ). Thus, the overalldistortion for all sour es n ∈ Ψ is given byd(Ψ) =

∑

∀n∈Ψ

d(n) =∑

∀n∈Ψ

(dq(n) + dd(n)) =∑

∀n∈Ψ

dq(n) +∑

∀n∈Ψ

dd(n), (2.9)de oupling the distortion (solely) aused by the quantization stage dq(Ψ) =∑

∀n∈Ψ dq(n) from the distortion (mainly) aused by the index assignment stage dd(Ψ) =∑

∀n∈Ψ dd(n). It is worth pointing out that dq(Ψ) is only ae ted by the propertiesof the quantizers, and not by the index assignments; therefore it an be seen as anarbitrary onstant and does not need to be onsidered during the index assignmentdesign, as presented next.2.3.2 Index-Reuse AlgorithmThe basi idea underlying the presented algorithm is to onstru t index assignments inan iterative fashion. In ea h step of this pro edure, the number of output odewordsis redu ed su h that, in general, more than one quantization index is assigned to ea h odeword index. This means that the odeword indi es are reused onsidering thein urred end-to-end distortion as optimization riterion.Starting with bije tive mappings between the quantization indi es in and the odewords wn, with the number of odewords being equal to the number of quanti-zation indi es, i.e., |In| = |Wn|, the algorithm subsequently modies the mappingfun tions mn for all onsidered en oders n ∈ Ω by merging two odewords (or,equivalently, the originating quantization indi es) to a single new odeword.3 This3We note that the sear h algorithm is not optimal due to the single-step nature of the optimization.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 37is repeated until the targeted number of odewords is rea hed. In ea h step ofthe pro edure, the algorithm hooses, among all possible andidates, the mergingoperation that yields the minimum distortion dd(Ψ).For the following dis ussion of the algorithm, we assume that in the beginningof the pro edure |In| = |Wn| = L for all n ∈ Ω; the goal is to ensure that at the endof the pro edure |Wn| = K, K < L, for all n ∈ Ω. For implementation purposes, wefurthermore assume thatWn = 0, 1, . . . , K−1, and represent the mapping fun tionsmn : In → Wn by ve tors fn ∈ WL

n , allowing us to des ribe the mapping fromthe indi es in to the odewords wn by simple ve tor referen ing, su h that wn =

mn(in) = fn(in) for all in ∈ In, n ∈ Ω. The merging of two odewords wn = a, b,a, b ∈ Wn, within the ve tor fn ∈ WL

n onto a new ve tor en ∈ VLn shall be des ribedby the merging fun tion g : WL

n ×Wn ×Wn → VLn , su h that en = g(fn, a, b), where

Vn ∈ 0, 1, . . . , K − 2 is the resulting odeword alphabet with a redu ed number ofK − 1 odewords. Assuming that at the initialization of the algorithm the ve torfn was initialized su h that fn = (0, 1, . . . , L − 1)T and that a < b, then it is easyto show that en an be obtained from fn by performing the following assignment forin = 0, 1, . . . , L− 1:

en(in) =

a , for fn(in) = a or fn(in) = b

fn(in)− 1 , for fn(in) > b

fn(in) , otherwise.Let En = fm : m ∈ Ω, m 6= n ∪ en be the olle tion of mapping ve tors aftermerging two odewords in fn. We use the notational onvention that dd(Ψ, En) an beused to indi ate that the distortion dd(Ψ) a ording to (2.8) was al ulated based onthose mapping fun tions. A detailed formulation of the whole pro edure an be foundin Algorithm 1.It is worth pointing out that the presented algorithm onstru ts index assign-ments that are surje tive fun tions.4For the important ase where the set of onsidered sour es is equal to the set of onsidered en oders, i.e., for the ase where Ψ = Ω, the omplexity of the optimizationalgorithm is dis ussed in detail in Appendix A.5. It is shown that the algorithm an beimplemented with a omputational omplexity that grows exponential with |Ω|makingit feasible only for a small number of en oders |Ω|. A reasonable way to de rease4The algorithm is initialized with bije tive fun tions whi h, by denition, are also surje tive. Inthe following step of the pro edure two odewords are merged onto a single new odeword resultingin a surje tive fun tion. This holds true for ea h step of the pro edure and therefore, by indu tion,the mapping reated after any number of steps is (still) a surje tive fun tion.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 38Algorithm 1: Index-reuse optimizationInitialization1• start with one-to-one mapping2

fn ← (0, 1, . . . , L− 1)T for all n ∈ Ω3• set initial number of odewords4

k ← L5 Main Loop6 while (k > K) do7 for (n ∈ Ω) do8• set referen e distortion to maximum9

d∗ ←∞10 for (a = 0, 1, . . . , k − 2) do11 for (b = a + 1, a + 2, . . . , k − 1) do12• merge ell a and ell b within fn13

en = g(fn, a, b)14• al ulate resulting overall distortion15

d = dd(Ψ, En)16 if (d < d∗) then17• save urrent mapping and distortion18

d∗ ← d19fn ← en20

• redu e number of odewords by one21k ← k − 122

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 39the overall omplexity for a large number of en oders is to form sour e lusters andoptimize the index assignments for ea h luster individually, as explained next.2.4 Sour e-Optimized ClusteringThe need for a omputationally feasible ode design motivates us to partition the entireset of en oders into mutually ex lusive subsets, so- alled lusters. The en oders anthen be optimized within ea h luster, thus, redu ing the optimization eort for theen oding side. Moreover, this lustering and oding strategy an be easily ombinedwith the s alable de oder presented in [BT06 whi h relies on a arefully hosen fa torgraph model and allows for joint de oding of the data sent by all en oders. Thekey towards omputationally feasible joint de oding is for the de oder to use anapproximated probability density fun tion (PDF) p(u) instead of p(u) as basis fore ient de oding onsidering only the statisti al dependen ies within ertain subsetsof sour es. Therefore, it be omes ru ial to build the de oding model and the sour e lusters alongside to ensure that the statisti al dependen ies, whi h are exploitedduring en oding, are still available at the de oder to ompensate for the informationloss in urred at the index assignment stage. In [BT06 the Kullba k-Leibler Distan e(KLD) was deemed to be a suitable measure to estimate the impa t of the hosende oding model onto the overall system performan e, i.e., the mean squared error(MSE) distortion. We therefore hose the KLD as optimization riterion to nd notonly a suitable sour e approximation but also adequate lusters.2.4.1 PreliminariesThe PDF p(u) an be approximated by assuming a fa torization of the form p(u) =∏M

m=1 fm(uSm) where Sm ⊆ N for m = 1, 2, . . . , M are subsets of sour e indi es su hthat ⋃M

m=1 Sm = N . Sin e generally p(u) 6= p(u), the resulting PDF p(u) is anapproximation of p(u).Spe i ally, we shall onsider so- alled onstrained hain rule expansions(CCREs) [BT06 of p(u) that an be obtained from the regular hain rule expansionby removing some of the onditioning variables. More formally, a fa torizationp(u) =

M∏

m=1

fm(uSm) =

M∏

m=1

p(uAm|uBm

), (2.10)where Am, Bm and Sm = Am ∪ Bm are subsets of the elements in N , is a CCRE of

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 40p(u), if the following onstraints are met for m = 1, 2, . . . , M :

Am ∩ Bm = ∅,M⋃

m=1

Am = N , Bm ⊆m−1⋃

l=1

Al. (2.11)Noti e that the set B1 is always empty and that Bm =⋃m−1

l=1 Al holds for the usual hain rule expansion. We all a CCRE symmetri if Bm is a subset of Sl for somel < m and for m = 2, 3, . . . , M .The KLD between a PDF p(u) and its approximation p(u) is dened as

D(p(u)||p(u)) =

∫

· · ·

∫

p(u) log2

p(u)

p(u)du, (2.12)e.g., see [CT91, whi h is used as optimization riterion to onstru t sour e fa toriza-tions. In [BT06 it was shown that the KLD an be al ulated expli itly for CCREsof Gaussian PDFs N (0T

N ,R) as followsD(p(u)||p(u)) = −

1

2log2 |R|+

M∑

m=1

∆D(Sm,Bm), (2.13)where the KLD benet obtained by introdu ing the fa tor p(uAm|uBm

) is given by∆D(Sm,Bm) =

1

2log2

|RSm|

|RBm|, (2.14)where RSm

as well as RBmare the ovarian e matri es of the Gaussian PDFs p(uSm

)and p(uBm), respe tively.It is worth pointing out that a sour e fa torization a ording to (2.10) an dire tlybe used for an e ient de oder implementation as dis ussed in Appendix A.6. Inparti ular, this holds for the ase where the number of variables in the fa tors isbounded su h that |Sm| ≤ S for m = 1, 2, . . . , M . A omplexity analysis for s alablede oding based these assumptions an be found in Appendix A.6. It is shown thatthe omputational omplexity for the ase where |Bm| = 1 for m = 1, 2, . . . , M is of

O(MSF S). In the other ases the omputational omplexity is of O(TMSF S) whereT > 1 spe ies the maximum number of iterations used for de oding. Noti e that M ,the number of fa tors in the fa torization, is onsidered as a parameter here. However,it shall be shown later in this work that M ≤ 2N + 1 holds.2.4.2 Clustering AlgorithmThe lustering algorithm des ribed in the following is based on the prin iples ofhierar hi al lustering [JMF99 and an be seen as a variant of the Ward algorithm

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 41[War63. The goal is to luster the set of sour es N into subsets Λc ⊆ N , su h that⋃

∀c∈Γ Λc = N and Λi ∩Λj = ∅ for all i 6= j with i, j ∈ Γ, where Γ = 1, 2, . . . , C isthe set of luster indi es and C = |Γ| is the number of lusters. The maximum lustersize S is assumed to be given and dened su h that |Λc| ≤ S for all c ∈ Γ.The lusters itself are onstru ted by a su essive merging pro ess. The algorithmstarts with a set of single-element lusters su h that Λ′s = s for all s ∈ Γ′, where

Γ′ = N = 1, 2, . . . , N is the initial set of luster indi es. In ea h of the followingsteps two of those lusters are sele ted and merged into a new luster. The lustersare sele ted using the KLD D(p(u)||p(u)) between the original PDF p(u) and theapproximated PDF p(u) =∏

∀s∈Γ′ p(uΛ′s) as an obje tive fun tion, where p(u) dire tlyresults from the urrent hoi e of lusters and D(p(u)||p(u)) is dened analog to (2.12).For ea h possible pair of lusters (Λ′

k, Λ′l) with k 6= l and k, l ∈ Γ′, the algorithmdetermines the urrent value of the obje tive fun tion to nd the pair (Λ′

k, Λ′l) leadingto the smallest KLD between original and approximated PDF. The indi es of thesele ted lusters (k, l) are then removed from the urrent set of luster indi es Γ′whereas the index of the newly reated luster r is added to it. This pro edure isrepeated until only a single luster remains and a history of all merging performedduring the dierent stages of the optimization pro edure is obtained.Using (2.13), it is possible to show that the overall KLD an be al ulated asfollows

D(p(u)||p(u)) = −1

2log2 |R|+

∑

∀s∈Γ′

∆D(Λ′s, ∅), (2.15)where ∆D(Λ′

s, ∅) is the KLD benet imposed by an arbitrary luster Λ′s. Sin e theobje tive fun tion has to be evaluated many times during the optimization pro ess,it is useful to express (2.15) in terms of intermediate results to redu e omputational omplexity. The dierential KLD benet reated by merging an arbitrary pair of lusters (Λ′

k, Λ′l) with k 6= l and k, l ∈ Γ′ into a single new one an be expressed asfollows∆D′(Λ′

k, Λ′l) = ∆D(Λ′

k ∪ Λ′l, ∅)−∆D(Λ′

k, ∅)−∆D(Λ′l, ∅), (2.16)whi h an be used to lo ally evaluate the impa t of the onsidered merging onto theoverall KLD given by (2.15). Assuming that t is the number of merging operationsperformed at a ertain stage of the pro edure, then the expression

D(p(u)||p(u)) = −1

2log2 |R|+

t∑

s=1

∆D′(Λ′k(s), Λ

′l(s)) an be used to evaluate the overall KLD in (2.15) based on the dierential KLDbenets in (2.16) only.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 42A detailed des ription of the entire pro edure an be found in Algorithm 2,where r labels the lusters in as ending order and h (a two-dimensional array) is usedto store a history of the merging performed during dierent stages of the lusteringpro edure. In Figure 2.3(a) the merging pro ess is illustrated for an exemplarys enario. A graphi al representation of the merging performed during dierent stagesof the optimization, the so- alled dendrogram [JMF99, is shown in Figure 2.3(b).Algorithm 2: KLD optimized lusteringInitialization1• start with one-element lusters2

Γ′ ← 1, ..., N3Λ′

s ← s for all s ∈ Γ′4t← 1; r ← N + t5 Main Loop6 repeat7• nd the pair of luster (Λ′

k, Λ′l) with k 6= l and k, l ∈ Γ′8 su h that |∆D′(Λ′

k, Λ′l)| is maximized9

• store intermediate results:10Λ′

r ← Λ′k ∪ Λ′

l11• delete original lusters from index list:12

Γ′ ← Γ′\k, l13• add new luster to index list:14

Γ′ ← Γ′ ∪ r15• save lustering history:16

h(t, 1)← k; h(t, 2)← l17• update internal variables:18

t← t + 1; r ← N + t19 until (|Γ′| = 1)20 Using the dendrogram derived before, the sour e lusters Λc with a maximum luster size of S an be onstru ted. We start at the root of the dendrogram, whi his basi ally a tree, and des end along its bran hes to lower hierar hi al levels. Whilemoving from one level to the next (lower) level, the dendrogram bran hes into twosubtrees. The number of leafs, i.e., the number of sour es onne ted to ea h subtreeare ounted; and if the number of leafs of one (or both) subtree(s) is smaller or equal toS, we ut out the orresponding subtree from the dendrogram. This pruning pro essis repeated until all leafs are removed from the dendrogram. When the pruning is

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 43

0 0.2 0.4 0.6 0.8 1 0

0.2

0.4

0.6

0.8

1 u8

u2u

1

u6

u4

u7

u9

u5

u3

Λ10

’=5,9

Λ11

’=4,7

Λ13

’=1,6Λ12

’=Λ10

’ ∪ Λ11

’=4,5,7,9

5 9 4 7 1 6 2 8 30

1

2

3

4

5

6

|∆ D

(Λr’,

0)|

in b

it

r=10

r=11

r=12

r=13

r=14

r=15

r=16

r=17

x xx

xx

(a) (b)Figure 2.3: Clustering example. Sour e-optimized lustering pro edure with N = 9uniformly distributed sensors pi king-up observations u1, ..., u9: (a) Merging performedduring the hierar hi al lustering pro edure for the rst four iteration steps leadingto resulting lusters Λ′r with indi es r = 10, · · · , 13. (b) Tree representationof the merging performed during dierent the stages of the optimization pro ess(dendrogram). The KLD benet |∆D(Λ′

r, ∅)| in [bit imposed by the lusters Λ′r isprovided for all iteration steps. The bran hes of the tree that are ut during thepruning pro ess with a maximum luster size of S = 4 are marked with a ross.After pruning, the sour e lusters Λ1 = 3, Λ2 = 8, Λ3 = 2, Λ4 = 1, 6 and

Λ5 = 4, 5, 7, 9 an be dened.nished, the subtrees are labeled su essively by the index c = 1, 2, . . . , C. The sour e lusters Λc, c ∈ Γ, an then be determined from the subtrees by assigning the variablesn ∈ N (asso iated with ea h of the subtree's leafs) to the orresponding luster. Theoverall KLD D(p(u)||p(u)) between the original PDF p(u) and the approximated PDFp(u) =

∏

∀c∈Γ p(uΛc) an then be al ulated based on the resulting lustersD(p(u)||p(u)) = −

1

2log2 |R|+

∑

∀c∈Γ

∆D(Λc, ∅), (2.17)where D(p(u)||p(u)) is dened analog to (2.12). In Figure 2.3(b) the pruning pro essis illustrated for the previous example.Be ause of the hierar hi al merging on ept based on lo al de isions, the pro-posed lustering algorithm is in general sub-optimal. However, the hierar hi al ap-proa h has the advantage that the resulting dendrogram an be used elegantly to

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 44 onstru t lusters with a bounded number of sour e variables S.5In Appendix A.7 it is shown that the omputational omplexity of sour e-optimized lustering (evaluated in a very pessimisti fashion) is of O(N5 log N) whi hmakes the overall pro edure feasible for medium to large values of N . Furthermore,it is easy to show that the number of lusters C with a maximum luster size of S isbounded a ording to C ≤ ⌊NS⌋, whi h shall be required in the next se tion.2.4.3 Sour e-Optimized Fa torizationIn the last se tion we have shown how to onstru t KLD-optimized lusters ttingour purposes. The se ond step towards our goal of obtaining a sour e fa torizationis to transdu e the derived lusters into a symmetri CCRE of the form p(u) =

∏Mm=1 fm(uSm

) mat hing the onditions in (2.11). This an be a hieved by linkingthe lusters Λc, c ∈ Γ, su essively together.The basi prin iple of the linking pro edure is as follows: After hoosing a spe i luster as starting point for the pro edure, sele t one of the un onne ted lusters (i.e.,a luster whi h is not yet onsidered in the sour e fa torization) and link it with thealready onne ted lusters (i.e., in orporate it into the sour e fa torization). Assumingthat luster r ∈ Γ was hosen as the starting point for the optimization, we an denea set of linked lusters Γ′ = r and a set of un onne ted lusters Γ′ = Γ\r. Atea h step of the pro edure a luster k ∈ Γ′ and a luster l ∈ Γ′ are sele ted. The indexl is added to the set of linked lusters, i.e. Γ′ = Γ′ ∪ l, and removed from the setun onne ted lusters, i.e., Γ′ = Γ′\l. This is repeated until all lusters are linked,i.e., |Γ′| = |Γ|.More spe i ally, two lusters k, l ∈ Γ are linked by hoosing a set of variablesPk ⊆ Λk and a set of variables Ql ⊆ Λl. These sets will form the basis of the fa torintrodu ed into the sour e fa torization. Sin e the omplexity of s alable de oding ishighly dependent on the number of variables within the single fa tors of the underlyingsour e fa torization (see Appendix A.6, or [BT06 for details), we introdu e the designparameters A and B su h that |Pk| ≤ A and |Ql| ≤ B for all k, l ∈ Γ.The sour e fa torization starts with a single fa tor p(uIr

) ontaining the variablesof the initially hosen luster r, i.e., Ir = Λr. While establishing a link between thetwo lusters k and l, the fa tors p(uQl|uPk

) and p(uIl|uQl

) are added to the sour efa torization where Il = Λl\Ql. As the lusters are linked, a sour e fa torization is onstru ted step-by-step where the running index d is used to index the added lusters.5With partitional lustering methods, e.g., see [JMF99, this would be an arguably di ult task.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 45The resulting sour e fa torization an then be written asp(u) = p(uΛl(d=1)

)︸ ︷︷ ︸(a) C∏

d=2

(

p(uQl(d)|uPk(d−1)

)︸ ︷︷ ︸(b) p(uIl(d)

|uQl(d))

︸ ︷︷ ︸( ) ) (2.18)where Λl(d=1) = Λr = Il(d=1) = Ir and all fa tors with Il(d) = ∅ for d = 2, · · · , Care dis arded. Noti e that, when onstru ted a ording to the aforementioned linkingpro edure, there exists a one-to-one orresponden e between the running index d andthe luster indi es c ∈ Γ. Moreover, it an easily be shown that (2.18) fulllls the riteria of CCREs by verifying the onditions in (2.11).After dis ussing how sour e fa torizations tting our purposes an be onstru ted,we are ready to show how to hoose the subsets Pk and Ql and in whi h order the lusters should be linked su h that the overall KLD D(p(u)||p(u)) dened analogto (2.12) is minimized.It is easy to show that the KLD of the sour e fa torization given in (2.18) anbe expressed asD(p(u)||p(u)) = −

1

2log2 |R|+ ∆D(Λl(d=1), ∅)

+C∑

d=2

(

∆D(Pk(d−1) ∪Ql(d),Pk(d−1)) + ∆D(Λl(d),Ql(d)))

.(2.19)The KLD benet imposed by the fa tors ( ) in (2.18) an be written∆D(Λl,Ql) =

1

2log2

|RΛl|

|RQl|

=1

2log2 |RΛl

| −1

2log2 |RQl

|

= ∆D(Λl, ∅)−∆D(Ql, ∅). (2.20)The KLD benet imposed by the fa tors (b) in (2.18) an be expressed as∆D(Pk ∪ Ql,Pk) = ∆D(Pk ∪ Ql, ∅)−∆D(Pk, ∅). (2.21)Considering the KLD benet in (2.20) and (2.21), we noti e that ∆D(Λl, ∅) in (2.20)already was onsidered during the luster optimization in Se tion 2.4 . Thus, we areable to dene the KLD benet of establishing a link based on the sets Pk and Ql as

∆D∗(Pk,Ql) = ∆D(Pk ∪Ql, ∅)−∆D(Pk, ∅)−∆D(Ql, ∅). (2.22)Using (2.17), the KLD of the sour e fa torization in (2.18) an be written asD(p(u)||p(u)) = D(p(u)||p(u)) +

C∑

d=2

∆D∗(Pk(d−1),Ql(d)), (2.23)

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 46de oupling the link optimization from the luster optimization.If an already linked luster k is to be onne ted to a luster l in a KLD-optimalway, then the sets Pk ⊆ Λk and Ql ⊆ Λl have to be hosen su h that the KLD benet∆D∗(Pk,Ql) a ording to (2.22) is maximized in magnitude. The set of all possiblesubsets P ′

k ⊆ Λk with |P ′k| = A is denoted as T (A, Λk) and the set of all possiblesubsets Q′

l ⊆ Λl with |Q′l| = B is denoted as T (B, Λl). Pk and Ql are thereforedened as

(Pk,Ql) = argmin(P ′

k,Q′

l): P ′

k∈T (A,Λk)

Q′l∈T (B,Λl)

∆D∗(P ′k,Q

′l) (2.24)and we dene link ost as

ck,l = ∆D∗(Pk,Ql). (2.25)Noti e that generally ck,l 6= cl,k.To determine how the lusters are to be linked (i.e., whi h lusters are to be linkedand in whi h dire tion), a graph an be onstru ted representing the KLD optimallinks between the lusters. The verti es of the graph are obtained by ontra ting ea h luster Λc, with c ∈ Γ, to a single vertex vc and dening the set of verti es asV = vc : c ∈ Γ.The set of all possible dire ted edges ek,l = (vk, vl) between the verti es vk and vl,

k, l ∈ Γ, is dened asE = ek,l = (vk, vl) : k, l ∈ Γ, k 6= l,where the ost ck,l of ea h edge ek,l in terms of KLD benet is given by (2.25). A fully onne ted graph G = (V, E) is thus obtained. Provided that the lusters are onsideredxed, the overall KLD of the sour e fa torization (2.18) an be optimized solely byoptimizing the luster links, please refer to (2.23), whi h are in turn represented bythe dire ted edges in G. The optimization problem therefore redu es to the Minimum( ost) Dire ted Spanning Tree (MDST) problem for whi h rst algorithms were foundby Chu and Liu [CL65 as well as by Edmonds [Edm67 to be generalized later byGeorgiadis [Geo03. After applying one of these algorithms to the fully onne tedgraph G, the MDST G′ = (V, E ′) with E ′ ⊆ E and its root vertex (i.e., the vertex

vr ∈ V whi h only has outgoing edges) an be found. The sour e fa torization (2.18) an then be onstru ted by moving along the edges of the obtained tree G′ (possiblyinspired by a Depth-First Sear h, e.g., see [AU97, p. 484), and linking the lusters orresponding to the visited verti es together. More spe i ally, the root vertex of G′ orresponds to the fa tor denoted as (a) in (2.18), the visited edges orrespond to the

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 47

0 0.2 0.4 0.6 0.8 1 0

0.2

0.4

0.6

0.8

1 u8

u2

Λ3

p(u2|u

1,u

6)

p(u8|u

1,u

6)

Λ2

u1

u6

Λ4

u7

u4

u3

u9

u5

p(u3|u

1,u

6)

p(u1,u

6|u

4,u

9)

p(u4,u

5,u

7,u

9)

Λ5

Λ1

Figure 2.4: Linking example. KLD optimized sour e fa torization obtained by linking lusters with A = B = 2. The fa tor graph represents the symmetri CCRE p(u) =

p(u4, u5, u7, u9) · p(u1, u6|u4, u9) · p(u2|u1, u6) · p(u8|u1, u6) · p(u3|u1, u6).fa tors denoted as (b) and the visited verti es orrespond to the fa tors denoted as ( ).Noti e that this tree-based linking approa h also onforms with the aforementionedlinking pro edure, whi h requires that links result only from already onne ted lusters,and thus guarantees a valid CCRE.Considering the previous example with lusters Λ1 = 3, Λ2 = 8, Λ3 = 2,Λ4 = 1, 6, Λ5 = 4, 5, 7, 9 and A = B = 2, we get the MDST G′ = (V, E ′) withV = v1, · · · , v9, root v5 and E ′ = (v5, v4), (v4, v2), (v4, v3), (v4, v1). Figure 2.4 showsthe orresponding sour e fa torization.Appendix A.8 dis usses the omplexity of the sour e-optimized linking pro edureand shows that the omputational omplexity grows exponentially with S assumingthat A = B = S

2, whi h makes the overall pro edure feasible for small luster sizes S.Noti e that in the last se tion it was shown that C ≤ ⌊N

S⌋ allowing us to representthe omplexity only based on the system parameters N and S. It is easy to seethat, be ause the linking pro edure basi ally onstru ts a tree between the lusters,the number of fa tors M in the fa torization (2.10) an be bounded a ording to

M ≤ 2N + 1. This also means that the number of fa tors with a maximum size of Sis at most 2N + 1, as used in Se tion 2.4.1 to analyze the omplexity of the s alablede oder.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 482.5 Results and Dis ussionTo underline the ee tiveness and e ien y of our low- omplexity oding and lus-tering strategies, we present numeri al performan e results for two s enarios withrandomly pla ed sensors and two instan es of the so- alled CEO Problem [VB97.2.5.1 Randomly Pla ed SensorsWe onsider a unit square with N = 100 uniformly distributed sensors. The sensormeasurements un are Gaussian distributed a ording to N (0, 1). As outlined inSe tion 2.2.1, we assume that sensor measurements u = (u1, u2, . . . , uN)T are jointlydistributed a ording to a multivariate Gaussian distribution N (0TN ,R), where the orrelation between a pair of sensors uk and ul de reases exponentially with thedistan e dk,l between them, su h that ρk,l = exp(−β · dk,l). Sin e the performan e ofour te hniques depend on the orrelation between the sensors, we onsider two dierents enarios, one with β = 0.5 (strongly orrelated sensor measurements) and one with

β = 2 (weakly orrelated measurements). All s alar quantizers at the en oders areLloyd-Max optimized to minimize the MSE in the sensor readings un using identi alresolution for quantization and identi al rates for data transmission, i.e., |In| = Land Rn = R for all n ∈ N , N = 1, 2, . . . , N, where L ≤ 16 was hosen. The lusters Λc ⊆ N indexed by c ∈ Γ are derived as des ribed in Se tion 2.4.2 where amaximum luster size of S = 4 was hosen (see Figure 2.5). The index assignmentsare then designed su essively for all lusters with |Λc| > 1 and c ∈ Γ by employingthe IR algorithm des ribed in Se tion 2.3 with Ψ = Ω = Λc. Sin e it is not possibleto onstru t index assignments for single-element lusters, we hose in this ase as alar quantizer (Lloyd-Max optimized as before) with de reased resolution and noindex assignments su h that Rn = R is still guaranteed for all en oders n ∈ N . Thesour e fa torization used for de oding is onstru ted as des ribed in Se tion 2.4.3assuming that A = B = 1 (see Figure 2.5). The de oder is based on the sum-produ talgorithm as des ribed in [BT06 where the required PMFs were obtained by MonteCarlo simulation using Lloyd-Max optimized quantizers with resolution Ln = L for alln ∈ N . To evaluate the performan e of the oding strategies, we measure the outputsignal-to-noise ratio (SNR) given bySNR = 10 · log10

(

E||U||2

E||U−U||2

)

[dB]approximated by the sample average over a N × 10000 sour e samples. The obtainedsimulation results are shown in Table 2.1 for strongly and weakly orrelated sour es. In

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 49

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

Figure 2.5: Simulation s enario. Graphi al representation of the KLD optimized sour efa torization for N = 100 uniformly distributed sensors with orrelation fa tor β = 0.5.The lusters with a maximum size of S = 4 (indi ated by ir les) were reated usingthe hierar hi al lustering method and linked together by hoosing A = B = 1.Table 2.1: Simulation results - Low- omplexity distributed sour e oding s enario withN = 100 and β = 0.5, 2

β = 0.5 β = 2

R [bit 1 2 3 4 1 2 3 4SNRDe [dB 4.44 9.46 14.61 20.32 4.45 9.32 14.65 20.27SNRIR[dB 11.07 14.86 18.29 N.A. 7.54 11.72 16.21 N.A.both s enarios, we onsider the performan e a hieved when using s alar quantizationalone at the en oder, i.e., where the performan e is mainly governed by the propertiesof the de oder (De ), and the performan e a hieved when s alar quantization with asubsequent index-reuse (IR) is used for en oding. Table entries labeled as N.A. (notavailable) indi ate that those instan es ould not be onsidered here due to their high omputational demand of the index-reuse optimization.6 Noti e that only the indexassignments yielding best possible performan e were hosen for the experiments (e.g.,a rate of R = 1 [bit may be obtained from quantizers of resolution L = 4, 8, 16).Our simulation results reveal that simple index assignment te hniques appliedto lo al lusters an a hieve signi ant performan e gains using our oding approa h,6This instan e would require a high-rate quantizer with a resolution larger than L = 16.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 50espe ially for low data rates and strongly orrelated sour es.2.5.2 The CEO ProblemIn the following, we show the appli ability of our te hniques to another relevant sensornetwork model: the quadrati Gaussian CEO Problem [BZV96. Let u0 be the outputof a ontinuous-valued Gaussian sour e U0. For all n ∈ N , N = 1, 2, . . . , N,let un denote noisy observations of u0 whi h are orrupted by additive noise, i.e.,un = u0 + nn. The noise samples are generated by Gaussian noise pro esses Nnstatisti ally independent over n. The observations un are en oded and transmittedby independently operating en oders indexed by n. The main task of the CEO isto estimate u0 based on the data obtained from the en oders. In [MB05 we derivedthe optimal de oding rule exploiting the spe ial properties of this problem setup andstudied a feasible de oder using a sour e approximation based on the fa torization

p(u0, u1, ..., uN) = p(u0)N∏

n=1

p(un|u0),whi h an be easily represented by a fa tor graph [KFL01. In the following, we onsider a s enario of N = 100 en oders. The sour e pro ess is Gaussian distributedN (s0, σ

20) with mean s0 = 0 and varian e σ2

0 = 1. The noise pro esses are Gaussiandistributed N (ln, λ2n) with mean ln = 0 and varian e λ2

n = λ for all n ∈ N whereλ = 0.1, 0.5 was hosen depending on the onsidered s enario. All s alar quantizersat the en oders are Lloyd-Max optimized to minimize the MSE in the sensor readingsun using identi al resolution for quantization and identi al rates for data transmission,i.e., |In| = L and Rn = R for all n ∈ N where L ≤ 16 was hosen. We use the s alablede oder as des ribed in Se tion 2.2 where the required PMFs were determined usingMonte Carlo simulation with resolution |I0| = 64 for the sour e u0 and |In| = L for theobservations un for all n ∈ N . Noti e that in ase of our highly symmetri s enario,with |In| = L, λ2

n = λ2 and ln = 0, the probabilities p(in|i0) an be onsidered identi alfor all n ∈ N . Therefore, the index assignments need to be designed only on e fora single, arbitrarily hosen luster Λ ⊆ N with |Λ| = S where S = 4 was hosen.After employing the IR algorithm des ribed in Se tion 2.3 with Ω = Λ and Ψ = 0,the resulting index assignments an be assigned repeatedly to all lusters within thesystem. To evaluate the performan e of our oding strategies, we measure the outputsignal-to-noise ratio given bySNR = 10 · log10

(

EU20

E(U0 − U0)2

)

[dB]

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 51Table 2.2: Simulation results - Low- omplexity CEO oding with N = 100 en oders,sour e varian e σ20=1, and noise-varian es λ2=0.1, 0.5.

σ20=1, λ2=0.1 σ2

0=1, λ2=0.5R [bit 1 2 3 4 1 2 3 4SNRR/D[dB 28.66 29.70 29.93 29.99 21.72 22.74 22.96 23.01SNRDe [dB 9.67 19.37 26.21 28.48 15.25 20.84 22.49 22.74SNRIR[dB 22.71 26.70 28.21 N.A. 18.76 21.56 N.B. N.A.approximated by the sample average over 10000 sour e samples (whi h requires thetransmission of N × 10000 odeword symbols). For dierent (symmetri ) en odertransmission rates, we ompare the performan e with the (sum) rate-distortion fun -tion, oered by [CZBW04, whi h presents an upper bound found to be tight for noisepro esses with identi al varian e. In Table 2.2 we present some results to underline theee tiveness of our approa h. The performan e of the system without index assign-ments (De ) and the performan e obtained by using index-reuse (IR) is ompared tothe theoreti ally possible value as given by the (sum) rate-distortion fun tion (R/D)a ording to [CZBW04. Table entries labeled as N.A. (not available) indi ate thatthose instan e ould not be onsidered here due to their high omputational demandof the index-reuse optimization.7 Table entries labeled as N.B. (no benet) indi atethat in this ase index-reuse does not outperform standard quantization. Again, noti ethat only the index assignments yielding the best possible performan e were hosenfor the experiments.The numeri al results reveal that our index-reuse approa h leads in many asesto signi ant performan e improvements over standard quantization. It might happen,however, that our index assignments are not able to outperform s alar quantization.Whether this is true or not depends on several fa tors: (a) the quantizer resolution

L, (b) the number of output bits R, and ( ) the orrelation properties of the sour esdetermined by σ20 and λ. In ases where the sour es are weakly orrelated, e.g., forlarge values of λ, it be omes harder (or even impossible) to nd index assignmentsthat oer good rate-distortion trade-os. In parti ular this might be true in our asedue to the simpli ity of the onsidered oding on ept and the sub-optimality of theproposed index-reuse algorithm whose performan e is highly dependent on the hoi eof L and R.7This instan e would require a high-rate quantizer with a resolution larger than L = 16.

CHAPTER 2. SOURCE-OPTIMIZED CLUSTERING 522.6 Con lusionsWe presented a s alable solution for distributed sour e oding in large-s ale sensornetworks. Our methods rely on the ombination of a simple en oding stage (a s alarquantizer and an index assignment stage) and a sour e-optimized lustering algorithm.with standard s alar quantization. Despite the simpli ity of the proposed te hniques,our results show signi ant performan e gains as ompared to standard s alar quan-tization alone (i.e., without exploiting the sour e orrelation).

Chapter 3Diophantine Distributed Sour eCodingIn the previous hapter we showed that distributed sour e odes based on sour e-optimized index assignmentsdespite being memory-less and having a low opera-tional omplexityare ee tive in exploiting the sour e orrelation for an e ientdata transmission. In parti ular, we proposed an heuristi optimization algorithm thatsu essively onstru ts distortion-optimized index assignments based on (suboptimal)lo al de isions within ea h iteration step. In order to provide an alternative solutionto the ode design problema solution that is not based on lo al de isionswenow propose a method for the stru tured ode design that systemati ally exploitssymmetry properties ommon to many relevant sour e models. Using basi tools fromnumber theory, spe i ally from Diophantine analysis, we formulate a onstru tiveframework for the design of sour e-optimized index assignments under a zero-error riterion, whi h are then applied to s enarios that onsider the error probability orthe mean squared error distortion as the primary gure of merit. Besides a hieving agood performan e in the memory-less ase, the proposed index assignments also serveas a basis for more powerful trellis odes, allowing a exible trade-o between the odeperforman e and the in urred omputational omplexity.3.1 Introdu tion3.1.1 Stru tured Distributed Sour e CodingIn distributed sour e oding s enarios, where data from orrelated information sour eshas to be ompressed without intera tion or ommuni ation among en oders, stru -53

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 54

Figure 3.1: Considered system setup.tured odes apable of exploiting the sour e properties are key towards a hieving goodperforman e, in both, short and long blo k length s enarios, while keeping the de oding omplexity omputationally tra table for ases with two or more sour es.We onsider a s enario ( ompare Figure 3.1) with N orrelated sour es, U1,

U2, . . . , UN , generating the output sequen es u1(t)∞t=0 , u2(t)

∞t=0 , . . . , uN(t)∞t=0,independently and identi ally distributed (i.i.d.) a ording to the joint distribution

p(u1(t), u2(t), . . . , uN(t)) = p(u1, u2, . . . , uN) onstant over t. Ea h sour e output ispro essed by an independently operating en oder, whi h does not have any knowledgeabout the data observed, pro essed, or sent by any of the other en oders. After error-free transmission, the de oder uses the re eived data from all en oders jointly togetherwith its knowledge about the joint sour e statisti s p(u1, u2, . . . , uN), to produ e the es-timates of the original sour e symbol sequen es u1(t)∞t=0 , u2(t)

∞t=0 , . . . , uN(t)∞t=0.The goal is to en ode, transmit and de ode the sour e symbol sequen es subje t tothe primary performan e riterion, whi h is the error probability (whenever dis rete-valued sour es are onsidered) or the mean squared error distortion (in s enarios with ontinuous-valued sour es).Inspired by the seminal work of Slepian and Wolf [SW73, whi h hara terizes thefundamental limits of separate en oding of orrelated sour es, several papers have on-tributed with pra ti al oding solutions for multiterminal sour e oding problems (e.g.,see [XLC04 and referen es therein). In most ases those oding approa hes eitherexploit systemati ally the sour e properties, whi h is usually only possible for shortblo k lengths (e.g., see [FG87, Wit76), or build on powerful hannel odes that operateee tively at very long blo k lengths (e.g., see [PR03, GF01, BM01, LXG02). Theformer approa h takes into a ount the joint properties of the sour es for an opti-mized ode design that is potentially apable of a hieving a good system performan e;

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 55

Figure 3.2: Considered sour e models. (a) Bivariate Gaussian distribution p(u1, u2) withstrong orrelation between the ( ontinuous-valued) sour e outpus u1 and u2. (b) Probabilitymass fun tion p(i1, i2) of strongly orrelated, dis rete-valued sour e symbols i1 and i2. Inthis parti ular example i1 and i2 are obtained via (uniform) quantization from u1 and u2,respe tively. The symmetry axis δ : i2 − i1 = 0 is used to dene a onned area ontainingthe most probable sour e outputs, as indi ated by the shaded area.whereas the latter relies on the error orre ting apabilities of powerful hannel odesto a hieve a good performan e after joint de oding. This however requires that (atleast) part of the original sour e orrelation arries over into the binary domain inwhi h the hannel odes operate. In the binary domain the orrelation among sour esis often modeled by a (virtual) binary symmetri hannel des ribing the statisti aldependen ies between the (individual) sour e bits [GF01, BM01, LGS04, LXG02,or it is hara terized in terms of the maximum allowed Hamming distan e betweenthe binary representations of the sour e symbols [PR99, PR03. Clearly, this is notne essarily the best way to a ount for the sour e orrelations, e.g., whenever theinformation symbols result from natural pro esses; but the alternative of performingthe oding operations in the sour e symbol domain while exploiting their joint statisti sproves to be di ult, espe ially for long blo k length s enarios.In order to provide a sour e-optimized design that operates at symbol level, werestri t our attention to a relevant lass of sour e models. Re all that the sour esU1, U2, . . . , UN generate the i.i.d. outputs u1, u2, . . . , uN . We hara terize those sour emodels by the properties (the shape) of the joint probability distributionp(u1, u2, . . . , uN) for the sour e outputs u1, u2, . . . , uN . In parti ular, we assume thatthe sour e outputs u1, u2, . . . , uN lo ated within a onned area lose to some symmetry

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 56

Figure 3.3: Additive sour e model similar as onsidered in the CEO problem [BZV96. Thesour e samples un are obtained by adding the independent noise vn to the sample u0 of someunderlying sour e pro ess, su h that un = u0 + vn, n = 1, 2, . . . , N . The distributions p(vn)and p(u0) are arbitrary, however the support of p(vn) is small as ompared the support ofp(u0) to ensure that the most probable sour e outputs are ollo ated in the lose proximityof the symmetry axis δ, n = 1, 2, . . . , N .axis δ have a high probability p(u1, u2, . . . , uN) whereas the sour e outputs outsidethis area have a negligible probability (e.g., see Figure 3.2). The symmetry axis δitself orresponds to the all-equal sour e outputs: u1 = u2 = . . . = uN . Althoughthis assumption might appear to be a severe restri tion, a loser inspe tion revealsthat many important sour e models, espe ially those relevant for distributed sensings enarios, meet this riterion in a straightforward manner or an be modeled as su h.Examples in lude:• Sour es U1, U2, . . . , UN following, exa tly or approximately, a multivariate Gaus-sian distribution (e.g., see Figure 3.2).• Sour e pro esses U1, U2, . . . , UN that result from (or an be modeled by) a om-mon sour e pro ess U0 orrupted by independent, additive noise pro esses V1,

V2, . . . , VN (e.g., see Figure 3.3).• Sour es U1 and U2 that only (or mostly) generate outputs u1 and u2 with asquared Eu lidean distan e ||u1 − u2||

2 = (u1 − u2)2 smaller or equal to somethreshold.

• Dis rete valued sour e pro esses I1 and I2 with outputs i1 and i2 whose jointprobability p(i1, i2) an be expressed in terms of the typewriter hannel's tran-sition probabilities p(i2|i1) or p(i1|i2) (e.g., see Figure 3.4).Clearly sour e symbols resulting from natural pro esses might fall into one of the above

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 57 ategories but also "arti ially" generated symbols that are obtained via some datapro essing or transmission stage (e.g., see Figure 3.4).3.1.2 Key Idea and ContributionsThis work aims at providing a distributed sour e oding s heme apable of exploitingthe properties for the relevant lass of sour e models presented before. We will showthat a simple lass of index assignments, designed and operating in a purely memory-less fashion, an be used to do so. We onsider a system of N en oders: Ea h en odern = 1, 2, . . . , N has a dis rete-valued sour e input in ∈ 0, 1, . . . , L − 1 of alphabetsize L. The proposed index assignments are fully hara terized by the hosen integersKn ≤ L, n = 1, 2, . . . , N , whi h not only dene the sizes of the output alphabets but,more importantly, also dene the stru ture of the index assignments themselves. Inparti ular, all sour e indi es in ∈ 0, 1, . . . , L−1 satisfying the equation in = anKn+snfor some arbitrary integer an are mapped onto the same odeword; the odeword itselfis sele ted by hoosing sn ∈ 0, 1, . . . , Kn − 1. Seen from a dierent perspe tive,this means that Kn not only denes the number of odewords but also identies thesour e symbols that are mapped onto the same odeword. At this point it is worthpointing out that the integers Kn are expli itly allowed to be hosen dierently forthe individual en oders n whi h in many ases a tually turns out to be ne essary inorder to meet the requirements imposed by the given sour e model (e.g., ompare theexample in Figure 3.5).The main idea of the proposed ode design is to exploit the joint properties ofthe integers K1, K2, . . . , KN to identify odes suitable for the onsidered sour e modelassuming a zero-error de oding riterion, similar to the one onsidered, e.g., in [Sha56or [Wit76. In parti ular, this shall be a hieved by des ribing the oding systemthrough a set of Diophantine equalities1 of the form

a1K1 + s1 = a2K2 + s2 = . . . = aNKN + sN , (3.1)whose solutions an be hara terized using basi on epts from number theory. As a onsequen e, we will be able to identify a set of parameters, su h as the lowest ommonmultiple and the greatest ommon divisor of K1, K2, . . . , KN , that are well-suited forthe ode hara terization and eventually its design. In parti ular, we will be able toformulate a onstru tive design algorithm that uses the properties of numbers, e.g.,the prime fa tors of L, to identify good odes for a given sour e distribution, providedthat they exist.1Named after the Greek mathemati ian Diophantus of Alexandria who worked on problems on erning integer solutions to algebrai equations.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 58

Figure 3.4: Typewriter sour e model. Chara teristi graph showing the statisti aldependen ies between two dis rete-valued sour e symbols i1 and i2. The symbols i1 andi2 are onne ted by a line, if their joint probability p(i1, i2) = p(i2|i1)p(i1) = p(i1|i2)p(i2)is non-zero. In this parti ular ase the hara teristi graph strongly resembles the well-known typewriter hannel [CT91 with transition probabilities p(i2|i1) (or p(i1|i2)). Thelines between i1 = 0 and i2 = 7 as well as between i1 = 7 and i2 = 0 indi ate that the sour esymbols are allowed to wrap around their boundaries.Considering the resulting odes as a basis for long blo k length odes, it be omespossible to formulate a distributed sour e oding s heme that exploits the sour esymmetries at the en oder and the de oder, in short as well as long blo k lengths enarios to eventually improve the overall system performan e quantied by the errorprobability or the mean squared error distortion while lowering the resulting de oding omplexity. In parti ular, this shall be a hieved as follows.• System and Code Analysis: After dis ussing related work in Se ion 3.2 andproviding a pre ise problem formulation in Se tion 3.3, we show in Se tion 3.4how the Diophantine oding system an be analyzed using basi tools fromnumber theory.• Code Design for the Memory-less Case: Using insights from the Diophan-tine ode analysis, we formulate in Se tion 3.5 a systemati design algorithmbased on the fundamental theorem of arithmeti .• Coding without Memory: In Se tion 3.6 we show how to integrate the Dio-phantine index assignments into our system and, assuming a s alar quantizationapproa h, show how de oding an be performed and optimized for large numbersof en oders.• Trellis-based Binning: Using the Diophantine index assignments derived be-fore, we onstru t sour e-optimized trellis odes in Se tion 3.7 and show how thede oding omplexity an be kept manageable for a small numbers of en oders.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 59

Figure 3.5: Diophantine ode omparison for N = 2: (a) Diophantine ode with K1 =

K2 = 8. (b) Code with K1 = 9 and K2 = 6. In ase (a) two index ve tors with highprobability (as indi ated by the shaded area) are mapped onto the same odeword ve tor.This ambiguity makes the de oding de ision argueably 'hard' whereas in ase (b) de odingis quite straightforward. The ode onguration in (b) has the additional advantage ofrequiring a smaller total number of odewords K1K2 = 54, as ompared to K1K2 = 64 in onguration (a).• Quantization with Memory: We elaborate in Se tion 3.8 on how onventionaltrellis oded quantization an be integrated into our system setup to improve theoverall performan e whenever ontinuous-valued sour es are onsidered.• Performan e Evaluation and Dis ussion: After a detailed numeri al per-forman e evaluation in Se tion 3.9, we on lude with some nal notes in Se -tion 3.10.We believe that the proposed design on ept, and the resulting odes, provide apowerful alternative to existing distributed sour e oding solutions for an important lass of relevant sour e models.3.2 Related WorkConsidering ontinuous-valued sour es subje t to a mean squared error distortion rite-ria and short blo k lengths, Rebollo-Monedero et al. [RMZG03, Flynn and Gray [FG87,and Cardinal and Van Ass he [CVA02 proposed dierent quantizer optimizationalgorithms for the ase of two orrelated sour es. For dis rete sour es and short blo k

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 60lengths, the lossless (zero-error) ase2 was rst addressed for the related problemof sour e oding with side information available only at the de oder by Witsen-hausen [Wit76, where also a relevant onne tion to a bipartite graph- oloring problemwas also established. Similar results were obtained later for the general Slepian-Wolfsetting [SW73 by Al Jabri and Al-Issa [JAI97 as well as Yan and Berger [YB00.The near loss-less ase together with a pra ti al ode design was onsidered by Zhaoand Eros in [ZE03. Common to all of those stru tured oding approa hes is notonly that they are tuned to the urrent sour e settings but also that they are onlyfeasible for a small number of sour es (usually two) due to their inherent design omplexity. Furthermore, they usually an not be extended dire tly to larger blo klengths without making them intra table (in terms of their design omplexity or theirde oding omplexity or both).Using ideas from hannel oding, while assuming a Wyner-Ziv problem set-ting [WZ76, Pradhan and Ram handran [PR99, PR03 proposed a onstru tive frame-work for the design of distributed sour e odes alled distributed sour e oding us-ing syndromes (DISCUS). In parti ular, they onsidered the usage of hannel odesyndromes for binning purposes and showed how the proposed trellis odes an bede oded e iently using the Viterbi algorithm. Gar ia-Frias and Zhao [GF01 pro-posed an en oding on ept based on bit-pun turing that relies on the error- orre ting apabilities of highly evolved turbo- odes at the de oder to re over the pun turedbits. Using a similar approa h, based on turbo- odes and bit pun turing, Lajnef etal. onsidered the ase of three sour es in [LGS04. Baj sy and Mitran [BM01 alsoproposed the usage turbo- odes for distributed sour e oding, however, in ontrast tothe previous approa hes, ompression here was not a hieved by bit-pun turing but thespe ial design of the en oder trellises. Low density parity he k (LDPC) odes, as analternative to turbo odes, were onsidered by Liveris at al. in [LXG02. A ommondraw-ba k of these long-blo k length approa hes is that the de oder, whi h basi allyassures a good system performan e, is usually very omplex and does not s ale wellwith the number of en oders. However, those de oders generally do not take intoa ount the sour e properties (they only use the given transmission rates), whi h inprin ipal ould be exploited to lower the de oding omplexity.Our work diers from those ontributions in that we onsider a stru tured odedesign that systemati ally exploits the joint properties of the sour es in, both, short2Noti e that the notion of an asymptoti ally small error probability as, e.g., onsidered for thestandard the Slepian-Wolf s enario [SW73 does not apply well here. Instead a lossless s enario, losely orresponding to Shannon's notion of data transmission with zero-error [Sha56 is onsideredhere.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 61as well as long blo k length s enarios. Furthermore, at its most fundamental ore, ourapproa h diers from previous ones in that we use the number-theoreti al propertiesof the sour e parameters for the a tual ode design.3.3 Preliminaries and Problem StatementWe shall follow the same notational onventions as introdu ed in Se tion 2.2 withthe following extensions: The length−N all-one ( olumn) ve tor is denoted as 1TN =

(1, 1, . . . , 1)T . Dierent members of the same set are sometimes, whenever statedexpli itly, distinguished by the usage of a supers ript (instead of the usage of dierentsymbols) in order to highlight their aliation with the set, e.g., if there is a setI then i, i(0), i(1) and i(2) are all potential members of I. Furthermore, we follow the onvention that variables indexed by a set denote a set of variables, e.g., ifN = 1, 2, 3then uN = u1, u2, u3, and use the same on ept to dene variable ve tors, e.g.,uN = (u1, u2, u3)

T . The same on ept shall also be used to dene the rossprodu t ofsets, e.g., UN = U1 × U2 × U3.3.3.1 System SetupWe onsider a system setup ( ompare Figure 3.1) with N orrelated sour es, U1,

U2, . . . , UN , generating the output sequen es u1(t)∞t=0 , u2(t)

∞t=0 , . . . , uN(t)∞t=0,independently and identi ally distributed (i.i.d.) a ording to the joint probability

p(u1(t), u2(t), . . . , uN(t)) = p(u1, u2, . . . , uN) onstant over t. Ea h of those sour es ispro essed by an independently operating en oder, whi h does not have any knowledgeabout the data observed, pro essed, or sent by any of the other en oders.The en oders operate in two main stages: the quantization stage that transdu esthe ontinuous-valued sour e symbols un(t) ∈ R into a the dis rete domain (whenever ontinuous-valued sour es are onsidered), followed by the binning stage that maps thequantizer outputs onto odewords to be sent to the de oder. The quantizer ( ompareBlo k Q in Figure 3.1) uses the sour e sequen e un(t + θ)Θ−1θ=0 ∈ R

Θ to produ e thedis rete-valued quantization index in(t) ∈ I = 0, 1, . . . , L at ea h time instant t(assuming onvolution oding). The obtained quantization index in(t) ∈ I is thenmapped onto the odeword by the binning stage ( ompare Blo k Bn in Figure 3.1)produ ing the output xn(t) ∈ Xn at ea h time instant t (again, assuming onvolution oding). In this work, we spe i ally onsider the ase where |Xn| ≤ |I| to a hievedata- ompression, i.e., we generally have less odewords xn ∈ Xn than quantization

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 62indi es in ∈ I.3 The transmission rate for ommuni ating the odeword xn ∈ Xnto the de oder is dened as Rn = log2(|Xn|) [bit. The sum-rate is dened as RΣ =∑N

n=1 Rn [bit.After error-free transmission the de oder ( ompare Figure 3.1) uses the sequen ex(t + ϑ)T−1

ϑ=0 ∈ RT×N of re eived odeword ve tors x(t + ϑ) = (x1(t + ϑ), x2(t +

ϑ), . . . , xN(t + ϑ))T ∈ RN together with its knowledge about the joint sour e statis-ti s p(u1, u2, . . . , uN) to produ e the estimates (u1(t), u2(t), . . . , uN(t))T ∈ R

N of theoriginal sour e symbols (u1(t), u2(t), . . . , uN(t))T ∈ RN at ea h time instant t.The estimates (u1(t), u2(t), . . . , uN(t))T ∈ R

N are hosen subje t to the primarygure of merit. In s enarios with dis rete-valued sour es (i.e., whenever no quan-tization stage is involved) we onsider the error probability as the primary gureof merit, in whi h ase de oders based on the maximum a-posteriori (MAP) rulebe ome optimal [Poo94; this will be further addressed in Se tion 3.6.3. For thegeneral ase of ontinuous-valued sour es we onsider the mean squared error (MSE)distortion E||U − U||2, where U = (U1, U2, . . . , UN)T and U = (U1, U2, . . . , UN)Tare random variable ve tors, representing the re onstru ted and the original sour esymbols, respe tively. The optimal de oder in this ase is given by the onditionalmean estimator [Poo94. However, alternatively, we will also use a de oder based onMAP de oding ( ompare Blo k Φ in Figure 3.1) followed by an inverse quantizationstage ( ompare Blo k Q−1 in Figure 3.1); this will be further addressed in Se tion 3.6.3.For the ode design itself, however, we onsider a s alar (memory-less) s enario and azero-error riterion, as shown later in this se tion.3.3.2 Sour e and Correlation ModelGiven our system-setup with N orrelated sour es, U1, U2, . . . , UN , generating the i.i.d.outputs u1, u2, . . . , uN olle ted in the ve tor u = (u1, u2, . . . , uN)T , we assume that theprobability p(u1, u2, . . . , uN) = p(u) is known (exa tly, or by some approximation) forall u ∈ RN . For design purposes, we onsider dis rete representations I1, I2, . . . , IN ofthe sour es U1, U2, . . . , UN (e.g., as obtained by s alar quantization) with realizations

i1, i2, . . . , iN olle ted in the ve tor i = (i1, i2, . . . , iN )T . The realizations in, n =

1, 2, . . . , N , are hosen from the dis rete-valued alphabet I = 0, 1, . . . , L− 1 of sizeL and i ∈ I = IN . We furthermore assume that the probability p(i1, i2, . . . , iN) = p(i)is known for all i ∈ I.As pointed out in Se tion 3.1, we fo us on probability distributions where the3Without loss of generality, we assume symmetri quantizer setups for all en oders. The binningstages and the ardinality of the odeword alphabets |Xn|, on the other hand, are expli itly allowedto be asymmetri over n = 1, 2, . . . , N .

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 63sour e ve tors ( ontinuous or dis rete) with a small distan e to some symmetry axishave a high probability, whereas sour e ve tors that are farther away have a negligibleprobability ( ompare Figure 3.6). The distan e between a ve tor of sour e symbolsu ∈ R

N and a referen e ve tor u(0) ∈ RN is quantied by

d(u,u(0)) =

√√√√

N∑

n=1

(d1(un, u(0)n ))2. (3.2)By hoosing d1(un, u

(0)n ) = |un − u

(0)n |, n = 1, 2, . . . , N , this distan e orresponds tothe Eu lidean distan e. Regarding the dis rete ase, where the sour e symbol ve tor

i or the referen e symbol ve tor i(0) or both might be restri ted to positions in ZN ,we onsider i and i(0) as positions in Eu lidean spa e and use the distan e d(i, i(0))in (3.2) hoosing d1(in, i

(0)n ) = |in − i

(0)n |, n = 1, 2, . . . , N .It worth pointing out that sometimes it might be ome ne essary, or advanta-geous, to in lude the ase where the sour e symbols are allowed to wrap around theirboundaries (also ompare Figure 3.4) very similar to a line segment that an bendinto a ir le, or the sides of re tangle that an be onne ted to form a torus, and so on e.g., onsider the ase of angle measurements, where an angle of 2π [rad orrespondsto an angle of 0 [rad. In this ase the distan e between the sour e ve tors i and i(0) an be al ulated using (3.2) and hoosing d1(in, i

(0)n ) = min|in− i

(0)n |, L− |in− i

(0)n |,

n = 1, 2, . . . , N .The onsidered symmetry axis (see Figure 3.6 (a)), hen eforth alled the maindiagonal, is the lineδ : u = 0T

N + t · 1TN , (3.3)with referen e point 0T

N = (0, . . . , 0)T and dire tion ve tor 1TN = (1, . . . , 1)T , t ∈ R.We dene the minimum distan e between an arbitrary point u ∈ R

N and themain diagonal δ (see Figure 3.6 (a)), denoted as the diagonal distan e, as followsd(u, δ) = min

u(0)∈RN :u(0)∈δd(u,u(0)). (3.4)The diagonal distan e d(i, δ) for i ∈ I is dened a ordingly, repla ing d(u,u(0)) by

d(i,u(0)) in (3.4).The probability distributions onsidered in this work are hara terized by thefa t that the highest probable sour e symbol ve tors u ∈ R, for the ontinuous ase,and i ∈ I, for the dis rete ase, are lo ated in a onned area around the main diagonalδ. In order to dene this area, we introdu e the non-negative threshold r ∈ R, also alled the radius. For the dis rete ase, whi h is of parti ular interest for the odedesign presented later, we dene the set of all index ve tors i ∈ I with a diagonal

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 64

Figure 3.6: Geometri denitions. Example for N = 2. (a) Main diagonal δ and diagonaldistan e d(u, δ) for sour e ve tor u = (u1, u2)T . (b) Volume V(r) of radius r.distan e d(i, δ) smaller or equal to r, resulting in

V(r) = i ∈ I : d(i, δ) ≤ r, (3.5)hen eforth denoted as the volume of radius r (see Figure 3.6 (b)). We say that thesour e's probability mass is ontained in the volume V(r) of radius r, if∑i∈V(r) p(i) ≤

1− ε for a given pre ision ε.3.3.3 Diophantine Index AssignmentsWe introdu e a spe ial lass of index assignments, alled the Diophantine index as-signments, that are apable of exploiting the symmetries in our (memory-less) sour emodel. Those index assignments will serve as a basis for the ode design in, both,short as well as long blo k length s enarios.Let in ∈ I, I = 0, 1, . . . , L − 1, be the output of a dis rete-valued sour e In,the so- alled sour e index. Let wn ∈ Wn, Wn = 0, 1, . . . , Kn − 1, be the output ofthe index assignments, the so- alled odeword.An index assignment bn : I → Wn des ribes the mapping from the sour eindi es in ∈ I onto the odewords wn ∈ Wn, su h that wn = bn(in).4A Diophantine index assignment is dened as follows: All sour e indi es in ∈ Isatisfying the equation in = anKn + sn for some integer an = 0, 1, . . . are mapped ontothe same odeword wn ∈ Wn; the odeword wn ∈ Wn itself is sele ted by hoosing sn ∈

0, 1, . . . , Kn. In parti ular, we assume that wn = sn for sn = 0, 1, . . . , Kn − 1. The4At this point it is worth pointing out that in the memory-less ase and the most straightforward onguration of Xn =Wn, the index assignments an be dire tly used for en oding assigning xn(t) =

wn(t), n = 1, 2, . . . , N .

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 65Diophantine index assignment bn an then be des ribed by the modulo−Kn operation,su h that wn ≡ in (mod Kn).We dene the inverse image for a spe i odeword wn ∈ Wn, I−1(wn) = in ∈

I : bn(in) = wn, as the set of all sour e indi es in ∈ I mapped onto it.It is worth pointing out that, given the alphabet size Kn, the minimum distan ed(i

(1)n , i

(2)n ) between any pair of indi es i(1)n , i

(2)n ∈ I−1(wn), i

(1)n 6= i

(2)n , wn ∈ Wn, isalways greater or equal to Kn.5The key towards exploiting the sour e model's symmetry stru ture lies in the( olle tive) hoi e of the index assignments bn for all en oders n = 1, 2, . . . , N .Let i = (i1, i2, . . . , iN )T ∈ I, I = IN , be a ve tor of sour e indi es. Let w =

(w1, w2, . . . , wN)T ∈W, W =∏N

n=1Wn, be a ve tor of odewords.We dene the inverse image for a spe i odeword ve tor w ∈W ,I−1(w) = i ∈ I : bn(in) = wn, n = 1, 2, . . . , N, (3.6)as the set of all index ve tors i ∈ I that are mapped onto it.Considering the index ve tors i ∈ I, it is easy to verify that the minimumdistan e d(i(1), i(2)) between any pair of index ve tors i(1) 6= i(2) ∈ I−1(w) is alwaysgreater or equal to Kmin = minn=1,2,. . . ,NKn. This means that the (joint) distan eproperties of the index assignments are governed by the smallest odeword alphabetsize Kn. This in turn might lead to the (wrong) on lusion that the minimum alphabetsize should be as large as possible to a hieve a good ode performan e. However, thisis not true for all ases, parti ularly when the memory-less ase is onsidered ( ompareFigure 3.5 (a), with Kmin = 8, and Figure 3.5 (b), with Kmin = 6). This motivates asour e-aware design riterion, as des ribed below.3.3.4 De odability and AdmissibilityAlthough the system performan e is quantied using either the error probability, or themean squared error distortion as performan e riterion, we will employ the so- alledde odability riterion for the index assignment design.Let A ⊆ I be an arbitrary subset of the index ve tors I in whi h we areinterested in; all it the admissible set. We dene the de oded set I(w,A) for aspe i odeword ve tor w ∈W as the set of all index ve tors i ∈ I that are in theadmissible set A as well as the inverse image I−1(w) (see Figure 3.7), i.e.,

I(w,A) = I−1(w) ∩ A. (3.7)5This is very similar to Ungerboe k's set partitions [Ung82 used in hannel oding/modulationto maximize the minimum (Eu lidean) distan e between hannel symbols.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 66

Figure 3.7: Example. The index ve tor i(0) in the admissible set A is en oded ontothe odeword ve tor w(0). Given the odeword ve tor w(0), the index ve tor i(0) an bede oded orre tly by identifying the inverse image I−1(w(0)), and onstru ting the de odedset I(w(0),A).We say that a ertain index ve tor i ∈ I is de odable, if it an be re overed from theresulting odeword ve tor w ∈W with zero-error. We say that a set of index ve torsP ⊆ A is de odable, if all i ∈ P are de odable. In ases where the entire admissibleset A is de odable, the de odability riterion is met, and the design goal is a hieved.Noti e that this denition of de odability follows the notion of oding with zero-error introdu ed by Shannon for a hannel oding problem [Sha56 as well as the notionof oding with nite error, e.g., as onsidered in [ZE03 for a distributed sour e odings enario.The basi idea is as follows: The index ve tor i ∈ A, en oded onto the odewordve tor w, an be de oded with zero-error, if the de oded set I(w,A) has only onemember, namely the original index ve tor i itself. In this ase, using Shannon'sterminology, the index ve tor i is not ' onfuseable' with any other index ve tor inI(w,A) and therefore an be re overed with zero-error. This provides a su ient ondition for de odability. On the other hand, if the de oded set is either empty, orhas more than one member, de oding errors ne essarily o ur.For the Diophantine index assignment design, we will onsider the de odability riterion together with the sour e properties. The onguration of the Diophantineindex assignments is fully spe ied by the tuple (L, K1, K2, . . . , KN). The sour e is hara terized by the volume V(r) of radius r. Then, when evaluating the de od-ability, will onsider the volume V(r) as admissible set A. We say that a tuple(L, K1, K2, . . . , KN) is admissible for a given radius r, if V(r) is de odable. In this ase, we say that the tuple (L, K1, K2, . . . , KN , r) is admissible.66It is worth pointing out that the stated problem ould also be formulated in terms of a graph- oloring problem, e.g., similar as in [ZE03 where a s enario with two en oders is onsidered andthe design problem is formulated in terms of a bipartite graph- oloring problem on the sour e's

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 673.3.5 Problem StatementThe problem an then be formulated as follows: Given a sour e satisfying aboverequirements, identify distributed sour e odes that a hieve a pres ribed trade-obetween the system performan e (e.g., measured by the error probability or by themean squared error) and the sum-rate RΣ.In order to a hieve this, we onsider the design of Diophantine index assignments(L, K1, K2, . . . , KN) under the de odability riterion using the volume V(r) of radius ras admissible set. The prin ipal goal is to formulate algorithms that e iently identifyadmissible ode tuples (L, K1, K2, . . . , KN , r). The se ondary goal is to show how theobtained index assignments an be used for the design of distributed sour e odes inshort as well as long blo k length s enarios.3.4 Diophantine Code Analysis and CombinatorialDesign for the S alar CaseExploiting the hara teristi properties of the Diophantine index assignments, we shallnow provide the mathemati al means for the ode analysis.3.4.1 Preliminaries3.4.1.1 De odabilityThe following Lemma provides us with a ne essary and su ient ondition for de od-ability assuming the usage of surje tive mapping fun tions, as our Diophantine indexassignments:Lemma 3.4.1. (De odability) The admissible set A ⊆ I is de odable, if and only ifall index tuples i ∈ A are en oded on dierent odeword tuples w.Proof. Please refer to Appendix B.1 for details.Lemma 3.4.1 will prove very useful for the ode design when onsidering theelements in the admissible set A only. In parti ular, we will onsider the volume V(r)of radius r as admissible set, whi h we now abstra t further.3.4.1.2 Sour e SegmentationConsidering the volume V(r) of radius r as the admissible set, the goal is to ensure that hara teristi graph (very similar to the graph shown in Figure 3.4).

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 68

Figure 3.8: Example for N = 2. Diagonal segment D(c1), referen e point r(c1) and segmentlength λ(c1) for c1 = a, b.V(r) is de odable. It is helpful to onsider the main diagonal δ as a referen e withinthe volume V(r) and partition the index ve tor spa e I into subsets ea h ontainingonly index tuples that lie on parallels to δ. We dene the diagonal segment D(c) atposition c = (c1, c2, . . . , cN−1)

T ∈ ZN−1 as

D(c) = i ∈ I : i = r(c) + s · 1TN , s = 0, 1, . . . , λ(c)− 1, (3.8)where r(c) is the segment's referen e point, λ(c) is the segment length, and the dire tionve tor 1T

N ensures that always points parallel to the main diagonal are aptured. Unlessmentioned otherwise, we hoose the referen e point as r(c) = (0 + s, c1 + s, c2 +

s, . . . , cN−1 + s)T sele ting s su h that all oordinates of r(c) are non-negative withat least one oordinate equal to zero. The segment length λ(c) is hosen su h thatall points in the segment have oordinates smaller or equal to L− 1 with at least one oordinate equal to L− 1 (see Figure 3.8).We say that an index tuple i lies on a ertain diagonal segment D(c), if i ∈ D(c)and, similarly, we say that it lies within the volume V(r) of radius r, if i ∈ V(r).At this point it is worth pointing out that all index ve tors i ∈ D(c) have thesame minimum distan e d(i, δ) = d(r(c), δ) to the main diagonal. Therefore if one ofmember of the diagonal segment D(c) lies within the volume V(r) then all its memberslie within the volume, giving all of them the same 'relevan e' in terms of de odability.We shall refer to d(i, δ) shortly as d(c). In Appendix B.2 and Appendix B.3 we showthatd(c) =

√√√√

N−1∑

n=1

c2n −

1

N

(N−1∑

n=1

cn

)2

. (3.9)The fun tional relationship between the Diagonal segment's position c and its distan eto the main diagonal d(c) is illustrated in Figure 3.9 (a), (b), and ( ) for N = 2, 3,

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 69

Figure 3.9: Graphi al representation of the main diagonal distan e d(c) as a fun tion ofthe diagonal segment position c = (c1, c2, . . . , cN−1)T . (a) For N = 2 there are two possiblediagonal segment positions c = ±c1 with the same distan e to the main diagonal of d(c) = r.(b) For N = 3 all diagonal segment positions c = (c1, c2)

T with d(c) = r an be found on atwo-dimensional ellipse. ( ) Similarly, for N = 4 the positions c = (c1, c2, c3)T with d(c) = rlie on a three-dimensional ellipsoid.and 4, respe tively. We observe, without further proof, that in the ase of N sour esthe equality d(c) = r des ribes a (N − 1)−dimensional quadrati surfa e, a quadri ,whi h in our parti ular ase be omes a (N − 1)−dimensional ellipsoid.3.4.1.3 De odability and SegmentationConsidering the sour e segments D(c), the de odability riterion in Lemma 3.4.1 anbe reformulated. We say that a odeword tuple w lies on a diagonal segment D(c), ifthere is an index tuple i ∈ D(c) that is mapped onto it. Then, using Lemma 3.4.1, weare able to state that the volume V(r) of radius r is de odable, if there are no dupli ate odeword tuples w on any diagonal segment D(c) or any pair of diagonal segmentswith d(c) ≤ r. This new formulation of de odability allows for the mathemati al odeanalysis we now present.3.4.2 Diophantine Code AnalysisFor the ode design it be omes ne essary to determine if and where a onsidered odeword tuple w = (w1, w2, . . . , wN)T an be found on a given diagonal segment

D(c).

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 703.4.2.1 Properties of the Diagonal Segments D(c)Given the referen e point r(c) = (r1(c), r2(c), . . . , rN(c))T and using the denition ofthe diagonal segments D(c), it is easy to verify that any index tuple i lying on D(c) an be expressed relative to r(c), su h that i = r(c) + s · 1N , where s is an arbitraryinteger. The orresponding indi es in an be expressed relative to rn(c) su h thatin = rn(c) + sn, where sn is an arbitrary integer, n = 1, 2, . . . , N . In the following, weshall also refer to s and sn as the position, or the lo ation, of the index tuple i andof the indi es in on the diagonal segment D(c), n = 1, 2, . . . , N . In this ase, we shallalso say that the index tuple and the indi es an be found, or lo ated, on the diagonalsegment.3.4.2.2 Properties of the Individual Diophantine Index AssignmentsLet wn be an arbitrary odeword of interest. Let s

(0)n be the position of an arbitraryindex i

(0)n mapped onto wn. Let Kn be the output alphabet size. Using the denitionof the Diophantine index assignments, it is easy to verify that an index in an only bemapped onto wn, if its position sn equals akKn + s

(0)n for some integer an. Otherwise,

in is not mapped onto wn.3.4.2.3 Joint Properties of the Diagonal Segments D(c) and the IndexAssignmentsLet w = (w1, w2, . . . , wN)T be an arbitrary odeword tuple of interest. Let s(0) =

(s(0)1 , s

(0)2 , . . . , s(0)

N ) be the position ve tor of an arbitrary index tuple i(0) = (i(0)1 ,

i(0)2 , . . . , i(0)N )T mapped onto w whi h is possibly, but not ne essarily, lo ated on D(c).Let K = (K1, K2, . . . , KN) be the ve tor of output alphabet sizes. Using the obser-vations above, it follows that an index tuple i = (i1, i2, . . . , iN)T lo ated on D(c) anonly be mapped onto the odeword tuple w, if its position s jointly equals

∆(K, s(0)) : a1K1 + s(0)1 = a2K2 + s

(0)2 = . . . = aNKN + s

(0)N (3.10)for some integer ve tor a = (a1, a2, . . . , aN). Otherwise, the index tuple i is not mappedonto w.Noti e that the variables an, Kn, and s

(0)n , n = 1, 2, . . . , N , as well as the sought-after solutions to ∆(K, s(0)) ( orresponding to the potential positions s) are integeronly. Therefore, we shall also refer to the equalities ∆(K, s(0)) as the Diophantineequation.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 713.4.2.4 Finding the Codeword Tuple w on the Diagonal Segment D(c)The observations above allow us to formulate a strategy to determine whether andwhere the odeword tuple w = (w1, w2, . . . , wN)T an be found on the diagonal segmentD(c). It an be summarized as follows: Given the odeword tuple w (and the alphabetsize ve tor K), we are able to determine an arbitrary index tuple i(0) mapped onto w.The position ve tor s(0) an then be obtained by subtra ting the individual oordinatesof the index tuple i(0) and the referen e point r(c). Using K together with s(0) allowsus to set up the Diophantine equation ∆(K, s(0)) a ording to (3.10). The obtainedsolutions to ∆(K, s(0)), if they exist, then provide us with the possible positions s ofthe odeword tuple w on D(c). In the following, we will show how to solve ∆(K, s(0)):rst for the ase of N = 2, then for the general ase of N > 2.3.4.2.5 Solutions for the Case of N = 2Let K = (K1, K2). Let s(0) = (s

(0)1 , s

(0)2 ). Let gcd(K1, K2) denote the greatest ommondivisor of K1 and K2, e.g., see [Ros10. The Diophantine equation

∆(K, s) : a1K1 + s(0)1 = a2K2 + s

(0)2 (3.11) an be solved using the following lemma:Lemma 3.4.2. (Solutions for N = 2) The Diophantine equation ∆(K, s(0)) has anintegral solution, i gcd(K1, K2) divides (without reminder) s

(0)1 − s

(0)2 . A parti ularsolution a(0) = (a

(0)1 , a

(0)2 ) is then given by

a(0)1 = −x

s(0)1 − s

(0)2

gcd(K1, K2)and a

(0)2 = +y

s(0)1 − s

(0)2

gcd(K1, K2), (3.12)where the integers x and y are hosen to fullll xK1 + yK2 = gcd(K1, K2).7 If thereis one solution then it has indeed innitely many solutions, whi h are given by

a1 = a(0)1 + t ·

K2

gcd(K1, K2)and a2 = a

(0)2 + t ·

K1

gcd(K1, K2), (3.13)where t runs through all integers. If gcd(K1, K2) does not divide s

(0)1 − s

(0)2 , then thereis no solution.Proof. Reformulating ∆(K, s(0)) yields (−K1)a1 + K2a2 = (s

(0)1 − s

(0)2 ). This orre-sponds to a standard linear Diophantine equation in two variables whose solutions arepresented in [Ros10.7Noti e that be ause of Bézout's identity the existen e of the integers x and y is guaranteed. They an be determined using the extended Eu lidean algorithm.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 72Using the previous result, we are able to on lude that the (all) solutions s to theDiophantine equation ∆(K, s(0)) an be expressed in the form s = aK + s(0), where aruns through all integers and K as well as s(0) are integer parameters. The parametersK and s(0) an be obtained by inserting the result in (3.12) as well as (3.13) into (3.11),after some manipulation, we obtain the following equality

t︸︷︷︸

a

·K1K2

gcd(K1K2)︸ ︷︷ ︸

K

+ a(0)1 K1 + s

(0)1

︸ ︷︷ ︸

s(0)

= t︸︷︷︸

a

·K1K2

gcd(K1K2)︸ ︷︷ ︸

K

+ a(0)2 K2 + s

(0)2

︸ ︷︷ ︸

s(0)

. (3.14)Comparing the oe ients yields K = K1K2

gcd(K1K2)and s(0) = a

(0)1 K1+s

(0)1 = a

(0)2 K2+s

(0)2 .Noti e that K = K1K2

gcd(K1K2) orresponds to the denition of the lowest ommon multipleof K1 and K2, hen eforth denoted as lcm(K1, K2), e.g., see [Ros10.3.4.2.6 Solutions for the General Case of N > 2Assume that M is an arbitrary subset of N = 1, 2, . . . , N and let KM and sM bethe parameters to the Diophantine equation ∆(KM, sM).Proposition 3.4.3. (Requirement for Solution, N > 2) If the equation ∆(KM, sM)has no solution for someM⊆ N , then the equation ∆(K, s) has no solution.Proof. We rst show that the solutions to ∆(K, s) will always be a subset to thesolutions of ∆(KM, sM). (a) The ase of |M| = 1 is trivial. (b) The ase of |M| = 2:let M = k, l. Using the result in Lemma 3.4.2 and the subsequent observationsin (3.14), it be omes lear that the solutions to ∆(Kk,l, sk,l) : akKk + sk = alKl + sl,whi h an be expressed in the form ak,lKk,l + sk,l, will always be a subset of thesolutions to akKk + sk and alKl + sl, if they exist. ( ) The ase of |M| > 2: Let

L ⊂ M, |L| = |M| − 1. By indu tion, the solutions to ∆(KM, sM) will always be asubset of the solutions to ∆(KL, sL). This parti ularly also holds true for M = Nestablishing the rst part of the proof.Sin e the solutions to ∆(K, s) will be a subset of the solutions to ∆(KM, sM), itfollows that ∆(K, s) annot have a solution when ∆(KM, sM) does not have a solution,establishing the laim.Proposition 3.4.4. (Existen e of Solution, N > 2) The equation ∆(K, s) has asolution, i ∆(KM, sM) has a solution for anyM⊆ N .Proof. If ∆(KM, sM) has a solution for anyM ⊆ N , then it is possible to onstru tan overall solution to ∆(K, s) (see hierar hi al algorithm below). On the other hand,if there is no solution to ∆(KM, sM) for someM ⊆ N , then it follows (see Proposi-tion 3.4.3) that there an be no overall solution to ∆(K, s).

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 73

Figure 3.10: Hierar hi al onstru tion method. Example for N = 3.We are able to formulate a hierar hi al method for onstru ting the overallsolution to ∆(K, s). Choosing an arbitrary pair k, l ∈ N as starting point andassuming, in on ordan e with the requirement in Proposition 3.4.4, that the orre-sponding equation ∆(Kk,l, sk,l) has a solution, we are able to use Lemma 3.4.2 and onstru t the solution to ∆(Kk,l, sk,l). This solution, given by the single expressionak,lKk,l + sk,l, an then be used to repla e the equality ∆(Kk,l, sk,l) in ∆(K, s)leading to a new Diophantine equation with a redu ed number of equalities. Thesame pairwise onstru tion prin iple an then be repeated, while assuming that therequirement in Proposition 3.4.4 holds, until only a single expression of the formaNKN + sN remains, also denoted as aK + s, orresponding to the overall solution to∆(K, s) (also see Figure 3.10). The parameter K and the parti ular solution s are theoutputs of the pro edure. It is easy to verify, as a dire t onsequen e of the presentedpro edure, that K = lcm(K1, K2, . . . , KN).3.4.2.7 Analysis of Subsequent SolutionsThe results so far provided us with the mathemati al means to solve the Diophantineequation ∆(K, s) in (3.10) for a given alphabet size ve tor K = (K1, K2, . . . , KN)and individual realizations of the position ve tor s = (s1, s2, . . . , sN). In the followingthe goal is to generally determine, for shifted versions of the position ve tor s + t =

(s1 + t, s2 + t, . . . , sN + t), t ∈ Z, the solutions to ∆(K, s+ t). The following propositionis useful:Proposition 3.4.5. (Shifted Solutions) The equation ∆(K, s + t) has solutions, ithe equation ∆(K, s) has solutions. The solutions to ∆(K, s + t) are then given by

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 74aK + s + t, where aK + s are the solutions obtained for ∆(K, s) and a runs throughall integers.Proof. We start with the ase of N = 2. Considering the pair k, l ∈ N andreformulating the onditions in Lemma 3.4.2 by repla ing sk by sk + t and sl bysl + t, we noti e that sk + t − (sl + t) = sk − sl does not alter the ondition for thedivisibility and, thus, the existen e of a solution. The expression for the solution anthen be obtained by substituting the ounterpart of sk,l in (3.14) by sk,l + t. For the ase of N > 2 it is generally possible to onstru t the overall solution to ∆(K, s), ifa solution exists, by applying the pairwise hierar hi al onstru tion method des ribedabove. This orrespond to solving ∆(K, s + t) for t = 0. We noti e that for anyt > 0 and at ea h step of the pro edure, the ondition for the existen e of a solutionremains un hanged and it is easy to verify that, similar to the ase of N = 2, theoverall solution an be obtained by substituting s by s + t, establishing the desiredresult.We observe that shifted position ve tors lead to shifted solutions, if a solutionexists. This shall turn out useful to simplify the ode analysis, parti ularly be ause itallows us to treat entire odeword intervals by solving a single Diophantine equation.This shall be addressed now.Let I1 = [0, 1, . . . , λ1 − 1] and I2 = [0, 1, . . . , λ2 − 1] be two integer intervals of(stri tly) positive lengths λ1 and λ2. The interval I1 is used to dene a olle tion ofsubsequent position ve tors s + t = (s1 + t, s2 + t, . . . , sN + t), where t runs throughall integers in I1. The interval I2 denes a range of potential solutions to ∆(K, s + t).Eventually, we want to test whether there is some t ∈ I1 su h that ∆(K, s + t) has asolution in I2.Proposition 3.4.6. (Solution Intervals)(a) Case λ1 + λ2 ≤ K: There is some t ∈ I1 = [0, 1, . . . , λ1 − 1] su h that theequation ∆(K, s+t) has a solution in the interval I2 = [0, 1, . . . , λ2−1], i the equation∆(K, s) has a solution in the interval It = [−λ1 + 1,−λ1 + 2, . . . , λ2 − 2, λ2 − 1].(b) Case λ1 + λ2 > K: There is always some t ∈ I1 su h that the equation∆(K, s + t) has a solution in the interval I2.Proof. (a) We onsider the su ient part rst. A ording to the previous results, thesolutions to ∆(K, s) are then given by the expression aK + s, where s and K theobtained parameters after solving the equation and a runs through all integers. Usingthe insights about shifted solutions in Proposition 3.4.5 observe that the solutions to∆(K, s + t) are then given by aK + s + t, where t is an arbitrary integer and the

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 75Figure 3.11: Interval test.other parameters remain un hanged. Considering the entire interval for t runningthrough I1, the resulting solutions an then be expressed by the olle tion of intervals

Is = [aK + s, aK + s + 1, . . . , aK + s + λ1 − 1] with a running through all integers.Comparing the intervals Is with the test interval I2, we observe that, for them tooverlap (also see Figure 3.11), the equation ∆(K, s) must have, for some integer a, asolution aK + s in the interval It, establishing the su ient part. The ne essary partis trivial.(b) For the ase of λ1 + λ2 > K, it is easy to verify that the intervals Is and I2will always overlap for some t ∈ I1, thus, establishing the laim.We an now use the obtained results to formulate a sear h algorithm for the odeanalysis.3.4.3 Combinatorial Sear hWe propose an iterative algorithm to determine whether there are dupli ate odewordtuples w within the volume V(r) of radius r. The main prin iple is to determine for allpossible pairs of diagonal segments D(1) = D(c(1)) and D(2) = D(c(2)) with distan esd(1) = d(c(1)) and d(2) = d(c(2)) smaller or equal to r, whether there are dupli ate odeword tuples. Let D(1) and D(2) be alled the start and destination segment,respe tively. The dupli ity is evaluated by hoosing one parti ular odeword tupleon the start segment and nding its position on the destination segment. Spe i ally,we onsider the odeword tuple w obtained by mapping the referen e point r(1) =

r(c(1)) on the start segment and employ the Diophantine analysis des ribed in theprevious se tion to determine i and where w appears on the destination segment (seeFigure 3.12). Using the obtained solution for this parti ular odeword tuple, we aninfer if there are any dupli ate odeword tuples within the entire s ope of the start anddestination diagonal segments. The whole pro edure is summarized by Algorithm 3.Noti e that the ase of c(1) = c(2) orresponds to the trivial ase where thestart and destination diagonal segments are identi al. Clearly the out ome of theDiophantine analysis would indi ate dupli ate odeword tuples in this ase. However,

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 76Algorithm 3: Combinatorial ode test• Sele t a start segment D(1) and a destination segment D(2) by hoosing1c(1), c(2) ∈ Z

N−1 with d(1), d(2) ≤ r.• Look-up (or al ulate) the orresponding referen e points r(1), r(2) and the segment2 lengths λ(1), λ(2).• Cal ulate the parameters of the Diophantine equation ∆(K, s) to sear h for the3 odeword tuple w resulting from r(1) on the destination segment D(2) with referen epoint r(2):sn ← r

(1)n − r

(2)n for n = 1, 2, . . . , N .

• Solve Diophantine equation ∆(K, s) for K = (K1, K2, . . . , KN) and4s = (s1, s2, . . . , sN) using Lemma 3.4.2 and Proposition 3.4.4 and determine theparti ular solution (if it an be found) as well as the orresponding position s.• Use s to de ide whether there are dupli ate odeword tuples within the entire s ope5 of the start and destination segments of length λ(1) and λ(2) using Proposition 3.4.6.• Repeat for all possible segment pairs and break if there are dupli ate odeword6 tuples.sin e those 'dupli ate' odeword tuples normally orrespond to the original ones, thetest out ome would be ome a false positive. This trivial ase an be treated separately;it an be omitted whenever λ(m) = λ(c(m)) ≤ K = lcm(K1, K2, . . . , KN), m = 1, 2,and therefore shall not be dis ussed further here.Using the des ribed pro edure it an be ensured that there are no dupli ate odeword tuples in the volume V(r) of radius r, if all possible segment ombinationsD(1) and D(2) with distan e d(1) and d(2) smaller or equal to r are tested. In orderto do so it would be ne essary to identify all pairs c(1) and c(2) that jointly satisfythis distan e onstraint. In (3.9) we provided an analyti expression for the distan ed(c) given a spe i (segment) position c and observed that d(c) = r des ribes an(N −1)−dimensional ellipsoid. In order to ount the possible ve tors c ∈ Z

N−1 fallingwithin the ellipsoid with a distan e d(c) smaller or equal to r one ould transformthe ellipsoid (in luding the positions of c) su h that solutions to the Gauss-Cir leproblem [Coh80 (and its higher dimensional extensions) are appli able. Assuming weget n su h points, then it is easy to onstru t a list of all pairs whose length is givenby the binomial oe ient (n2

). We observe, without providing further details, thatthe presented ombinatorial ode analysis will only be omputationally tra table forsmall values of N and r, motivating the following approa h.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 77

Figure 3.12: Finding the position of a odeword tuple on the destination diagonal.3.5 Diophantine Code Design and the FundamentalTheorem of Arithmeti Re all that the prin ipal goal pursued during the ode design is to ensure that thereare no dupli ate odeword tuples within the volume V(r) of radius r. As shown in theprevious se tion this an be a hieved by abstra ting the sour e through a olle tion ofdiagonal segments and analyzing the ode by means of a ombinatorial test strategy.In parti ular, we needed to test, for all diagonal segments D(c) with d(c) ≤ r, whetherthe odeword tuples on an arbitrary start segment D(1) = D(c(1)) appear (doubly) onan arbitrary destination segment D(2) = D(c(2)). In the following, we on eptuallyfollow a similar approa h, assuming a sour e abstra tion based on diagonal segments,but restri t ourselves to the most relevant sour e ongurations whi h eventually allowsus to formulate a onstru tive framework for the systemati ode design.For the ode design it shall be useful to introdu e the re tilinear distan e metri d1(c, c

(0)) =

N−1∑

n=1

d1(cn, c(0)n ), (3.15)whi h will allow us to quantify the distan e between the diagonal segments D(1) and

D(2) at positions c = (c1, c2, . . . , cN−1)T and c(0) = (c

(0)1 , c

(0)2 , . . . , c(0)

N−1)T , c, c(0) ∈

ZN−1, in a fashion that is well-suited for the ode design. The re tilinear distan emetri an be seen as an (N−1)−dimensional version of the (one-dimensional) distan emeasure introdu ed in Se tion 3.3.2 with d1(cn, c

(0)n ) = |cn − c

(0)n | or d1(cn, c

(0)n ) =

min|cn− c(0)n |, L−|cn− c

(0)n |, depending on whether the sour e symbols are bounded

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 78or are allowed to wrap around their boundaries, respe tively.3.5.1 Symmetri Design for N = 2We start by rening the ode analysis for the ase of N = 2. The rst step is to onsider an arbitrary pair of start and destination diagonal segments D(1) = D(c(1)1 )and D(2) = D(c

(2)1 ) at positions c(1)

1 , c(2)1 ∈ Z (also ompare Figure 3.12). The orresponding referen e points are r(1) = r(c

(1)1 ), r(1) = (r

(1)1 , r

(1)2 )T , and r(2) = r(c

(2)1 ),

r(2) = (r(2)1 , r

(2)2 )T , respe tively. The position of any odeword tuple w on the startsegment D(1) an then be des ribed in the form r

(1)1 + t and r

(1)2 + t, where t is anarbitrary integer. This allows us to set up the Diophantine equation ∆(K, s) with thealphabet sizes K = (K1, K2) and the shifts s = (s1, s2). The shifts s are hosen toidentify the odeword tuple w on the destination segment D(2), where s1 = r

(1)1 +t−r

(2)1and s2 = r

(1)2 + t− r

(2)2 . Thus, s1− s2 = (r

(1)1 − r

(1)2 )− (r

(2)1 − r

(2)2 ). Using the fa t that

r(m)1 − r

(m)2 = −c

(m)1 , m = 1, 2, we obtain s1 − s2 = c

(2)1 − c

(1)1 .Based on this observations and using Lemma 3.4.2, we are able to on ludethat the equation ∆(K, s) has a solution, i gcd(K1, K2) divides c

(2)1 − c

(1)1 (withoutremainder) or, alternatively, i c

(2)1 − c

(1)1 = t · gcd(K1, K2) for some integer t. Only inthis ase there might be dupli ate odeword tuples depending on the a tual segment ongurations.8 Otherwise, i.e., if c

(2)1 − c

(1)1 6= t · gcd(K1, K2) for any integer t, theequation ∆(K, s) has no solution and, thus, there an be no dupli ate odeword tuplesirrespe tive of the segment lengths. This in turn means that, given the start diagonal

D(1), we only need to onsider the destination diagonals D(2) at positions satisfyingc(2)1 − c

(1)1 = t · gcd(K1, K2) for some integer t 6= 0. All other destination diagonals donot need to be onsidered at all. This an be exploited to simplify the ombinatorial ode design onsiderably.Exploiting the symmetry properties of the sour e, parti ularly the hara teristi sof the volume V(r) of radius r, we are able to simplify the ode design onsiderably.We observe that the diagonal segments D = D(c1) falling into the volume V(r) aresymmetri ally bounded around the main diagonal (also ompare Figure 3.13 (a1)and (a2)). Furthermore, those diagonal segments an be hara terized by a set ofdiagonal segments at subsequent positions c1. Assuming that C subsequent diagonalpositions c1 are required to des ribe the volume V(r), we introdu e the threshold8Noti e that the ase of c

(2)1 = c

(1)1 or, equivalently, the ase of t = 0 orresponds to the trivialsetting where the start and destination diagonal segments are identi al. Clearly, we obtain dupli ate odeword tuples in this ase. However, similar as in Se tion 3.4.3, this trivial ase an be omitted for

λ(m) ≤ lcm(K1, K2), m = 1, 2. In the following, we shall assume that t 6= 0 whenever this is the ase.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 79

Figure 3.13: Symmetri design example for N = 2. Considering a sour e with a rotationalsymmetri al volume V(r) of radius r around the main diagonal, the index tuples i = (i1, i2)T ∈

V(r) an be represented by a olle tion of diagonal segments D(c1) with d1(c1, 0) ≤ Γ(C).For the ase without wrapping-around the orresponding diagonal segments are indi ated byshaded areas in (a1), and the orresponding interval for c1 is shown in (b1). This is illustrateda ordingly in (a2) and (b2) for the ase where the wrap-around ee t is taken into a ount.Γ = ⌊C−1

2⌋,9 su h that hoosing c1 with a re tilinear distan e to the main diagonalof d1(c1, 0) ≤ Γ ensures that the Eu lidean distan e to the main diagonal satises

d(c1) ≤ r.10 In the following, we shall sometimes refer to Γ as Γ(C) to highlight itsdependen y on C (also ompare Figure 3.13 (b1) and (b2)). Noti e that, sin e d1(c1, 0)is bounded by Γ(C), also d1(c(2)1 , c

(1)1 ) will be bounded by it. In parti ular, it is easy toverify that if d1(c

(m)1 , 0) ≤ Γ(C), m = 1, 2, it must follow that d1(c

(2)1 , c

(1)1 ) ≤ 2 ·Γ(C).Let C ≤ gcd(K1, K2). It is easy to verify that Γ(C) < gcd(K1, K2) and 2·Γ(C) <

gcd(K1, K2). Therefore, if d1(c(2)1 , c

(1)1 ) ≤ 2 · Γ(C), it follows that d1(c

(2)1 , c

(1)1 ) <

gcd(K1, K2). In the following, we shall assume that for the onsidered index assign-9As a dire t onsequen e of the symmetry requirement, it is only possible for odd values of Cto a ommodate the maximum number of distin t diagonal positions c1 between the thresholds ±Γ.For even values of C this is only possible for C−1 positions resulting in an overly pessimisti , butnevertheless valid, ode evaluation.10How Γ and r relate to ea h other is dis ussed in Se tion 3.5.3

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 80ment pair only start and destination segments at positions d1(c(2)1 , c

(1)1 ) ≤ 2 · Γ(C) arerelevant for the ode design.For the ase without wrapping-around (as des ribed in Se tion 3.3.2, also om-pare Figure 3.13 (a1) and (b1)), it dire tly follows that c

(2)1 − c

(1)1 6= t · gcd(K1, K2) forany integer t 6= 0. Consequently, there are no dupli ate odeword tuples in this ase.For the ase with wrapping-around (also ompare Figure 3.13 (a2) and (b2)),the equality c

(2)1 − c

(1)1 = t · gcd(K1, K2) might generally be satised for some t 6= 0,even though d1(c

(2)1 , c

(1)1 ) ≤ 2 · Γ(C) holds. In this ase, in order to guarantee the ode admissibility, we also need to ensure that there are no dupli ate odeword tuplesamong the relevant start and destination diagonal segments satisfying c

(2)1 − c

(1)1 = t ·

gcd(K1, K2) for any integer t 6= 0. This is parti ularly true for all start and destinationsegments satisfying d1(c(2)1 , c

(1)1 ) ≤ 2 · Γ(C), sin e those segments are the ones used todetermine the ode admissibility ( ompare shaded areas in Figure 3.13 (a2)). In orderto simplify the following dis ussion, we shall now fo us on the relevant ase of L being hosen equal to the lowest ommon multiple lcm(K1, K2).11 Using the fa t that thedieren e c

(2)1 − c

(1)1 must be a multiple of gcd(K1, K2) for a solution to exist, togetherwith the fa t that only diagonal segments satisfying d1(c

(2)1 , c

(1)1 ) ≤ 2 ·Γ(C) need to be onsidered, where d1(c

(2)1 , c

(1)1 ) itself depends on L, we on lude that c

(2)1 − c

(1)1 must bea multiple of L = lcm(K1, K2) to jointly a ount for both requirements. In parti ular,in order to apture both requirements, the equality c

(2)1 − c

(1)1 = s · lcm(K1, K2) mustbe satised for some integer s. Noti e that the ase of s = 0 orresponds to the trivial ase (whi h does not need to be onsidered expli itly). Furthermore, the ases where

|s| ≥ 2 an not arise within our system setting sin e the possible dieren es c(2)1 − c

(1)1are restraint by the (symmetri ) quantizer resolution L. Consequentely, when testingfor odeword dupli ity, we only need to onsider the ases of |s| = 1, i.e., the aseswhere c

(2)1 − c

(1)1 = ± lcm(K1, K2) is satised. The following proposition ensures thatthere are no dupli ate odeword tuples in this ases:Proposition 3.5.1. If L ≤ lcm(K1, K2), then there are no dupli ate odeword tuplesamong the diagonal segments satisfying c

(2)1 − c

(1)1 = ± lcm(K1, K2).Proof. Considering the ase where L ≤ lcm(K1, K2) and assuming that the twodiagonal segments satisfy c

(2)1 − c

(1)1 = ± lcm(K1, K2) (as shown in Figure 3.14 (a)),it is easy to verify that it is always possible to jointly map both diagonal segmentsonto the main diagonal at position c1 = 0 without any overlap among them (see11Using Proposition 3.4.6 and the properties of our system setup, it is easy to verify that L =

lcm(K1, K2) orresponds to the maximum possible hoi e of L without enfor ing dupli ate odewordtuples.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 81

Figure 3.14: Example for N = 2 and L ≤ lcm(K1,K2). The diagonal segments at positionc1 = c

(m)1 , m = 1, 2, satisfying c

(2)1 − c

(1)1 = ± lcm(K1,K2) in (a) an jointly be mapped ontothe main diagonal at postion c1 = 0 in (b). In parti ular, this an be a hieved by a simplepermutation of the index blo ks A and B.Figure 3.14 (b)). In parti ular, this an be a hieved by a simple permutation of theindi es i1 or i2 ( ompare blo k A and B in Figure 3.14). It is easy to onvin e yourselfthat all te hniques ( on erning the ode analysis) derived so far also apply to thesystem obtained after index permutation. We are therefore able to on lude that forthe ase of L ≤ lcm(K1, K2) there are no dupli ate odeword tuples among the newlyderived diagonal segments at position c1 = 0 (shown in Figure 3.14 (b)). Furthermore,sin e the performed index permutations do not alter the odeword label of the diagonalsegment members, this must also hold true for the original pair of diagonal segments(shown in Figure 3.14 (a)) establishing the laim.We on lude that for the relevant ase of L = lcm(K1, K2), no dupli ate odewordtuples o ur whenever c

(2)1 − c

(1)1 = ± lcm(K1, K2). Consequentely, the same odeproperties hold as for the ase without wrapping-around.3.5.2 Generalization for N > 2We now onsider the ode analysis for N > 2. In parti ular, we are interested inestablishing su ient onditions that allow us to assure that there is no solution to

∆(K, s), with K = (K1, K2, . . . , KN) and s = (s1, s2, . . . , sN), eventually providingus with the means for a worst- ase ode evaluation that an be dire tly used for thesystemati ode design. As a dire t onsequen e of Proposition 3.4.3, we observe that

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 82∆(K, s) an only have an overall solution, if ∆(Kk,l, sk,l), with Kk,l = (Kk, Kl)and sk,l = (sk, sl), has a solution for any index assignment pair k, l ∈ N , k 6= l.Only in this ase there might be dupli ate odeword tuples. Otherwise, there is nosolution and, thus, there are no dupli ate odeword tuples.For our worst- ase ode evaluation we parti ularly onsider the olle tion of indexassignment pairs k, l = 1, n+1, n = 1, 2, . . . , N−1, satisfying⋃N−1

n=1 1, n+1 = N ,despite the fa t that other hoi es would also be possible.12 This parti ular hoi ehowever has the advantage that it ni ely agrees with the denition of the sour eparameter cn used to des ribe the joint properties of the sour e pairs 1, n + 1,n = 1, 2, . . . , N − 1.Using the previous observations together with Lemma 3.4.2, we are able to on- lude that there only an be dupli ate odeword tuples, if c

(2)n −c

(1)n = t ·gcd(K1, Kn+1)for some integer t and any n ∈ 1, 2, . . . , N − 1. Otherwise, i.e., if c

(2)n − c

(1)n 6=

t·gcd(K1, Kn+1) for any integer t and some n ∈ 1, 2, . . . , N−1, there are no dupli ate odeword tuples. We now use these insights to simplify the ombinatorial ode design.Let C ≤ gcd(K1, K2, . . . , KN), an assumption, whi h dire tly ensures that C ≤

gcd(K1, Kn+1), n ∈ 1, 2, . . . , N−1. Consequently, whenever d1(c(2)n , c

(1)n ) ≤ 2·Γ(C), itfollows that d1(c

(2)n , c

(1)n ) < gcd(K1, Kn+1), n ∈ 1, 2, . . . , N−1. We shall assume thatonly diagonal segments at positions c(m) = (c

(m)1 , c

(m)2 , . . . , c(m)

N−1)T ∈ Z

N−1, m = 1, 2,that satisfy d1(c(2)n , c

(1)n ) ≤ 2 · Γ(C) for some n ∈ 1, 2, . . . , N − 1 are relevant for the ode design. Furthermore, onsidering the same symmetri sour e onguration as forthe ase of N = 2, we assume that the diagonal segments lose to the main diagonalare most relevant. This allows us to identify a ompa t set of diagonal segmentssymmetri ally bounded around the main diagonal.Let c(m) = (c(m)1 , c

(m)2 , . . . , c(m)

N−1)T , m = 1, 2, be an arbitrary pair of positionve tors with d1(c

(m), 0TN−1) ≤ b for some (positive) integer b. The following propositionis useful:Proposition 3.5.2. I b ≤ (N − 1) · Γ(C), then d1(c

(2)n , c

(1)n ) ≤ 2 · Γ(C) for some

n ∈ 1, 2, . . . , N − 1.Proof. Using the proposition's requirement it follows that d1(c(1), 0T

N−1)+d1(c(2), 0T

N−1)

≤ 2(N−1) ·Γ. Sin e d1(c(2)n , c

(1)n ) ≤ d1(c

(2)n , 0)+d1(c

(1)n , 0) for any n ∈ 1, 2, . . . , N−1,it follows that ∑N−1

n=1 d1(c(2)n , c

(1)n ) ≤ 2(N − 1) · Γ(C). Using this result it is easy tosee that d1(c

(2)n , c

(1)n ) > 2 · Γ(C) annot hold jointly for all n ∈ 1, 2, . . . , N − 1 and,thus, it must be true that d1(c

(2)n , c

(1)n ) ≤ 2 · Γ(C) for some n ∈ 1, 2, . . . , N − 1.12At this point we are merely interested in establishing su ient onditions for the non-existen eof an overall solution to ∆(K, s).

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 83

Figure 3.15: Example for N = 3. (a) Supported set C(C) for given threshold Γ(C). (b) Themaximum supported radius (C) an determined by hoosing the radius r su h that theellipsoid with d(c) ≤ r is as large as possible but still fully ontained in Γ(C). In parti ular,(C) is an be determined by hoosing the point c = (c1, c2)

T = ±(Γ(C),Γ(C))T . ( ) Giventhe radius r, the maximum required segment number B(r) is determined using the integerthreshold ∆(r).Otherwise, i.e., if b > (N − 1) · Γ(C), it is possible that d1(c(2)n , c

(1)n ) > 2 · Γ(C) for all

n ∈ 1, 2, . . . , N − 1 establishing the laim.The set of position ve tors c ∈ Z

N−1 satisfying d1(c, 0TN−1) ≤ (N − 1) · Γ(C)shall also be denoted as the supported set C(C) = c ∈ Z

N−1 : d1(c, 0TN−1) ≤ (N − 1) ·

Γ(C) (see Figure 3.15 (a)).For the ase without wrapping-around (as des ribed in Se tion 3.3.2) and when-ever c(1), c(1) ∈ C(C) it follows that the dieren e c(2)n −c

(1)n an not be a (non-trivial)multiple of gcd(K1, Kn+1) for some n ∈ 1, 2, . . . , N − 1. Consequently, there are nodupli ate odeword tuples in this ase.For the ase with wrapping-around this is not ne essarily true and a moredierentiated ode analysis is generally required. However, for the relevant ase of

L = lcm(K1, K2, . . . , KN), it is possible to show analyti ally that the same odeproperties an be obtained as for the ase without wrapping-around. This an bea hieved using the result for the ase of N = 2 presented in Se tion 3.5.1.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 843.5.3 Maximum Supported Radius andMinimum Required Seg-ment NumberIn the previous se tion we showed how to ensure that there are no dupli ate odewordtuples among all diagonal segments D(c) with c ∈ C(C), C(C) = c ∈ ZN−1 :

d1(c, 0TN−1) ≤ (N − 1) · Γ(C). In order to a hieve this, for a given segment number

C, we introdu ed the threshold Γ(C) that symmetri ally bounds the diagonal segmentpositions c ∈ ZN−1 around the main diagonal. In the following, we want to derive therelationship between the threshold Γ(C) and the possible radii r of the volume V(r).Let the maximum supported radius be the maximum possible radius r of thevolume V(r) su h that hoosing r ≤ ensures the nonexisten e of dupli ate odewordtuples among all diagonal segments D(c) falling into the volume V(r), i.e., among alldiagonal segments with d(c) ≤ r.Using the geometri properties of the supported set C(C) together with theproperties of the (N − 1)−dimensional ellipsoid des ribed by d(c) = r it followsby inspe tion (also see Figure 3.15 (b)) that the maximum supported radius anbe determined by hoosing the diagonal segment positions c = Γ(C) · 1T

N−1 or c =

−Γ(C) · 1TN−1 su h that

= d(Γ(C) · 1TN−1) =

√

N − 1

NΓ(C). (3.16)In the following, we shall also refer to as (C) or (C, N) to highlight its dependen yon C and N .In the limit, we obtain limN→∞ (Γ(C)) = Γ(C). Considering the ase for N = 2as referen e and letting N →∞, we obtain a maximum possible gain of 1.51 dB, verysimilar to the granular gain a hievable for ve tor quantizers and N →∞.For a sour e-optimized ode design it is even more important to onsider thereverse ase (also ompare Figure 3.15 ( )). For a given radius r, we introdu ethe minimum required segment number B su h that hoosing C ≥ B ensures thenonexisten e of dupli ate odeword tuples in the volume V(r). The number B isdetermined via the auxiliary threshold ∆, whi h needs to be an integer number, su hthat hoosing Γ(C) ≥ ∆ ensures those properties. Using the same approa h as beforewhen deriving (3.16) and taking into a ount that ∆ needs to be an integer numberwhereas r is ontinuous-valued, we are able to state that

∆ =

⌈√

N

N − 1· r

⌉

. (3.17)Sin e ∆ is a symmetri al threshold around the main diagonal, we obtain B = 2∆ + 1

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 85and, thus,B = 2 ·

⌈√

N

N − 1· r

⌉

+ 1. (3.18)In the following, to highlight their dependen y on r, we shall also refer to ∆ and B as∆(r) and B(r), respe tively.Using the results so far we observe that the ode properties strongly dependon the number-theoreti al properties of the alphabet sizes K1, K2, . . . , KN , namelythe greatest ommon divisor gcd(K1, K2, . . . , KN) and the lowest ommon multiplelcm(K1, K2, . . . , KN).Assuming that the alphabet sizes K1, K2, . . . , KN are known, we are able to on lude that there are no dupli ate odeword tuples in the volume V(r), if L ≤

lcm(K1, K2, . . . , KN) and there is some C ≤ gcd(K1, K2, . . . , KN) su h that r ≤ (C).Given the quantizer resolution L and the radius r as starting point for the odeevaluation, we observe that there are no dupli ate odeword tuples in the volumeV(r), if there exists K1, K2, . . . , KN su h that L ≤ lcm(K1, K2, . . . , KN) and there issome C su h that B(r) ≤ C ≤ gcd(K1, K2, . . . , KN). In those ases the ode tuple(L, K1, K2, . . . , KN , r) is admissible.Noti e that the latter problem of nding possible ongurations K1, K2, . . . , KNwhile ensuring that the sum-rate RΣ =

∑Nn=1 Rn =

∑Nn=1 log2(Kn) is minimized turnsout to be more ompli ated. This is mainly due to the fa t that it involves more degreesof freedom. In the following, we propose a design strategy exploiting systemati allythe prime fa tors of L to a hieve this goal.3.5.4 The Fundamental Theorem of Arithmeti and its Impli- ationsWe onsider the following ode design problem. Given the quantizer resolution L andthe radius r, the goal is to identify possible ongurations of K1, K2, . . . , KN su h thatthe ode tuple (L, K1, K2, . . . , KN , r) is admissible and (at the same time) the sum-rate

RΣ =∑N

n=1 Rn =∑N

n=1 log2(Kn) is at a minimum.We will employ the fundamental theorem of arithmeti to identify and hara -terize possible ode ongurations; it an be stated as follows:Theorem 3.5.3. (Fundamental Theorem of Arithmeti ) Any integer number greaterthan one an be written as a unique produ t of primes.Proof. For example, see [Ros10.For example, the integer number 18 an be uniquely represented by the produ tof prime numbers 2 · 3 · 3.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 86It will prove useful to des ribe prime fa torizations by the multisets of their primefa tors. Generally, a multiset is a set in whi h ea h of its elements might o ur morethan on e (the order its elements is irrelevant). For example, the prime fa torizationof 18 an be des ribed by the multiset 2, 3, 3.Noti e that multiset union, multiset interse tion, and multiset dieren e aredened to a ount for dupli ate elements. For example, 2, 3 ∪ 2 = 2, 2, 3,2, 2, 3 ∩ 2, 2 = 2, 2 and 2, 2, 3\2 = 2, 3.A ording to the fundamental theorem of arithmeti we are able to uniquelyrepresent L by a produ t of prime numbers. Then, using the obtained prime fa -tors, the goal is to determine the best possible (i.e., the smallest) segment numberC that satises C ≥ B(r) and (at the same time) allows for ode ongurationsK1, K2, . . . , KN that satisfy lcm(K1, K2, . . . , KN) = L and gcd(K1, K2, . . . , KN) ≥ C.The key-idea is to use the prime fa tors of L in order to determine C, and thenuse the remaining fa tors to onstru t possible ongurations of K1, K2, . . . , KN ina ombinatorial fashion by forming subsets of those fa tors. The fa t that the odeperforman e parameters lcm(K1, K2, . . . , KN) and gcd(K1, K2, . . . , KN) an also beexpressed via the prime fa tors of L eventually allows us to formulate a systemati design pro edure dire tly a ounting for the ode properties throughout design.The overall design pro edure is summarized by Algorithm 4 (also ompare Fig-ure 3.16).Algorithm 4: Prime fa tor ode design• Given the quantizer resolution L, nd the fa torization L =

∏

∀i pi, the fa tors pi1 being prime, and dene the multiset P =⋃

∀i pi.• Given the radius r, determine the minimum required segment number B(r). If2B(r) ≤ L then set C = B(r), otherwise set C = L.• Sele t the multiset G ⊆ P su h that g =

∏

∀pi∈Gpi is greater or equal to C but (at3 the same time) as small as possible (see Figure 3.16 (a)).

• Determine the multiset of remaining primes R = P\G (see Figure 3.16 (a)).4• Partition the multiset R into N multisets R1,R2, . . . ,RN , empty sets allowed, su h5 that ⋃N

n=1Rn = R (see Figure 3.16 (b1)-(b4)).• For n = 1, 2, . . . , N : Determine the multiset Kn = G

⋃Rn. If Rn is non-empty,6 al ulate rn =

∏

∀pi∈Rnpi, otherwise, set rn = 1. Cal ulate the alphabet size

Kn =∏

∀pi∈Knpi = g · rn.In the following, we omment on the main steps of Algorithm 4.Step 1: The prime fa torization of L provides us with the fundamental ompo-nents for our ombinatorial design approa h, the prime fa tors. Finding the prime

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 87

Figure 3.16: Systemati ode onstru tion based on prime fa tors for N = 3, L = 36,and C = B(r) = 3. (a) The prime fa tors of L are olle ted in the multiset P. We sele tthe multiset G ⊆ P su h that the produ t of the primes in G is greater or equal to C anddetermine the multiset of remaining primes R = P\G. (b1)-(b4) Possible partitions of Rinto N = 3 multisets R1,R2,R3. For the ases (b3) and (b4) it is not true that Rk

⋂Rl = ∅for any pair k, l ∈ N , k 6= l. ( 1)-( 4) Given the multisets Kn = G

⋃Rn, n = 1, 2, 3, thelowest ommon multiple lcm(K1,K2,K3) and the greatest ommon divisor gcd(K1,K2,K3) an be determined using the Venn diagram. For the ases ( 3) and ( 4) (resulting fromthe ases (b3) and (b4)) we on lude that L > lcm(K1,K2,K3), leading to an invalid ode onguration.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 88fa torization itself does not pose a serious problem, at least for pra ti ally relevantvalues of L. For example, it is possible to apply the so- alled Tree-Method [Ros10 to onstru t the fa torization.Step 2: Setting C = B(r) ensures that the full radius r will be supported by the onstru ted ode. The ase of B(r) > L orresponds to an exoti setting not relevantin pra ti al appli ations.Step 3: Choosing the multiset G ⊆ P is the rst step towards xinggcd(K1, K2, . . . , KN). The multiset G will ontain the prime fa tors ommon to K1,

K2, . . . , KN and therefore the produ t g =∏

∀pi∈Gpi will be smaller or equal to

gcd(K1, K2, . . . , KN). Choosing g ≥ C ensures that C ≤ gcd(K1, K2, . . . , KN).13 Therequirement that g should be as small as possible ensures, as shown later, that the prod-u t ∏N

n=1 Kn, thus, the sum-rate RΣ =∑N

n=1 Rn =∑N

n=1 log2(Kn) = log2(∏N

n=1 Kn),be omes as small as possible for the given value of C.Usually the multiset G ⊆ P an be easily found by inspe tion. However, itgenerally would be possible to onstru t all possible multisets G ⊆ P, in a brute-for e fashion, in order to identify the multiset G resulting in the smallest produ tg =

∏

∀pi∈Gpi that is greater or equal to C. An e ient implementation ould bebased on multiset partitions, as des ribed in [Knu05, and onsidering the partitionsthat onsist of one or two multisets only.Step 4: The multiset of remaining prime fa tors R = P\G will be used as thestarting point for the ombinatorial design of K1, K2, . . . , KN .Step 5 + 6: Goal is to distribute the primes inR into the subsets R1,R2, . . . ,RNto eventually dene the alphabet sizes K1, K2, . . . , KN .14 The requirement that

⋃Nn=1Rn = R ensures that all prime fa tors are used in onstru ting the alpha-bet sizes K1, K2, . . . , KN be ause, otherwise, we will never be able to ensure that

lcm(K1, K2, . . . , KN) = L. If additionallyRk

⋂Rl = ∅ holds for any pair k, l ∈ N =

1, 2, . . . , N, k 6= l, then ⋂m∈MRm = ∅ must also hold true for any subsetM⊆ N ,|M| ≥ 2. In parti ular, this must be the ase for M = N . Using the denitionof the greatest ommon divisor, it is easy to verify that gcd(K1, K2, . . . , KN) = gholds in this ase.15 Furthermore, the same additional requirement ensures thatlcm(K1, K2, . . . , KN) = g ·

∏N

n=1 rn = L. Thus, all riteria for the admissibility ofthe ode tuple (L, K1, K2, . . . , KN , r) are satised whenever Rk

⋂Rl = ∅ for any pair

k, l ∈ N , k 6= l.The lowest ommon multiple and the greatest ommon divisor an be derived,13In total, we obtain B(r) = C ≤ g ≤ gcd(K1, K2, . . . , KN).14Similar as before, the partitioning an be performed e iently by onstru ting the multisetpartitions of R and onsidering the partitions that onsist of N or less multisets only.15This would mean that B(r) = C ≤ g = gcd(K1, K2, . . . , KN ).

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 89among other methods, via the Venn diagram of the multisets Kn = G⋃Rn, n =

1, 2, . . . , N (see Figure 3.16 ( 1)-( 4)). The produ t of the alphabet sizes be omes∏N

n=1 Kn =∏N

n=1 g · rn = gN ·∏N

n=1 rn, whi h learly indi ates that g should be assmall as possible in order to minimize the sum-rate RΣ =∑N

n=1 Rn =∑N

n=1 log2(Kn) =

log2(∏N

n=1 Kn).It is worth pointing out that all of above steps generally an be performed forany setting of L and r. This is obvious for all steps, but possibly Step 6. However,it is easy to verify that in this ase it is always possible to hoose the anoni al odesetting of R1 = R and Rn = ∅, n = 2, 3, . . . , N , ensuring that ⋃N

n=1Rn = R andRk

⋂Rl = ∅ for any pair k, l ∈ N , k 6= l. The resulting ode, the anoni al ode, onsequently inherits alphabet sizes of K1 = g · r1 and Kn = g, n = 2, 3, . . . , N − 1(see Figure 3.16 ( 1)).In on lusion, given the parameters L and r, and therefore C = B(r), thepossible ongurations of K1, K2, . . . , KN merely depend on the possible arrangementsof the prime fa tors in L as well as their impa t on the lowest ommon multiple

lcm(K1, K2, . . . , KN) and the greatest ommon divisor gcd(K1, K2, . . . , KN). On onehand, this enables us to view the ode design problem from a strongly number-theoreti al perspe tive. In parti ular, it shows that the possible ode ongurations(and their resulting properties) are largely predetermined by the prime fa tors of L,whi h in itself is quite intriguing. On the other hand, from a more pra ti al point-of-view, it enabled us to formulate the ombinatorial design pro edure presented above.3.6 Coding without MemoryWe now onsider a system setup in whi h the sour e symbols are en oded, transmitted,and de oded in a memory-less fashion. This assumes that no knowledge about previous(or subsequent) sour e and hannel symbols is available when performing the odingoperations. Consequentely, we will dis ard the time index t for the remainder of thisse tion.3.6.1 S alar QuantizationWhenever the observed sour es symbols are of ontinuous-valued nature16 and when-ever a memory-less system setting is onsidered, we assume the usage of s alar quan-tizers optimized for the (marginal) probability distributions of the observations p(un),16For s enarios with dis rete sour e inputs, we simply omit the quantization pro ess des ribed here.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 90n = 1, 2, . . . , N .17 The design problem then onsits of nding a quantizer parti-tion (whi h denes the quantization regions) and the odebook (whi h denes there onstru tion values) su h that the mean squared error (MSE) in the observations(i.e., without onsidering any subsequent data pro essing) is minimized. Sin e thejoint probability distribution p(u) and, thus, the marginal distributions p(un) areknown, it be omes possible to employ the Lloyd-Max algorithm [JN84 to nd thequantizer partitions and odebooks for n = 1, 2, . . . , N . Alternatively, one ould hooseoptimized uniform quantizers [JN84.In our system setup (see Figure 3.1) the quantizer partitions and odebooksare required, both, for en oding as well as de oding purposes. At the en oder, thequantizer partition des ribes how the sour e samples un ∈ R are mapped onto thequantization indi es in ∈ I ( ompare Blo k Q in Figure 3.1). At the de oder, thequantization pro ess needs to be reversed in the last stage of our de oding pro e-dure ( ompare Blo k Q−1 in Figure 3.1). Here, the quantizer odebook des ribes howan estimate of the original ( ontinuous-valued) sour e samples un ∈ R is formed giventhe de oded indi es in ∈ I.At this point it is worth pointing out that the magnitude of the index in mustree t, from a topologi al point of view, the magnitude of the original sour e samplesun, sin e otherwise the ode properties would be destroyed. For example, onsider anordering of the quantization regions from left to right on the real line: if an arbitraryregion lies left of some referen e region, it must always follow that its index is smallerthan the index of the referen e region. In our parti ular ase, this is ensured byindexing the quantization regions from left to right, starting with zero and ending atL− 1.3.6.2 Memoryless BinningIn the following, we shall drop the time index t. The binning stage at en odern ( ompare Blo k Bn in Figure 3.1) des ribes how the quantization indi es in ∈ Iare mapped onto the odewords xn ∈ Xn.The index assignments bn : I → Wn des ribe how the quantization indi esin ∈ I are mapped onto the odewords wn ∈ Wn su h that wn = bn(in). For n =

1, 2, . . . , N we dene the binning stage at en oder n using the index assignment bn byassigning xn = wn, assuming that Xn =Wn.The index de oder ( ompare Blo k Φ in Figure 3.1) onsits of two prin ipalstages: the inverse binning stage where the binning stages are reversed (in a ombina-17Noti e that, generally, it also would be possible to optimize the quantizers for the joint statisti sp(u). However, for brevity, this idea is not further explored here.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 91torial fashion), followed by the index sele tion stage where the nal de ision is madeand the individual indi es are hosen.At the inverse binning stage, individually for ea h en oder n = 1, 2, . . . , N , wedetermine the inverse image I−1n (wn) = in ∈ I : bn(in) = wn for the re eived odeword wn ∈ Wn, i.e, the set ontaining all indi es in ∈ I mapped onto wn. Jointly onsidering the re eived data of all en oders, we also determine the inverse image

I−1(w) = i ∈ I : wn = bn(in), n = 1, 2, . . . , N for the re eived odeword ve torw ∈ W , W =

∏Nn=1Wn, i.e., the set ontaining all index ve tors i ∈ I = IN thatare mapped onto the given odeword ve tor w. It an be determined by forming the rossprodu t I−1(w) =

∏N

n=1 I−1n (wn).At the index sele tion stage, the de oder sear hes, for the given odeword ve tor

w ∈W , through all index ve tors in the inverse image I−1(w) and hooses one of theindex ve tors18 optimizing for the delity riterion of interest; all it i. The obtainedindex ve tor i ∈ I, the de oder output, shall also be denoted as i(w) to highlight itsdependen y on the odeword ve tor w ∈W . In the following, we will show how theindex sele tion an be implemented e iently.3.6.3 De oding Con epts and ComplexityWhenever s enarios with dis rete sour e inputs are onsidered and in a ordan e withthe problem statement in Se tion 3.3.1, we are interested in minimizing the error prob-ability. The optimal de oding rule is then given by the maximum a-posteriori (MAP) riterion [Poo94. Fo using on the memory-less ase and using the inverse imageI−1(w) for the re eived odeword ve tor w ∈ W , we an reformulate the MAP riterion as follows:

i = argmaxi∈I−1(w)

p(i). (3.19)For ontinuous-valued sour es and, again, fo using on the memory-less ase, the meansquared error (MSE) E||U−U||2 has to be minimized, where U = (U1, U2, . . . , UN)Tand U = (U1, U2, . . . , UN )T are random variable ve tors, representing the re onstru tedand the original sour e symbols, respe tively. The optimal de oder in this asewould be given by the onditional mean estimator su h that un = EUn|w =∑N

n=1 (EUn|in · p(in|w)), n = 1, 2, . . . N , where the onditional probabilities anbe expressed in terms of the probability mass fun tion p(i). Assuming that only oneindex ve tor i ∈ I−1(w) has signi ant probability, it follows that u ≈ EUn |in,18There might be more than one.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 92n = 1, 2, . . . , N , where i is hosen a ording to the MAP rule in (3.19).19Under the above assumptions, de oding for both the dis rete as well as the on-tinuous ase an be expressed via the MAP de oding rule in (3.19). The problem withan a tual implementation of the MAP de oder is, on one hand, that the probabilitiesp(i) need to be known and, on the other hand, that a sear h through all index ve torsi ∈ I−1(w) is required, whi h learly be omes intra table for large values of N , sin e|I−1(w)| in reases exponentially with N . The rst problem an be approa hed byexploiting the hara teristi properties of our sour e model, with the most probableindex tuples i ∈ I being lo ated in the proximity of the main diagonal δ. Usingthis property, we are able to evaluate the probability of the index tuples i ∈ I basedon their distan e d(i, δ) to the main diagonal δ al ulated a ording to (B.5). Thisparti ularly also holds true for the index tuples i in the inverse image I−1(w) thatare needed for de oding. Additionally, assuming that the probability p(i) is a non-in reasing fun tion of the distan e d(i, δ),20 the index tuples i ∈ I−1(w) with a smalldistan e to the main diagonal δ always have a larger (or equal) probability than theindex tuples further away. In this ase the MAP rule in (3.19) an be repla ed by

i = argmini∈I−1(w)

d(i, δ), (3.20)eliminating the need to know the a tual value of p(i) when sele ting the (MAP) indextuple i. A solution to the se ond problem will be presented in the next se tion.3.6.4 S alable Joint De odingWe have seen that the de oding omplexity is largely ae ted by the number of indextuples i in the inverse image I−1(w) for the re eived odeword tuple w ∈W . We willnow present a method to redu e the number of index tuples that need to be onsideredduring de oding, and propose a s alable de oding algorithm, feasible for large numbersof en oders N .The basi idea of the algorithm is to identify, among all index tuples in i ∈

I−1(w), the ones that are losest to the main diagonal δ in terms of their diagonaldistan e d(i, δ) al ulated a ording to (B.5), while ex luding others that do not need19Noti e that EUn|in an be al ulated easily using the quantizer partition. For quantizers with entroidal re onstru tion levels, as e.g. the ase for Lloyd-Max quantizers, the onditional expe tationvalues EUn|in orrespond to the odebook re onstru tion levels and do not need to be al ulatedat all.20For example, if the sour e only generates equiprobable index tuples i within the volume V(r)of some radius r, the requirement of p(i) being a non-in reasing fun tion of d(i, δ) is immediatelysatised.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 93

Figure 3.17: S alable de oding based on the su essive onstru tion method for N = 3.(a) At iteration step n = 2 the set H(a)2 is onstru ted for a = 0, 1 by forming the rossprodu t of the initial index i

(a)1 and losest index "below" the main diagonal i

(a,l)2 as well asthe losest index "above" the main diagonal i

(a,u)2 . (b) At iteration step n = N = 3 this isrepeated and H(a) = H

(a)3 is determined for a = 0, 1 by forming the rossprodu t betweenthe members of H(a)

2 and the newly found indi es i(a,l)3 and i

(a,u)3 .to be onsidered, either be ause they are farther away than other suitable andidates,or be ause they an not be onsidered, due to omplexity onstraints.21 In parti -ular, the losest index tuples are onstru ted by su essively stepping through thedimensions n = 1, 2, . . . , N while onstraining the total number of onsidered indextuples to a maximum and delaying the nal de ision, until the last dimension N isrea hed. Assume, e.g., that for n = 1 the index i

(a)1 ∈ I is mapped onto the odeword

w1 ∈ W1. Then, for n = 2, we are able to determine, among all the indi es i2 ∈ Ithat are mapped onto the odeword w2 ∈ W2, the pair of indi es i(a,l)2 , i

(a,u)2 ∈ Ithat are lo ated "below" and "above" the main diagonal δ at position i1 = i

(a)1 (seeFigure 3.17 (a)). Forming the rossprodu t between the index i

(a)1 and the index pair

i(a,l)2 , i

(a,u)2 , we obtain the set H(a)

2 = (i(a)1 , i

(a,l)2 )T , (i

(a)1 , i

(a,u)2 )T of the orrespondingindex tuples. The same pro ess an then be repeated for n = 3, 4, . . . , N ( ompareFigure 3.17 (b)).The overall de oding pro edure is summarized by Algorithm 5. Let i

(a)1 ∈ I,

a = 0, 1, . . . , A − 1, be the indi es mapped onto the odeword w1 ∈ W1. In order21Using the ' loseness' to the main diagonal as a riterion to dis ard index tuples an be justiedbe ause, onsidering our sour e model, the most probable index tuples are lo ated in the proximityof the main diagonal.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 94to keep the resulting de oding omplexity as low as possible, and without loss ofgenerality, we assume that K1 ≥ K2 ≥ . . . ≥ KN . Furthermore, we introdu e the onstant M , limiting the total number of index tuples tra ked during de oding.Algorithm 5: S alable de odingInitialization1• Given the odeword w1 ∈ W1, determine the inverse image2I−1

1 (w1) = i1 ∈ I : b1(i1) = w1.• Let i

(0)1 , i

(1)1 , . . . , i(A−1)

1 be the members of the set I−11 (w1), where A = |I−1

1 (w1)|.3 Set H(a)1 = i(a)

1 , a = 0, 1, . . . , A− 1.Main Loop4 for (a = 0, 1, . . . , A− 1) do5 for (n = 1, 2, . . . , N − 1) do6• Given the odeword wn+1 ∈ Wn+1, determine the inverse image I−1

n+1(wn+1).7• Determine the indi es i(a,l)

n+1, i(a,u)n+1 ∈ I

−1n+1(wn+1) below and above the main8 diagonal δ at position i1 = i

(a)1 .

• Form the ross produ t H(a)n+1 = H(a)

n × i(a,l)n+1, i

(a,u)n+1 .9

• If the set H(a)n+1 ontains more than M of index tuples, al ulate for all10 tuples i

(a)1,2,. . . ,n+1 ∈ H

(a)n+1 the diagonal distan e d(i

(a)1,2,. . . ,n+1, δ). Remove allof them but the M index tuples with the smallest distan e to δ.Upon termination, we obtain the index sets H(a)

N for a = 0, 1, . . . , A − 1. Theseindex sets ontain a total of minA2N−1, AM index tuples in the inverse imagethat surround the main diagonal δ. Let H =⋃A−1

a=0 H(a)N be the union all index sets,also denoted as H(w) to highlight its dependen y on w. Using the obtained union

H(w) instead of the inverse image I−1(w) in (3.20), the de oder output i is given byi = argmin

i∈H(w)

d(i, δ). (3.21)Using the proposed approa h, the number of index tuples that need to be onsideredduring de oding is limited to minA2N−1, AM redu ing the omplexity onsiderably,espe ially whenever 2N−1 >> M . Clearly, limiting the number of onsidered indextuples means that some of the potential andidates (for the de oder output) might bedis arded during de oding. This surely ompromises the delity of the derived result,however, our experiments in Se tion 3.9.2 showed that, if M is hosen adequately, thede oding performan e is still more than a eptable.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 953.7 Binning with MemoryBased on our Diophantine index assignments designed for the memory-less ase, wenow show how to onstru t Diophantine odes with memory. In the memory-less ase,the quantization indi es in(t) ∈ I are mapped onto the odewords wn(t) ∈ Wn ina stati way, i.e., independent over the time t. Introdu ing temporal dependen iesbetween the (independent) indi es in(t) while onsidering entire sequen es of indi esi(t)bt=a, it is possible to add exibility to the ode potentially making it more resilientin low- orrelation settings. In parti ular, we shall a hieve this by allowing for moreexibility in the hoi e of the odewords to be sent to the re eiver. The odewords inthis ase, hen eforth denoted as xn(t), then do not only orrespond to a xed, stati olle tion of indi es in(t) but generally allow for a more exible arrangement of them.In other words, the hosen odeword xn(t) does not only depend on the index in(t)but also on the other elements in the sequen e i(t)bt=a. In the following, we des ribehow to use our Diophantine index assignments at the ore of a trellis-based odings heme to improve the ode performan e parti ularly at low- orrelation settings.3.7.1 Trellis Code Constru tionThe work in [PR99 and [PR03 provides a onstru tive framework for the design ofdistributed sour e odes based on Ungerboe k trellises [Ung82. We now further extendthis idea and show how the Diophantine index assignments an be used to identifystru tured, sour e-optimized trellis odes.The used trellises are hara terized by the trellis states mn = 0, 1, . . . , M − 1,where M is the total number of states, the edges en = 0, 1, . . . , E − 1, where E is thenumber of outgoing edges from ea h state, and the edge label an = 0, 1, . . . , A − 1,where A is the number of distin t label.22 We will use the trellises derived in [Ung82as a basis for our ode onstru tions (see Figure 3.18 (a)), parti ularly requiring that Edivides A (without remainder). Furthermore, we assume that the odeword alphabetsfor the trellis oding approa h and the memory-less ase are the same, i.e., we assumethat Xn =Wn. For ea h odeword xn ∈ Xn and ea h label index an, we introdu e thelabel set An(an, xn) whi h ontains all indi es in ∈ I that are mapped onto xn.23 The22At this point is is worth pointing out that the parameters E and A, in a regular hannel oding ontext, orrespond to the alphabet sizes of the ingoing sour e symbols and outgoing hannel symbols,respe tively.23As a onsequen e, and this shall simplify the dis ussion of the de oder later on, this means that,given a re eived odeword xn, we are able draw a trellis ea h of its edges labeled with the indi esin ∈ I mapped onto xn.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 96

Figure 3.18: Trellis-based binning: (a) Ungerboe k trellis with M = 4 states, E = 2 outgoingedges per state, and A = 4 edge label. (b) Diophantine edge label for N = 2, L = 36, K1 = 9,K2 = 6, ∆1 = ⌊K1

2 ⌋ = 4, and ∆2 = 0.label set itself is dened via the auxiliary setBn(sn) = in ∈ I : in = bn · FKn + sn, an = 0, 1, . . . , (3.22)whi h ontains all indi es in ∈ I that are spa ed bn · FKn (positions) apart from thereferen e point sn ∈ I, where bn is an arbitrary integer and F = A

E. For ea h possiblepair of trellis state an and the odeword xn, we then hoose some referen e index snand assign An(an, xn) = Bn(sn). In parti ular, we hoose sn ≡ xn + ⌊an

F⌋Kn + G∆n

(mod L), where the term ⌊an

F⌋Kn ontrols the distribution of the indi es in ∈ I amongall possible states an ∈ 0, 1, . . . , M − 1 and odewords xn ∈ Xn, and the integer

∆n is used together with the intermediate parameter G ≡ an (mod F ) to introdu esome additional shift among the en oders. Jointly onsidering the trellis label for theen oders n = 1, 2, . . . , N , we fo us on the ase where∆n =

⌊K1

F⌋, if n = 1,

0, otherwise, (3.23)leading to an asymmetri ode setting that results in shifted label indi es for the rstsour e only24 (see Figure 3.18 (b)).24This in fa t leads to the ase where the en oders for n = 2, 3, . . . , N do not perform trellis odingin the stri test sense. However, as pointed out later on, this has no severe drawba ks (the jointdistan e properties are maintained) but simplies the de oding task onsiderably.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 97We briey point out some of the properties of the onstru ted trellis ode.Dire tly omparing the trellis-based oding s heme with the s alar index assignments,while assuming that the orresponding odeword alphabets have the same ardinality|Xn| = |Wn|, we observe that generally more indi es in ∈ I an be mapped onto atrellis odeword xn ∈ Xn than onto a s alar odeword wn ∈ Wn. In parti ular, theunion of all indi es in ∈ I previously mapped onto F distin t s alar odewords wn,in a stati fashion, are now dynami ally mapped onto a single trellis odeword xn.This adds to the exibility of the ode whi h eventually an be exploited to makethe ode more error-resilient, espe ially in low- orrelation settings. We observe thatthe indi es in ∈ I ontained in any auxiliary set Bn(sn) and, thus, in any label setAn(an, xn) are spa ed FKn (positions) apart. This has dire t impli ations for thede oder performan e be ause whenever the trellis label an are de oded orre tly, wevirtually deal with a ode having (L, FK1, FK2, . . . , FKN) hara teristi s, as opposedto (L, K1, K2, . . . , KN). In this ase, the greatest ommon divisor g and, thus, therequired segment number C are virtually in reased by the fa tor F . This prin ipallymakes it possible to orre tly de ode the index tuples in an in reased volume V(r) ofradius r = (FC) instead of (C). We now look at the inner workings of the en odersand the de oder in more detail.3.7.2 En oder DesignThe en oders n = 1, 2, . . . , N work as follows. Considering the urrent index in(t)at time instant t, the en oder uses its knowledge about the previous state mn(t − 1)as well as its knowledge about the trellis stru ture and the edge label to identify the urrent edge en(t) labeled with in(t). In parti ular, this is done by hoosing the edgeen(t), orresponding to the label index an(t), su h that in(t) ∈ An(an(t), xn(t)) forsome odeword xn(t).25 Noti e that this sele tion pro ess an be implemented bymeans of simple table lookups. We thus obtain the odeword xn(t) subsequently sentto the de oder. We then tra e the edge en(t) to determine the state mn(t) needed forthe next en oding step, where the index in(t + 1) is en oded, repeating the overallpro edure. (Figure 3.19 (a) shows a s hemati illustration of the en oder; it an bedire tly substituted in Figure 3.1.)25Noti e that for any state mn(t− 1) there is always an outgoing edge en(t) labeled with any of allpossible in ∈ I. This is ensured by the denition of the trellis label presented before.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 98

Figure 3.19: Trellis-based binning and index de oding.3.7.3 De oder DesignThe trellis-based de oder uses the re eived odeword sequen es xn(t + ϑ)T−1ϑ=0 fromall en oders n = 1, 2, . . . , N to jointly determine the de oded index ve tor i(t) = (i1(t),

i2(t), . . . , iN(t))T ∈ I at time instant t. For all time instants t + ϑ, ϑ = 0, 1, . . . , T − 1,for all index label an = 0, 1, . . . , A − 1, and over all en oders n = 1, 2, . . . , N , wedetermine the index sets An(an, xn(t+ϑ)) using the knowledge about the trellis label.Now, onsidering a parti ular ve tor of label indi es a = (a1, a2, . . . , aN )T , and usingthe re eived odeword ve tor x(t + ϑ) = (x1(t + ϑ), x2(t + ϑ), . . . , xN (t + ϑ))T at timeinstant t + ϑ, we form the rossprodu t A(a,x(t + ϑ)) =∏N

n=1An(an, xn(t + ϑ)). Forall of the index ve tors i ∈ A(a,x(t+ϑ)), we al ulate the diagonal distan e a ordingto (3.9) to determine the one leading to the minimum value of the diagonal distan e.26The obtained value for the minimum diagonal distan e is stored and serves as a ostmeasure, hen eforth denoted as the edge ost, for the orresponding trellis edges. Theprevious steps are repeated for all time instants t + ϑ, ϑ = 0, 1, . . . , T − 1, and forall possible realizations of a. Having done so, we basi ally have everything that isneeded to initialize the de oding pro ess over the entire sequen es. However, this is on eptually as well as omputationally not an easy task sin e it would require, in themost general setting, a de oder that jointly onsiders all trellises for n = 1, 2, . . . , N .Mostly due to dependen y issues arising from this joint de oding requirement, wewere not able to resolve how to e iently implement the de oder in the most generalsetting.26This an be implemented in a s alable fashion as presented in Se tion 3.6.4.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 99In order to provide a feasible de oder implementation, we fo us in the followingon the asymmetri trellis ode setting dened in (3.23), onsidering en oder n = 1as referen e. This allows us to simplify the de oding task signi antly, sin e in thissetting only the odewords from the rst en oder are in fa t trellis en oded and thejoint de oding task an be redu ed to standard Viterbi de oding [G. 73 over one trellisonly. In order to initialize the de oding pro ess, we need to determine the edge ostsfor the rst trellis. In parti ular, we have to determine for ea h time instant t + ϑ,ϑ = 0, 1, . . . , T − 1, and for all label indi es a1 = 0, 1, . . . , A − 1, the label indi esa2, a3, . . . , aN leading to the minimum diagonal distan e. Assuming that a1 = a, wesear h over all possible ongurations of a, with a1 = a, to determine the ve tora = b(a1 = a, t+ϑ) that leads to the minimum diagonal distan e. This is repeated forall label indi es a = 0, 1, . . . , A− 1, and for all time instants t + ϑ, ϑ = 0, 1, . . . , T − 1.At ea h step the trellis is labeled and we assign the edge osts a ordingly. AfterViterbi de oding [G. 73, we then obtain a sequen e of de oded edges e1(t + ϑ)T−1

ϑ=0whi h an dire tly be mapped to the de oded label index sequen e a1(t+ϑ)T−1ϑ=0 . Forea h time instant t, we look-up the previously stored label indi es in order to obtainthe de oded label index ve tor a(t) = b(a1(t) = a1(t), t).After onvolution de oding of the label indi es a(t), the indi es i(t) an bere overed in a memory-less fashion for all time instan es t. Given the de oded indexlabel an(t) and the re eived odeword xn(t), we are able to determine by simpletable look-up the set of index label An(an(t), xn(t)) for n = 1, 2, . . . , N and form the ross-produ t A(a(t),x(t)) =

∏Nn=1An(an(t), xn(t)) to determine the orrespondingindex ve tors. Then, after al ulating the diagonal distan e for all index ve tors

i ∈ A(a(t),x(t)) a ording to (3.9), we hoose the one leading to a minimum; allit i(t).27 (Figure 3.19 (b) shows a s hemati illustration of the de oder; it an bedire tly substituted in Figure 3.1.)3.8 Quantization with MemoryAllowing for the quantization of entire sour e symbol sequen es, potentially leadingto an improved system performan e, we now show how to in orporate trellis odedquantization [MF90 within our setup.27Again, this an be implemented in a s alable fashion as presented in Se tion 3.6.4.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 100Figure 3.20: TCQ en oder (a) and TCQ de oder (b).

Figure 3.21: TCQ Example: (a) Used Ungerboe k trellis with M = 4 states. (b) Trellislabel for standard TCQ.3.8.1 Preliminaries - Trellis Coded QuantizationConsidering the quantization of the sour e sample un(t) ∈ R at time instant t, trellis oded quantization (TCQ) [MF90 uses the entire sour e sample sequen e un(t +

θ)Θ−1θ=0 of length Θ for the quantization ( ompare Blo k TCQ in Figure 3.20 (a)).Assuming a targeted transmission rate of Rn = log2(L) [bit, the en oder performs ahigh-rate quantization with a quantizer resolution of 2L levels and hooses the resultinghigh-rate quantization index jn ∈ J = 0, 1, . . . , 2L−1 su h that the distortion for thewhole sequen e (and the used trellis) is minimized using the Viterbi algorithm [G. 73,e.g., see [MF90 for details. We will onsider the same trellises as derived in [Ung82and used in Se tion 3.7 (see Figure 3.21 (a)), parti ularly requiring that E divides A(without remainder)28 whi h is ensured by the setting E = 2 and A = 4. Ea h of thetrellises label an ∈ 0, 1, 2, 3 is asso iated with a subset of the high-rate quantization28Under this assumption, we shall only allow for even values of L, dire tly ensuring that 2L isdivisible by A. For odd values of L, one ould avoid the resulting problems by using a quantizer witha redu ed resolution of L− 1 levels instead.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 101indi es jn ∈ J given by the label set An(an) = jn ∈ J : jn = vn · 4 + an, vn =

0, 1, . . . (see Figure 3.21 (b)). Those trellis label are used when sear hing for thedistortion-optimal sequen e using the Viterbi algorithm. The en oder output at timeinstant t ( ompare Blo k TCQ in Figure 3.20 (a)) is then given by the edge indexen(t) ∈ 0, 1 together with the index vn(t) ∈ 0, 1, . . . , L

2− 1 orresponding to theso- alled un oded bits [MF90. Both indi es together (together with the previous state

mn(t− 1)) an be used to identify the en oded high-rate quantizer index jn(t) ∈ J .The TCQ de oder ( ompare Blo k TCQ−1 in Figure 3.20 (b)) uses the re eivededge index en(t), the re eived index vn(t), and its knowledge about the previous statemn(t− 1) to re over the en oded high-rate quantization index jn(t). This is a hievedby tra ing the trellis path, similar as in a onvolution en oder. The estimate un(t) isthen given by the high-rate quantizer re onstru tion level orresponding to jn(t).3.8.2 System IntegrationThe output of the TCQ en oding stage is given by the edge index en(t) ∈ 0, 1,the index vn(t) ∈ 0, 1, . . . , L

2− 1 (representing the un oded bits), and the previousstate m(t − 1) ∈ 0, 1, 2, 3 whi h an be jointly used to identify the high-rate index

jn ∈ J = 0, 1, . . . , 2L − 1. This representation of the quantizer outputs, however,is unsuitable as an input of the binning stage (with and without memory). This ismainly due to the fa t that the magnitude of the quantizer output must ree t, froma topologi al point of view, the magnitude of the original sour e samples un(t) sin e,otherwise, the ode properties would be destroyed.We over ome this problem by adding a post-pro essing stage after the (standard)TCQ en oder ( ompare Figure 3.20 (a)). The task of the post-pro essing stage is tojointly map the indi es en(t) ∈ 0, 1 and vn(t) ∈ 0, 1, . . . , L2− 1 onto an equivalentrepresentation, the index in(t) ∈ I = 0, 1, . . . , L − 1, su h that ea h of the newlyobtained indi es in(t) ∈ I exa tly orrespond to two subsequent high-rate indi es

jn ∈ J . Considering our parti ular TCQ onguration, we are able to a hieve this bydetermining, as an intermediate result, the urrent label index an(t) using the edgeindex en(t), the previous trellis state mn(t − 1), and the trellis stru ture ( ompareFigure 3.21 (a)). Then, as the se ond and nal step, we use the obtained label indexan(t) together with the index vn(t) (representing the un oded bits) and jointly mapthem onto the desired low-rate quantization index in(t). In parti ular, we assignin(t) = vn(t) · 2 + ⌊an(t)

2⌋.Consequently, we need to add a pre-pro essing stage before the standard TCQde oder ( ompare Figure 3.20 (b)). The task of the pre-pro essing stage is to mapthe re overed low-rate quantization index in(t) ∈ I onto the re overed edge index

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 102en(t) ∈ 0, 1 as well as the re overed index vn(t) ∈ 0, 1, . . . , L

2− 1 (representingthe re overed un oded bits) to allow for standard TCQ de oding. On one hand, thisis a hieved by assigning vn(t) = ⌊ in(t)

2⌋. On the other hand, if the re overed index

in(t) is even, we know that the re overed label index an(t) must be in the set 0, 1,otherwise, an(t) must be in the set 2, 3. Using this knowledge together with thepreviously re overed trellis state mn(t− 1) ∈ 0, 1, 2, 3, we are able to determine there overed edge index e(t) ∈ 0, 1 using the trellis stru ture ( ompare Figure 3.21 (a)).By adding the post- and pre-pro essing stages, it be omes possible to integratethe TCQ into our system without hanging the binning stages both at the en oderside and the de oder side.29 (Consequentely, the TCQ en oding and de oding stagesin Figure 3.20 an be dire tly substituted in our system setup in Figure 3.1.)3.9 System Analysis and Performan e EvaluationIn order to underline the versatility and ee tiveness of our sour e-optimized odingstrategy, we analyze the system performan e for two important lasses of sour emodels. On one hand, we onsider a system setup with dis rete-valued, uniformlydistributed sour es where the sour e are allowed to wrap around their boundaries (seeSe tion 3.3.2). The goal is to en ode, transmitt, and re over the original sour e sym-bols subje t to a zero-error riterion; and identify ases in whi h this an be a hievedindeed. On the other, hand we present numeri al results for a symmetri , multivariateGaussian sour e setup to underline the mean squared error (MSE) performan e ofour odes, omparing the obtained simulation results against fundamental boundsprovided by rate-distortion theory.3.9.1 Dis rete-valued Sour esWe start by onsidering a s enario with N = 2 dis rete-valued sour es, I1 and I2, withrealizations i1, i2 ∈ I, I = 0, 1, . . . , L− 1 (also ompare Figure 3.4). The sour esare orrelated in the sense that their joint output, given by the index ve tor (i1, i2)T ∈

I × I, only has a non-zero probability p(i1, i2), if (i1, i2)T lies within the volume V(r)of radius r (see Se tion 3.3.2), and has a zero probability, otherwise. In parti ular, weassume that the index ve tors (i1, i2)

T within the volume V(r) are equiprobable, and onsider the ase where the radius r orresponds to the maximum supported radius29At this point it is worth pointing out that a trellis oded quantization s heme in luding theadditional post- and pre-pro essing stages an be seen as a renement odeupon re eiving the indexin(t), it is already possible to form a low-rate estimate of un(t); additionally performing onvolution oding, a high-rate estimate an be found, rening the de oding result.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 103(C) for some (positive) integer C al ulated a ording to (3.16). Furthermore, weassume that the indi es wrap-around their boundaries (see Se tion 3.3.2).Slepian and Wolf [SW73 provided the a hievable rates for separate en oding oftwo orrelated, dis rete-valued sour es with asymptoti ally vanishing error probability.Considering the orrelated sour es I1 and I2, the orresponding a hievable rates R1and R2 are then given by

R1 ≥ H(I1|I2)

R2 ≥ H(I2|I1)

R1 + R2 ≥ H(I1, I2).

(3.24)The al ulation of these entropies generally requires full knowledge about the jointprobabilities p(i1, i2). Given the sour e parameters L and C, we ount a number of LCindex ve tors tuples within the volume V(r = (C)), ea h of them having a probabilityof p(i1, i2) = 1LC

(under the equiprobability assumption above). Given the properties ofour sour e model, this also implies that the indi es i1 and i2 are uniformly distributedsu h that p(i1) = 1Land p(i2) = 1

L. As a onsequen e: H(I1) = H(I2) = log2(L),

H(I1|I2) = H(I2|I1) = log2(C), and H(I1, I2) = log2(LC).Using the de odability results established in Se tion 3.5.3, we observe that the in-dex ve tors within the volume V(r = (C)) are de odable with zero-error,if L ≤ lcm(K1, K2) and C ≤ gcd(K1, K2). Without restraining the possibility for other ode ongurations, we assume the usage of a anoni al ode (see Se tion 3.5.3) withK1 = L and K2 = C, where L = lcm(K1, K2) and C = gcd(K1, K2). Consequently,we obtain the rates R1 = log2(L) and R2 = log2(C), satisfying the Slepian-Wolf onditions in (3.24) with equality (for all but one inequality). Therefore, under theabove assumptions, we on lude that the proposed odes are optimal.For odes satisfying those properties, zero-error de oding an be a hieved, pro-vided that an optimal de oder is used. This would be a de oder, that dire tlyoperates on the admissible set V(r = (C)) as des ribed in Se tion 3.3.4 whi h, underthe onsidered sour e model, be omes equivalent to maximum a-posteriori (MAP)de oding.30 Noti e that for odd values of C even the less omplex diagonal distan ede oder presented in Se tion 3.6.3 an be used for de oding without ompromisingoptimality.31We now turn our attention to the ase of N > 2 sour es, I1, I2, . . . , IN , withrealizations olle ted in the index ve tor i = (i1, i2, . . . , iN)T ∈ IN . Sin e in this aseit is harder to make statements on erning the optimality of the ode, we shall fo us30Memory-less de oding is su ient to ensure the optimality of the obtained de oding result.31For even values of C this is generally not the ase, and an optimal de oder (or a modied version ofthe diagonal distan e de oder) is required. This is mainly be ause in his ase the volume V(r = (C)) an never be symmetri ally ollo ated around the main diagonal δ.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 104on the ase where the joint probability p(i) an be expressed by a fa torization. Inparti ular, we onsider a fa torization of the formp(i) =

N∏

k=1

p(iak|ibk

), (3.25)where the subs ripts ak, bk ∈ 1, 2, . . . , N are used to identify the sour e indi esin the fa tors k = 1, 2, . . . , N and the following onditions hold: (a) The indi es iakand ial, k, l ∈ N , k 6= l, are disjoint, i.e., ak ∩ al = ∅. (b) The indi es ibkare (statisti ally) onne ted, i.e., bk ∈

⋃k−1l=1 al and b1 = ∅. ( ) The indi es iak

are(statisti ally) onne ted, i.e., ⋃Nk=1ak = 1, 2, . . . , N. We assume that ea h ve torof index pairs (iak

, ibk)T has a joint probability p(iak

, ibk) = p(iak

|ibk) ·p(ibk

) distributedin the same way as onsidered for N = 2; for simpli ity, we furthermore assume thatthe volume V(r) is of onstant radius r = (C, 2) al ulated a ording to (3.16) for allsu h pairs and some given (positive) integer C.For the sour e fa torization in (3.25) the generalized Slepian-Wolf onditions forN > 2 simplify as follows:

Rak≥ H(Iak

|Ibk), k = 1, 2, . . . , N

∑N

n=1 Rn ≥ H(I1, I2, . . . , IN).(3.26)Under the equiprobability assumption, we obtain p(iak

, ibk) = 1

LCand p(iak

) = p(ibk) =

1Lfor all fa tors in (3.25). We obtain the entropies: H(Ia1) = log2(L), H(Iak

|Ibk) =

log2(C), k = 2, 3, . . . , N , and H(I1, I2, . . . , IN) = H(Ia1)+∑N

k=2 H(Iak|Ibk

) = log2((N−

1) · LC).Again, without restraining the possibility for other ode ongurations, we as-sume the usage of a anoni al ode (see Se tion 3.5.3) with Ka1 = L and Kak= C,

L = lcm(K1, K2, . . . , KN), C = gcd(K1, K2, . . . , KN), k = 2, 3, . . . , N . Consequentely,we obtain the rates Ra1 = log2(L) and Rak= log2(C), k = 2, 3, . . . , N , satisfying theSlepian-Wolf onditions in (3.24) with equality (for all but one inequality). Given theabove assumptions, we on lude that the onsidered odes are optimal.Following the previous observations, the proposed ode ensures that all indexve tors i ∈ IN generated by the sour e (i.e., all index ve tors with p(i) > 0) arede odable with zero-error. Provided that an optimal de oder is used (e.g., a de oderdire tly operating on the admissible set, or a de oder based on MAP de oding) thisis also a hieved. We furthermore observe that, even for odd values of C, the diagonaldistan e de oder only allows us to orre tly de ode the index ve tors within theredu ed volume V(r) of radius r = (C, 2) ≤ (C, N), i.e., the admissible volumehas the same radius as obtained for the ase of N = 2.3232De oding an be performed in a pairwise, su essive fashion. For example: Using sour e a1 as

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 1053.9.2 Multivariate Gaussian CaseWe now present numeri al results for a multivariate Gaussian sour e setup. We onsider N ontinuous-valued sour es olle ted in the ve tor U = (U1, U2, . . . , UN)Twith realizations u = (u1 u2, · · · , uN)T ∈ RN . The sour e samples are Gaussiandistributed with mean µ = EU and ovarian e matrixΣ = EUUT−EUEUTsu h that their joint probability is given by

p(u) =1

√

(2π)N |Σ|exp

(

−1

2(u− µ)TΣ−1(u− µ)

)

. (3.27)In parti ular, in order to ompare our results with theoreti al bounds, we fo us on asymmetri s enario with the ovarian e matrixΣ =

1 ρ · · · ρ

ρ 1 · · · ρ... ... . . . ...ρ ρ · · · 1

N×N

, (3.28)and the all-zero ve tor of mean values µ = 0TN = (0, . . . , 0)T .The optimality riterion of interest is the mean squared error (MSE) E||U−U||2between the de oder outputs U = (U1, U2, . . . , UN)T and sour e samples U. Thesystem's MSE

E||U−U||2 = E(U−U)T · (U−U) = EN∑

n=1

(Un − Un)2 =N∑

n=1

E(Un − Un)2(3.29)is evaluated by approximating E(Un−Un)2 via numeri al simulation. In parti ular,we generate t = 0, 1, . . . , T−1 sour e samples un(t) ∈ R for the sour es n = 1, 2, . . . , N ,and simulate the en oding and de oding pro edure to obtain the orresponding esti-mates un(t) ∈ R. Thus, we are able to approximate E(Un − Un)2 by the sampleaverage ∑T−1t=0 (un(t)− un(t))2, n = 1, 2, . . . , N .The obtained results are ompared to theoreti al bounds known from rate-distortion theory. Under above requirements, and fo using on a symmetri s enariowhere the target distortions E(Un−Un)2 for all sour es n = 1, 2, . . . , N are equal tosome onstant D, the work in [YX08 (whi h itself is based on [WTV08 and [WTV06)provides us with the theoreti al bounds. Given the number of sour es N , the orre-lation oe ient ρ, and the target distortion D, the theoreti ally a hievable sum-rate

RΣ =∑N

n=1 Rn an be al ulated using the result in [YX08.a referen e, it is possible to de ode sour e a2, in a memory-less fashion, as des ribed for the ase ofN = 2. Then, using the obtained result, the sour e a3 an be de oded. This is repeated, until allsour es are de oded.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 106The simulations were implemented in Matlab, and ea h data-point is determinedby simulating NT = 1000000 sour e samples. Assuming the distribution of the simu-lated distortions is approximately Gaussian, the onden e intervals an be al ulated.In parti ular, we provide for ea h data-point the 95% onden e intervals. The resultsare measured by the output signal-to-noise ratio (SNR) al ulated as followsSNRout = 10 · log10

(

E||U||2

E||U−U||2

)

[dB], (3.30)where E||U||2 = N . A ordingly, the theoreti al rate-distortion (R-D) performan eis given by SNRR-D = 10 · log10

(1D

) [dB. Sin e we want to evaluate the system perfor-man e for a large range of possible orrelation settings, we introdu e the orrelationSNR SNR orr = 10 · log10

(1

1− ρ

)

[dB], (3.31)where a orrelation SNR orr of 0 dB orresponds to the ase of un orrelated sour es.Throughout the following subse tions, we shall evaluate the system performan e forthe memory-less binning approa h presented in Se tion 3.6, as well as the trellis-based odes presented in Se tion 3.7 employing a M = 8 state Ungerboe k trellis. For both ases, we onsider s alar quantization as well as trellis oded quantization des ribedin Se tion 3.8 based on a M = 32 state Ungerboe k trellis. The de oders are basedon the s alable de oder presented in Se tion 3.6.4.3.9.2.1 Comparison of Quantization and Binning S hemesWhen designing a distributed sour e ode it is important to know how well dierentquantization and distributed sour e oding approa hes perform when dire tly om-pared to ea h other. In the following, we ompare the rate-distortion performan efor uniform and Lloyd-Max (L-M) optimized rst-stage quantizers, for memory-lessbinning s hemes and binning with memory, as well as for s alar quantization (S alarQ) and trellis oded quantization (TCQ).Mainly on erned about omparing the dierent approa hes in terms of theirrate-distortion performan e, we measure the ode performan e by its gap to thetheoreti al rate-distortion bound; this gap is dened as ∆SNRout = SNRR-D− SNRout[dB. We onsider the ase of N = 2 sour es. Two prin ipal sour e settings are onsidered. On one hand, we onsider the ase of L = 42 and C = 7, leading toalphabet sizes K1 = 21 and K2 = 14 (see Se tion 3.5.4). On the other hand, we onsider the ase of L = 48 and C = 8, leading to alphabet sizes K1 = 24 and

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 107

Figure 3.22: Rate-distortion performan e for N = 2: Uniform and Lloyd-Max (L-M)optimized rst-stage quantizers are onsidered. S alar Quantization (S alar Q) is onsideredtogether with trellis oded quantization (TCQ). Memory-less binning is ompared to binningwith memory.K2 = 16. Figure 3.22 shows the obtained simulation results; it shows ∆SNRout versusSNR orr for the onsidered settings. The verti al bars indi ate the 95% onden eintervals.

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 108We observe the following: (a) binning with memory performs loser to thetheoreti al rate-distortion limit (at ∆SNRout = 0 dB) than the memory-less approa h;(b) hoosing TCQ over s alar quantization generally improves the performan e towardslarger values of SNR orr (strong orrelation); ( ) for a xed C, binning with memoryperforms better towards smaller values of SNR orr (weak orrelation) than memory-less binning; (d) Lloyd-Max based rst-stage quantization usually a hieves better rate-distortion performan e when ompared to uniform quantization. However, for a given ode setting, uniform quantization might work better towards smaller values of SNR orr(weak orrelation).The main on lusions drawn from this observations are: (a) depending on whethermemory-less binning, binning with memory, s alar quantization, or TCQ are used,exible trade-os between the rate-distortion performan e and the inherent en oderand de oder omplexities are possible; (b) the usage of Lloyd-Max optimized rst-stagequantizers (as opposed to uniform ones) is preferable sin e it yields better performan eat the same operational omplexity. This surely was expe ted for the s alar ase,but also proved true for TCQ where the rst-stage quantizer only partly ae ts theoverall quantizer performan e. In a ordan e with this observations, we shall restri tourselves to the ase of Lloyd-Max optimized rst-stage quantizers in the remainderof this se tion.3.9.2.2 Impa t of Code ParametersWe now take a loser look at the impa t of dierent ode parameters onto the systemperforman e. We onsider the ase of N = 2 sour es.For our rst evaluation, we keep the sour e parameters L and C onstant andallow for dierent realizations of the alphabet sizes K1 and K2. The alphabet sizes K1and K2 are sele ted by hoosing the parameters r1, r2, and C among the prime fa torsof L, su h that K1 = r1C and K2 = r2C (see Se tion 3.5.4). We obtain r1r2 = LC,where r1 and r2 an be freely hosen among the prime fa tors of L

C. For both sour esettings under onsideration, i.e., for L = 42 and C = 7, as well as L = 48 and C = 8,we obtain r1r2 = 6. Thus, we are able to hoose either r1 = 6 and r2 = 1, resulting inthe anoni al ode setting, or r1 = 3 and r2 = 2. Consequentely, for the sour e setting

L = 42 and C = 7, we either obtain the alphabet sizes K1 = 42 and K2 = 7, or thealphabet sizes K1 = 21 and K2 = 14; and for the sour e setting L = 48 and C = 8,we either the alphabet sizes K1 = 48 and K2 = 8, or the alphabet sizes K1 = 24 andK2 = 16. Figure 3.23 shows the obtained SNRout versus SNR orr.We note the following: (a) If r1r2 is kept onstant (equal to L

C), and the samebinning on ept is used (memory-less binning versus binning with memory), then the

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 109

Figure 3.23: System performan e for N = 2: The ratio LC

= r1r2 is kept onstant. Theimpa t of the (sour e) parameter C and the impa t of possible alphabet size ongurationsK1 = r1C and K2 = r2C is onsidered.maximum rate-distortion performan e is a hieved at approximately the same value ofSNRout. This is expe ted sin e the value of L

Cdire tly ree ts the sour e orrelation'smagnitude. (b) In reasing C, for onstant r1r2, also in reases SNRout. This is expe tedsin e L = r1r2C and SNRout dire tly depends on L. For the ode design this meansthat given the produ t r1r2, the value of C an be hosen to meet the required targetSNRout. ( ) The anoni al ode setting with r1 = 6 and r2 = 1 shows a similar,although slightly better, rate-distortion performan e as ompared to the non- anoni al ode with r1 = 3 and r2 = 2. The fa t that the anoni al ode performs slightlybetter is not ompletely unexpe ted sin e the de oding pro ess for the sour es (not

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 110

Figure 3.24: System performan e for N = 2: The sour e parameter C is kept onstant and a anoni al sour e setting of K1 = r1C = L, r1 = LC, and K2 = r2C = C, r2 = 1 is onsidered.The ratio L

C= r1r2 is varied to adjust the ode for dierent orrelation settings. onsidering the quantization stage here) is a joint pro ess, i.e., information re eivedfrom one sour e is used to de ode the other, and vi e versa. In the anoni al odesetting however, one sour e is already known, i.e., there is no un ertainty about thissour e whi h also makes de oding of the other more robust. (d) The non- anoni al

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 111

Figure 3.25: System performan e for an in reasing number of sour es N = 2, 3, 4, 8:Memory-less binning and a anoni al ode onguration are onsidered. The de oder worksas des ribed in Se tion 3.6.4. ode shows a steeper drop-o of SNRout while approa hing smaller values of SNR orr.This phenomena, although not relevant for the onsidered distributed sour e odings enario, might be of some interest for other appli ations, e.g., for appli ations in dualproblem settings as dis ussed later in this work.For the se ond ode evaluation, we assume a onstant value of C = 8 and varythe setting of r1 and r2 to realize dierent ompression settings. Noti e that sin eL = r1r2C, K1 = r1C, and K2 = r2C, the settings of L, K1, and K2 are also varied.In parti ular, we on entrate on a anoni al sour e setting where r1 = L

Cand r2 = 1.Figure 3.24 shows the obtained SNRout versus SNR orr. We note that the produ t of

r1r2 an be hosen, very exible, to a hieve good rate-distortion performan e for abroad range of possible SNR orr's.3.9.2.3 S aling with NWe now evaluate the system performan e for an in reasing number of sour es N =

2, 3, 4, 8. Sin e the de oder omplexity already be omes prohibitive for N > 2 inthe ase of binning with memory, we fo us on the memory-less ase and assume theusage of the s alable de oder des ribed in Se tion 3.6.4. We assume the usage of a anoni al ode (see Se tion 3.5.4) with K1 = L and Kn = C, n = 1, 2, . . . , N . Similaras before, we onsider the sour e parameters L = 48 and C = 8. Figure 3.25 showsthe obtained ∆SNRout versus SNR orr. We observe that the anoni al odes togetherwith the s alable de oder are apable of exploiting the sour e orrelations, even in a

CHAPTER 3. DIOPHANTINE DISTRIBUTED SOURCE CODING 112higher-dimensional s enarios, despite their low design and de oding omplexity.3.10 Con lusionsWe presented a onstru tive framework for the design of sour e-optimized distributedsour e odes. Using insights from fundamental number theory, we proposed a new lass of memory-less index assignments that an be tailored to exploit (stru tural)properties ommon to many important sour e models. Using Diophantine equationsas a powerful design tool, we were able to formulate algorithms for a sour e-optimized ode design whi h generally work for more than two sour es. We showed how thememory-less index assignments an be used at the ore of trellis-based distributedsour e odes. Thus, we were able to obtain odes for short as well as long blo k lengths enarios, allowing a exible trade-o between system performan e and omplexity. Inorder to ountera t omplexity issues arising at the de oder, we proposed a s alablede oding solution to keep the omputational omplexity low. We underlined thee ien y and versatility of our odes based on a theoreti al as well as numeri alperforman e evaluation. The numeri al results were ompared with fundamentalbounds provided by rate-distortion theory, and we showed that the proposed odesare apable of a hieving a good end-to-end performan e for a wide range of possible orrelation setups and generally more than two sour es.Interestingly, the proposed anoni al ode onstru tions proved to be very ee -tive, despite the fa t that they are simple in design and have additional advantagesfor an e ient de oder implementation. This also highlights that the hoi e of thesour e parameters and, in parti ular, their number-theoreti al properties stronglyae t (a tually, even predetermine) the resulting ode properties as well as the numberof possible ode ongurations.

Chapter 4Joint Sour e-Network CodingIn the previous hapters we addressed the problem of nding odes for a distributedsour e oding s enario where orrelated data from several sour es has to be sent toa single sink only. Fo using on the data ompression aspe t of the problem, we didnot expli itly address the data transmission itselfwe simply assumed that the dataemitted at the sour e nodes is also delivered at the sink nodeand did not take intoa ount that the data is usually ommuni ated over a multi-hop network till it rea hesits destination. In this hapter we address the general s enario where orrelated datafrom several sour es has to be ommuni ated over a multi-hop network to more thanone sink. In su h s enarios, from an information theoreti point of view, joint sour e-network oding solutions are generally required for an optimal data dissemination.However, as we shall see, this omes at the ost of using an optimal de oder whi his hard to implement and operate due to its inherent omplexity. Motivated by thisobservation, we present a feasible solution for joint sour e-network oding subje t toa delity riterion su h as the error probability or the mean squared error distortion.Primarily on erned about providing a pra ti al de oding solution, we show how thefa tor graphs and the s alable de oding prin iples presented in Chapter 2 an beextended and applied to the problem under onsideration here. In parti ular, we showhow to probabilisti ally represent the overall system (in luding the pa kets' pathes) bya fa tor graph on whi h the iterative sum-produ t algorithm an be run for de oding.Furthermore, we briey address the problem on how pra ti al odes for joint sour e-network oding might be onstru ted onsidering a data gathering s enario and thesour e models introdu ed in Chapter 3. A proof-of- on ept and a omplexity analysisunderline the ee tiveness of our fa tor graph based solutions.113

CHAPTER 4. JOINT SOURCE-NETWORK CODING 1144.1 Introdu tionDistributed ompression and ooperative forwarding are key towards improving theperforman e of a wide range of networked systems in luding sensor networks, peer-to-peer streaming, and intera tive media systems over wireless networks. For example,when sensor nodes are monitoring a onned area, their measurements are often orrelated, whi h explains why distributed sour e oding te hniques [SW73[XLC04are able to redu e the amount of data to be sent. On the transmission side, network oding [ACLY00 has proven to be key towards maximizing the throughput in networkswith one or more data sour es and multiple sinks. Although distributed sour e odingand network oding an be arried out separately in a modular ar hite ture, it is notdi ult to show that this approa h is suboptimal in general networks with orrelatedsour es [RJCE06. The obvious solution from an information-theoreti point of view isto opt for joint sour e-network oding. In pra ti e, however, this is a rather hallengingproposition mostly be ause of the in urred omplexity [CME05, primarily on thede oder side.In Chapter 2, onsidering a distributed sour e oding s enario, we showed how toover ome omplexity issues at the de oder side by adopting a de oding strategy basedon fa tor graphs similar as proposed in [BT06 for a multiterminal sour e- hannel oding problem. In this hapter we take this idea one step further and provide apra ti able solution for joint sour e-network oding. We assume that the topology ofthe network is known and that the sour e statisti s are represented by an appropriatefa tor graph as addressed in Chapter 2. The main goal is to nd a probabilisti modelthat not only in ludes the statisti al dependen ies intrinsi to the sour e data butalso allows us to take into a ount the network oding operations that o ur as thenodes ooperate to send the data a ross the network. The ombination of these two omponents lead to a global fa tor graph used for an adequate and e ient de oderimplementation. The main ontributions of this paper an thus be summarized asfollows.• De oder Design and De oding Algorithm: We show how to implement ade oder with reasonable omplexity building on a fa tor graph representation ofthe overall system.• Code Design: We propose a lass of joint sour e-network odes that are suitablefor the onsidered data gathering problem.• Proof-Of-Con ept: We provide a proof-of- on ept in an form of a workingde oder implementation. In parti ular, we onsider the the joint sour e-network

CHAPTER 4. JOINT SOURCE-NETWORK CODING 115s enario presented in [RJCE06, providing an example where separation generallydoes not hold, as well as a data gathering s enario with a total of 30 sensor nodesrandomly distributed in an unit area element.This hapter is organized as follows. Related work is dis ussed in Se tion 4.2. Se -tion 4.3 des ribes the system model and the problem under onsideration. The fa torgraph de oder design and the iterative de oding algorithm are des ribed in Se tion 4.4.Se tion 4.5 des ribes a possible strategy for the design of joint sour e-network odes.Se tion 4.6 oers a proof-of- on ept followed by the on lusions in Se tion 4.7.4.2 Related WorkStarting with the problem of distributed sour e oding, Slepian and Wolf hara terizedin their landmark paper [SW73 the a hievable rates for the ase where (two) orrelatedsour es are to be en oded independently and ommuni ated (over perfe t hannels)to a single sink. Csiszar showed in [Csi82 that linear odes are su ient wheneither (non-universal) maximum a-posteriori (MAP) de oders or (universal) minimumentropy (ME) de oders are used at the sink. Subsequent resear h in this area yieldedpra ti al en oding and de oding solutions (mostly for a small number of sour es), e.g.,see [XLC04 and referen es therein.Ahlswede et al. onsidered in [ACLY00 the problem of ommuni ating (un or-related) sour es over a network to more than one sink and showed that the maximumrate supported by the network (i.e., the maximum throughput) an be a hieved byperforming network oding. Koetter and Medard presented in [KM03 an algebrai framework for network oding based on linear odes giving rise to pra ti al implemen-tations.For joint sour e-network oding s enarios where orrelated sour es have to be ommuni ated over a network the a hievable rates have been derived by Song andYeung in [SY01. Ho et al. showed in [HMEK04 that linear odes are su ient toa hieve those rates when either MAP or ME de oders are used at the sink. Althoughsome attempts have been made by Coleman et al. [CME05 to redu e the omplexityof ME de oders for in reasing blo k lengths, the omplexity of both de oder types(MAP and ME) is generally not tra table for a large number of en oders. Therefore,Ramamoorthy et al. [RJCE06 asked the question on whether the joint sour e-network oding problem an be separated and showed that this, in general, is not the ase. Insummary this means that it is su ient to use linear odes at the en oder side butalso that we have to deal with high omplexity at the de oder side.First attempts to provide pra ti al oding solutions for joint sour e-network

CHAPTER 4. JOINT SOURCE-NETWORK CODING 116 oding problems were made, e.g., by Wu et al. in [WSXK09. Sin e those approa hesare mostly of a sub-optimal nature and only work for a small number of en oders, welook at the problem from a dierent perspe tive, building on previous work on jointsour e- hannel oding by Barros and Tue hler [BT06, oering a trade-o between the omplexity and the end-to-end performan e.Given an arbitrary network the problem of onstru ting an overlay network forthe ost-e ient data delivery in network oding s enarios was addressed by Lun etal. [LRM+06. This work was later extended by Lee et al. [LMH+07 for joint sour e-network oding s enarios with two orrelated sour es.Despite these important ontributions towards a hieving a general de odingsolution, there is still no pra ti al de oding solution that works for arbitrary jointsour e-network oding s enarios and a large number of nodes.4.3 System Setup and Problem FormulationIn the remainder of this se tion we shall follow the same notational onventions asintrodu ed in Se tion 2.2 and Se tion 3.3.4.3.1 Network Topology and Sour e-Terminal CongurationWe onsider a network represented by the dire ted graph G = V, E ( ompareFigure 4.1). The set V = vn : n ∈ N is the set of verti es vn within the network,also alled the network nodes, whi h are uniquely identied by the indi es n ∈ N . Theset E is the set of dire ted edges ek,l = (vk, vl) in the network, with k ∈ N identifyingthe parent verti es vk and l ∈ N , l 6= k, identifying the hild verti es vl.The set S ⊆ N identies the sour es within the network. Ea h sour e node vs,s ∈ S, has a ess to the sour e symbol ve tor is. Similarly, the set T ⊆ N identiesthe sinks (terminals) within the network. Ea h sink node vt, t ∈ T , wants to re overthe sour e symbol ve tors is spe ied by S. The re overed sour e symbol ve tor is isdenoted as is, s ∈ S. (Figure 4.1 shows a simple example network with S = 1, 2, 3and T =6, 7 illustrating the used notation.)4.3.2 Sour e Model and Fa torizationEa h sour e s ∈ S observes a length-L ve tor is = (is,1, is,2, . . . , is,L)T ∈ Is of (dis rete-valued or ontinuous) i.i.d. sour e symbols is,l with probability distribution p(is,l) =

p(is), l = 1, 2, . . . , L, thus, p(is) =∏L

l=1 p(is,l). The joint output of all sour es s ∈ Sshall be shortly denoted as iS ∈ IS , where IS =∏

s∈S Is. As dis ussed later, it

CHAPTER 4. JOINT SOURCE-NETWORK CODING 117

Figure 4.1: Exemplary network with three sour e nodes v1, v2, v3 and two sinks v6,v7. The sour e symbols i1 are assumed to be independent of i2 and i3. As dis ussedin [RJCE06, separation does not hold in general here.shall turn out useful if p(iS) an be expressed (or approximated) by a produ t withfa tors p(iA|iB), A ⊆ S,B ⊆ S, where p(iA|iB) =

∏L

l=1 p(iA,l|iB,l). (Figure 4.1 showsan example where p(i1, i2, i3) fa tors into p(i1) · p(i2, i3).)4.3.3 System SetupThe following system omponents are parti ularly relevant (see Figure 4.2).Network node: We assume that the nodes within the network generally havefull sour e-network oding (SNC) apabilities. Ea h node n ∈ N might have a sour einput in ∈ In (i.e., if n ∈ S); it might have additional inputs, re eived pa kets, yl ∈ Ylfrom other nodes l ∈ L, l 6= n; and it might have outputs, emitted or sent pa kets,xk ∈ Xk to other nodes k ∈ K, k 6=n. Considering the most general setting, we allowthat the oding fun tion for ea h output k ∈ K, k 6= n, an be hosen independentlyof the other outputs. The oding fun tion γn,k at node n with output (node) k is thengiven by the (deterministi ) mapping γn,k : In × YL → Xk. One major hallenge isto nd appropriate oding fun tions that make it possible to ompress the informationin a ooperative fashion subje t to end-to-end performan e requirements.Sour e node (en oder): In the spe ial ase where the en oder at sour e nodes ∈ S has only a single sour e input in ∈ In and a single odeword output xk ∈ Xk (i.e.,whenever the node only emits pa kets), the en oding fun tion is given by γs : Is → Xs.Sink node (de oder): In the spe ial ase where the sink node t ∈ T has nosour e input (i.e., whenever t /∈ S), the goal is to re over the sour e symbol ve tor

CHAPTER 4. JOINT SOURCE-NETWORK CODING 118

Figure 4.2: Considered system model. Simplisti ase where the sour e and sink nodesonly have a single input and a single output.is ∈ Is for all sour es s ∈ S. The de oder uses the re eived hannel output ve torsy(mt) ∈ Y (mt) indexed by the pa ket index mt = 1, 2, . . . , Mt

1 (i.e., the re eived pa kets)together with the a-priori knowledge about the sour e statisti s p(iS) in order toprodu e the estimates is ∈ Is of all sour es s ∈ S. The de oding fun tion is then givenby φt :∏Mt

mt=1 Y(mt) → IS .2 Fo using on error probability as gure of merit3 themaximum a-posteriori (MAP) de oder be omes optimal [Poo94 hoosing the estimatesas follows

is = argmaxi∈Is

p(is = i|y(1),y(2), . . . ,y(Mt)). (4.1)The omplexity of this de oder turns out to be the main obsta le that needs to be over- ome when sear hing for pra ti al solutions for joint sour e-network oding [CME05.4.3.4 Pa ket SubgraphsAssuming that the re eived pa ket mt ∈ 1, 2, . . . , Mt at sink node t ∈ T is a fun tionof the emitted pa kets at the subset of sour e nodes S(mt)t ⊆ S, we introdu e the(dire ted) pa ket subgraph G(mt)

t des ribing how the emitted pa kets are ommuni ated(and oded together) as they traverse the (dire ted) graph G on their way from thesour es to the sink. In this work, we shall only onsider the ase where the pa ketsubgraph G(mt)t is a dire ted a- y li graph (i.e., it does not ontain any dire ted y les). In this ase, the pa ket subgraph G(mt)

t = E (mt)t ,V(mt)

t for pa ket mt is fullydes ribed by the olle tion of all edges E (mt)t ⊆ E and all verti es V(mt)

t ⊆ V traversed1Sin e the pa ket index mt an be used to uniquely identify ea h pa ket y(mt)L

at sink node t, nomatter from whi h set of nodes L they originate, the subs ript shall be dropped in the following.2The ase where the sink node also has a sour e input is, for brevity, not dis ussed separatelyhere.3Whenever the mean squared error (MSE) is the delity riterion of interest, we hoose a de oderbased on onditional mean estimation (CME). This ase is not dis ussed separately here.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 119while ommuni ating the (involved) pa kets from the sour es S(mt)t ⊆ S to the sink

t ∈ T .At this early stage of the work, we assume for all sink nodes t ∈ T and all re eivedmessages mt = 1, 2, . . . , Mt full knowledge about the orresponding pa ked subgraphsG(mt)

t . For the ase of two orelated sour es those subgraphs an be onstru ted, e.g.,by employing the onstru tion method des ribed in [LMH+07 (whi h itself buildson [LRM+06). For the ase of more than two sour es a ombination of [LMH+07with a sour e-optimized lustering strategy was proposed in [CMB11b, CMB11a.However, this is not part of this work. Noti e that these subgraph onstru tionmethods generally provide solutions that are dire ted a y li graphs, as required above,or an be set up to do so [LRM+06.4.3.5 Problem StatementUnder the system model des ribed above (see Figure 4.2), the main goal is to providealgorithms that yield feasible en oders and de oders for joint sour e-network odings enarios. As pointed out, the de oder for these s enarios is usually hara terized byits high omplexity [CME05. Extending the work in [BT06, we intend to use fa torgraph models to represent our joint sour e-network oding s enarios. The systemend-to-end performan e shall be evaluated based on a working prototype.4.4 Joint Sour e-Network De oder DesignThe main problem when putting joint sour e-network oding into pra ti e is, aspointed out before, that the omplexity of the required optimal de oders is generallyprohibitive. In parti ular, we shall onsider maximum a-posteriori (MAP) de odersand de oders based on onditional mean estimation (CME). The key to over ome this omplexity issue is to a tively exploit the topology of the network (in luding the sour emodel) for an e ient de oder implementation. In the following, we des ribe how this an be a omplished using a probabilisti fa tor graph model of the overall system.4.4.1 Graphi al De oding ModelThe de oder proposed in this work requires a probabilisti representation of the system omponents (i.e., of the sour e model, the nodes, and the edges). In parti ular, weshall represent the system omponents via a graphi al model that onsists of a setof variable nodes, representing the random variables within the system, and a set ofinter onne ting fun tion nodes, representing the probabilisti dependen ies between

CHAPTER 4. JOINT SOURCE-NETWORK CODING 120

Figure 4.3: Probabilisti model of the system omponents represented graphi ally viainter onne ted variable and fun tion nodes.the orrelated random variables. (Figure 4.3 shows the graphi al representations ofthe system omponents dis ussed.)The network nodes nodes, whi h are (generally) assumed to have full sour e-network oding (SNC) apabilities, are des ribed by the (deterministi ) mappingγn,k : In × YL → Xk. This mapping an be des ribed in a probabilisti fashionby hoosing the transition probabilities p(xk|in,yL) in 0, 1 (see Figure 4.3).In ontrast to the most simplisti ase, where the edges ek,l deliver the output xlof node k perfe tly as input yk to node l, we allow that the edges represent a dis retememoryless hannel (Xl, p(yk|xl),Yk) with input alphabet Xl, output alphabet Yk, andtransition probabilities p(yk|xl). (The edges ek,l are represented by the fun tion nodesfl|k in Figure 4.3.)Using the probabilisti representations of the network omponents (i.e., of thenetwork nodes and of the edges), it be omes possible to onstru t (by simple substitu-tion) a graphi al model of the overall system (in luding a model of the pa kets' patheswithin the network). This is also true the subgraphs G(mt)

t = V(mt)t , E (mt)

t (whi hdes ribes how the data in pa ket mt ∈ 1, 2, . . . , Mt is ommuni ated from the sour esS(mt)

t to the sink t ∈ T ). In parti ular, using the probabilisti representations of thenetwork omponents des ribed above, it be omes possible to onstru t a probabilisti model of the pa kets' pathes by a su essive substitution of the verti es V(mt)t and

CHAPTER 4. JOINT SOURCE-NETWORK CODING 121

full fullsimple

y(1)1

⇒⇒

simple

⇒

full simple

m6 = 1 m6 = 2

v(3)1

v(3)6

v(3)2

v(3)5

v(3)4

y(3)5

i(3)1 i

(3)2

m6 = 3

v(1)1

v(1)6

i(1)1

x(1)6

y(1)1

i(1)1

i(2)3

y(2)3

i(2)3

x(2)6

y(2)3

v(2)3

v(2)6

i(3)2

y(3)1,2

i(3)1

f(1)6|1

f(1)6|1

f(2)6|3

f(2)6|3

f(3)6|5

f(3)5|4

f(3)4|1

x(3)4

x(3)4

f(3)4|2

f(3)6|1,2

Figure 4.4: Probabilisti modelling - Pa ket model: Using the re eived pa kets m6 =

1, 2, 3 at sink node v6 the full and simplied probabilisti model of the orrespondingpa ket paths an be onstru ted.the edges E (mt)t of the graph G(mt)

t = V(mt)t , E (mt)

t by their probabilisti model. Theobtained representation shall in the following also be referred to as the pa ket model.(Figure 4.4 shows the pa ket models for sink node v6 and m6 = 1, 2, 3 as obtained forthe exemplary s enario in Figure 4.1.)Usually the probabilisti model an be simplied onsiderably. Consideringe.g. the graphi al model, the on atenation of several fun tion and variable nodesmight be repla ed by a single, equivalent, fun tion node. The fun tion nodes in thissimplied model shall be distinguished, in terms of notation, by the usage of a tildeand updated subs ripts, e.g., f(mt)L|K , and the variable nodes by updated subs ripts, e.g.,

y(mt)L . (Figure 4.4 shows the simplied pa ket models for sink node v6 and m6 = 1, 2, 3as obtained for the exemplary s enario in Figure 4.1.)The sour e model represents the statisti al dependen ies p(iS) among the sour eoutput ve tors iS , S ⊆ N , i.e., among all sour e output ve tors is, s ∈ S, and itsunderlying fa torization. The sour e outputs is, s ∈ S, are then simply representedby variable nodes and the onditional probabilities p(iA|iB), A ⊆ S, B ⊆ S, in thefa torization are represented by fun tion nodes pA|B (see Figure 4.3).We nally obtain the de oding model for sink t ∈ T by ombining the sour emodel of the sour e S together with the pa ket models for mt = 1, 2, . . . , Mt. (Fig-ure 4.5 shows the obtained de oding model for sink node v6 after onne ting thesimplied pa ket models for m6 = 1, 2, 3 with the sour e model.) This de oding modelwill be the basis for an e ient de oder implementation as presented now.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 122

y(2)3

y(3)1,2

y(1)1

i(3)2 = i2

i(2)3 = i3

p1 p2,3

f(2)6|3

f(1)6|1

f(3)6|1,2

i(1)1 = i

(3)1 = i1

Figure 4.5: Probabilisti modelling - De oding model: The simplied de oding modelis obtained by onne ting the simplied pa ket models for m6 = 1, 2, 3 with the sour emodel ( onsisting of the fun tion nodes p1 and p2,3).4.4.2 Iterative De oding AlgorithmWe onsider the de oder at some sink t ∈ T . The goal is to re over the olle tion ofsour e ve tors is ∈ Is for all s ∈ S shortly denoted as iS ∈ IS , IS =∏

s∈S Is. Thede oder jointly uses the inputs y(mt) ∈ Y (mt) of all re eived pa kets mt = 1, 2, . . . , Mtas well as the a-priori knowledge about the sour e statisti s p(iS) in order to produ ean estimate is ∈ Is of all sour es s ∈ S. Considering the ase of MAP de oding (whi his optimal in ase where the error probability has to be minimized), the de oder sele tsthe estimates is ∈ Is for all s ∈ S a ording to (4.1) su h thatis = argmax

i∈Is

p(is = i|y(1),y(2), . . . ,y(Mt)). (4.2)Similar as in Chapter 2, it an be shown that the onditional probability in (4.2) anbe derived via the marginalizationp(is= i|y(1),y(2),. . . ,y(Mt))=

1

γ

∑

iS∈IS :is=i

p(iS ,y(1),y(2),. . . ,y(Mt)), (4.3)where γ = 1/p(y(1),y(2), . . . ,y(Mt)) an be seen as normalization onstant, andp(iS ,y(1),y(2), . . . ,y(Mt)) = p(iS) · p(y(1),y(2), . . . ,y(Mt)|iS). (4.4)Considering the ase of CME, the de oding operation redu es, similar as before, tothe problem of al ulating the onditional probabilities (4.3), whi h is not dis ussedseparately here.It is easy to see that the previously derived de oding model des ribes a validfa torization of the fa tors in (4.4) and that its graphi al representation an be seen as a

CHAPTER 4. JOINT SOURCE-NETWORK CODING 123fa tor graph, e.g., see [KFL01. Employing the sum-produ t algorithm [KFL01, whi h an be run on the fa tor graph, the (global) marginalization in (4.3) an be performedvia (lo al) marginalization giving rise to an e ient al ulation. In parti ular thisis a hieved by running an appropriate message passing algorithm4 along the fa torgraph and, depending on whether the message passing pro edure terminates, or not(i.e., if the fa tor graph is y le-free, or not), the exa t or an approximated value ofp(is = i|y(1),y(2), . . . ,y(Mt)) is obtained simultaneously for all i ∈ Is and all s ∈ S.For the following dis ussion of the sum-produ t algorithm's de oding omplexity,we onsider (for the sake of simpli ity) the ase where S = N and set N = |N |.We dene the following parameters on erning the graphi al de oding model: themaximum alphabet size S = maxn∈N max|In|, |Xn|, |Yn|, the number of fun tionnodes Nf , the number of variable nodes Nv, the maximum degree of fun tion nodesdf , i.e., the maximum number of variable nodes (dire tly) onne ted to any fun tionnode, and (similarly) the maximum degree of variable nodes dv.If we assume that the omplexity of elementary operations (additions, multipli- ations, look-ups, et .) is of onstant omplexity, i.e., of O(1), then the omplexityof al ulating a single message at any fun tion node is of O(Sdf ) and the omplexityof al ulating a single message at any variable node is of O(S) similar as derivedin [BT06. Considering the ase where the fa tor graph is y le-free, an adaptedversion of the e ient forward-ba kward algorithm [KFL01 an be employed, and it an be shown that the omplexity of al ulating all messages at the fun tion nodes isof O(NfdfS

df ), that the omplexity of al ulating all messages at the variable nodesis of O(NvdvS), and that the overall omplexity is given by their sum. For the asewhere the fa tor graph has y les an iterative approa h is required and the omplexitiesderive to be ofO(INfdfSdf ) andO(INvdvS) for the fun tion and variable nodes, where

I ≫ 1 denotes the maximum number of iterations to be performed, respe tively.We observe the following: (a) sour e and pa ket models that an be representedby trees lead to a maximum number of fun tion nodes Nf that is of O(N), (b) dfdepends on the onne tivity of the nodes within the pa ket models (i.e., the numberof pa kets that are jointly en oded) and the properties of the given ( hosen) the sour emodel, ( ) Nv depends on the number of network nodes and their degree, and (d) dvdepends on the node onne tivity and the given ( hosen) sour e model.We on lude that the de oding omplexity, whi h is learly governed by thefun tion nodes, is strongly ae ted by the topology of the pa ket and sour e models(exponential dependen y on the node degree) and not ne essarily by the number of4For fa tor graphs without y les, the e ient forward-ba kward algorithm an be employed, e.g.,see [KFL01.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 124nodes (only linear dependen y on N , if no y les, and at most quadrati dependen y,otherwise). This in turn means that the de oding model used for the de oder imple-mentation should rather aim for a large number of fun tion nodes with a small degreethan a small number with a large degree, i.e., we always should exploit the stru turalproperties of the system to obtain a de oder with low omplexity.4.5 Code Design for Sour e-Network CodingAfter providing a s alable de oding solution for joint sour e-network oding, we nowturn our attention to the sour e-optimized ode design parti ularly onsidering thesour e model introdu ed in Chapter 3.4.5.1 Coding Instan esWe onsider the network with graph G = V, E. The goal is to design odes for all ases, for all oding instan es, that might o ur when ommuni ating the data from thesour es s ∈ S(mt)t , S(mt)

t ⊆ N , to the sink t ∈ T , T ⊆ N , i.e., the goal is to identifyall oding operations needed to produ e pa ket mt ∈ 1, 2, . . . , Mt. The networknodes and the edges involved in produ ing pa ket mt are des ribed by the subgraphG(mt)

t = V(mt)t , E (mt)

t , where V(mt)t ⊆ V are the involved nodes and E (mt)

t ⊆ E are theinvolved edges. Given this subgraph, we are able to determine for all nodes vn ∈ V(mt)tthe set of in oming and outgoing edges ek,l ∈ E

(mt)t and, thus, we are able to identifyfor ea h node vn ∈ V

(mt)t with n ∈ N the set L ⊆ N identifying all input (or parent)nodes and the set K ⊆ N identifying all output (or hild) nodes.5 On the basis ofthose sets, we are able to ategorize the oding instan es by the number of in omingedges |L| and the number of outgoing edges |K| as follows.

• One-to-One: The node vn has exa tly one input node vl, i.e., L = l, andexa tly one output node vk, i.e., K = k (see Figure 4.6 (a)).• One-to-Many: The node vn has exa tly one input node vl, i.e., L = l, andmore than one output nodes vk, k ∈ K (see Figure 4.6 (b1)). Sin e the node'soutputs are fully determined by the node's (single) input, the en oding operation an be split into |K| separate and independent oding operations of the one-to-one type (see Figure 4.6 (b2) for an example where the node vn is split into5Sin e we are onsidering a general sour e-network oding s enario, ea h node vn ∈ V

(mt)t might(optionally) have a sour e input on top of the other (regular) node inputs. In the following, we shalltreat this optional sour e input just like any other regular node input, no further distin tion requiredhere.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 125

Figure 4.6: Sour e-network oding instan es.the virtual nodes v(1)n , v

(2)n , . . . , v(|K|)

n ). The ode design for this oding operations an be approa hed individually, eventually allowing us to divide the overall odedesign problem into smaller, potentially more a essible, omponents.• Many-to-One: The node vn has the more than one input nodes vl, l ∈ L, andexa tly one output node vk, i.e., K = k (see Figure 4.6 ( )). Noti e that this ase has the aspe ts of both sour e and network oding.• Many-to-Many: The node vn has more than one input nodes vl, l ∈ L, andmore than one output nodes vk, k ∈ K (see Figure 4.6 (d1)). This is themost general ase. Similar as before, this oding problem an be split into |K|individual problems of the many-to-one type allowing us simplify and systematizethe ode design (see Figure 4.6 (d2)).Using above observations, the oding operation performed at ea h node vn ∈ V

(mt)t iseither a spe ial ase of, or an be expressed through, the many-to-one oding s enario

CHAPTER 4. JOINT SOURCE-NETWORK CODING 126with several input nodes vl, l ∈ L, and a single output node vk. In this ase the en oderat node vn an be des ribed by the (deterministi ) mapping fun tion γn,k : YL → Xk,where YL =∏

∀l∈L Yl is the ross produ t of the inputs alphabets and Xk is the outputalphabet. Assuming that the sizes of the input and output alphabets are known (e.g.,as obtained by means of the subgraph onstru tion methods in [LRM+06, LMH+07,CMB11b, CMB11a), we propose the following oding strategy:For the spe ial ase of one-to-one ommuni ation, we deem the Diophantine indexassignments (introdu ed in Chapter 3) as a potentially suitable solution for the sour eoptimized ode design (under the given sour e model). In parti ular, the Diophantineindex assignments an be applied to any one-to-one ode setting with |Xk| ≤ |Yl|.Noti e that the ase of |Xk| > |Yl| does not need to be onsidered here (in agreementwith intuition gained from the subgraph onstru tion methods in [LRM+06, LMH+07)sin e it would be wasteful to add redundan y to the node output (in ases where perfe ttransmission hannels are onsidered and, onsequently, no additional hannel oding apabilities are required).For the general ase of many-to-one ommuni ation, we do not have a suitablesolution yet. The problem is that the design of joint sour e-network odes is notyet well understood and that pra ti al oding solutions are still la king (at least forthe system setup under onsideration here). One of the few things we (possibly)know about the ode design for the many-to-one ase is that we should aim for odesettings with |Xk| ≤ |YL| be ause, otherwise, our ode would be wasteful (followingthe same observations as before); but how odes suitable for joint sour e-network oding s enarios should be designed is not yet fully understood. The only ase thatis well understood is the ase where network oding alone is performed (i.e., where nosour e oding is performed). In this ase it is usually assumed that |Xk| = |Yl| for alll ∈ L (i.e., the input and output alphabets are assumed to have the same ardinality).Furthermore, in the important ase of linear network oding [LYC03, it is generallyassumed that the input and output symbols are hosen in the Galois eld F2q (for someinteger number of q) whi h might be somewhat restri tive depending on the a tualsystem requirements. How odes for general seetings of Xk and Yl, l ∈ L, an bedesigned is mostly un lear. Surely it would be possible to resort to randomly hosenmapping fun tions that satisfy the input and output alphabet onstraints, and assignthem in a xed, pre-dened manner to the network nodes, but it seems doubtful thata good system performan e an be a hieved by adopting su h strategies. One possiblestrategy for the sour e-optimized ode design, a possible rst step, is presented below.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 1274.5.2 Joint Sour e-Network Code DesignWe onsider the design of sour e-optimized odes suitable for joint sour e-network oding s enarios. The goal is to outline some ideas without an in-depth dis ussionand without proof that all possible oding instan es are aptured on how pra ti al odes an be designed. Further resear h eorts are required here. As part of thispreliminary treatment of the problem, we shall only onsider the s alar ase where theinput symbols are mapped in a one-to-one fashion onto the output symbols; the ve tor ase is not onsidered here.The main idea of the proposed ode design is to ombine the Diophantine indexassignments introdu ed in Chapter 3 for a distributed sour e oding setting, with linearnetwork odes [LYC03 widely adopted in network oding s enarios. The relevantaspe ts and features of both approa hes are summarized in the following.Diophantine Index Assignments: Diophantine index assignments essentiallyare en oding fun tions that map a set of sour e symbols is ∈ I, I = 0, 1, . . . , L, ontoa set of odewords ws ∈ Ws, Ws = 0, 1, . . . , K1 − 1, s ∈ S. In parti ular, we onlyallow for the ase where Ks ≤ L. The odewords ws ∈ Ws are obtained from thesour e symbols is ∈ I by performing the modulo−Ks operationws ≡ is (mod Ks). (4.5)En oding fun tions operating in this manner have the property that the output ode-words ws ∈ Ws dire tly orrespond to the "high-resolution" information about the in-put symbols is ∈ I, i.e., the information that would be ontained in the least signi antbits of a natural oded number. This is opposed to the "low-resolution" informationthat would be ontained in the most signi ant bits.6 In the ase of Diophantineindex assignments the "low-resolution" information about the input symbol is ∈ I isdropped (to some extent) and not transmitted; it an not be inferred from the output odeword ws ∈ Ws of a single sour e s ∈ S alone. However, sin e the de oder jointlyuses the odewords ws ∈ Ws from several (or all) sour es s ∈ S, it is apable to re overthe "low-resolution" information from the re eived odewords. This, of ourse, onlyholds true if the Diophantine index assignments are designed appropriately, i.e., if the odeword alphabet sizes Ks, s ∈ S, are sele ted to jointly satisfy the needs imposedby the sour e model.Linear Network Codes: In linear network oding [LYC03, generally onsider-ing the many-to-one ase (see Figure 4.7), the node's output b is obtained by forming6Noti e that this on ept of "high-resolution" and "low-resolution" information is not denedfurther here; it is mainly used as a metaphor for the ode design strategy proposed in the following.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 128

Figure 4.7: Linear network oding example.the linear ombination of the node's inputs al, l ∈ L, as follows:b =

∑

l∈L

cl · al, (4.6)where cl, l ∈ L, are predened oding oe ients. The node's inputs, the node'soutput, and the oding oe ients (as well as the performed operations) are allassumed to be in the Galois eld F2q , where q is some integer number dening theeld size. In the following, we shall sometimes refer to the information arried by thenode's output b as the "linearly ombined" information of the node inputs al, l ∈ L.Diophantine Sour e-Network Coding: In order to obtain odes for jointsour e network oding, we now ombine the Diophantine index assignments with linearnetwork odes.Re all that for the many-to-one ase the en oder at node vn, n ∈ N , with theinput nodes vl, l ∈ L, L ⊆ N , and the single output node vk, k ∈ N , an be des ribedby a deterministi mapping YL → Xk, where YL =∏

∀l∈L Yl is the ross produ t ofthe inputs alphabets and Xk is the output alphabet.We generally allow for exible realizations of the (individual) input alphabet sizes|Yl|, l ∈ L, as well as the output alphabet size |Xk|, in order to ensure that |Xk| ≤ |YL| an be found satisfying the needs of the system. The idea of the proposed joint sour e-network oding s heme is to gain exibility in the ode design by allowing a exibleallo ation of the sour e oded data representing the "high-resolution" informationabout individual sour es, the network oded data representing the "linearly ombined"information about several sour es, and the data that a tually does not need to betransmitted representing the "low-resolution" information (i.e., the data that an bere overed by jointly pro essing the re eived sour e data at the de oder). This shall bea hieved by a systemati ode.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 129Figure 4.8: Systemati sour e-network oding.The proposed oding s heme assumes that the node's input and output symbolsare represented by a tuple of both sour e and network oded data. Considering node

vn, the node's input ylm is represented by the tuple(jm, am), (4.7)where the so- alled virtual sour e symbol jm represents the sour e oded data arryingthe "high-resolution" information of one or more sour es, and am represents the net-work oded symbol arrying the "linearly ombined" information of several sour es (seeFigure 4.8 and Figure 4.9). The union of the virtual sour e symbol jm and the network oded symbol am dire tly orresponds to the output xn of the parent node vlm re eivedas input ylm at the node vn. We allow that jm or am an be empty, e.g., in the asewhere node vs, s ∈ S, has only a sour e input, the virtual sour e symbol orrespondsto the a tual sour e symbol is, whereas the network oded symbol is onsidered asempty.Considering node vn, the node's output xk an prin ipally be represented twoways. The most dire t representation is given by the (|L|+ 1)−tuple

(w1, w2, . . . , w|L|, b), (4.8)where w1, w2, . . . , w|L| are the odewords obtained after applying Diophantine index as-signments to the node inputs j1, j2, . . . , j|L| and b is the result of the linear ombinationof the node inputs a1, a2, . . . , a|L| (see Figure 4.8 and Figure 4.9).The node output b is determined by the following linear ombination of the nodeinputs a1, a2, . . . , a|L| su h thatb =

|L|∑

m=1

cm · am, (4.9)where c1, c2, . . . , c|L| are the oding oe ients hosen to span a |L|−dimensional ve torspa e. In our ase, we hoose cm = 1, m = 1, 2, . . . , |L|. Furthermore, we assume

CHAPTER 4. JOINT SOURCE-NETWORK CODING 130

Figure 4.9: Diophantine sour e-network oding example.that the eld size is onstant over all network nodes involved in ommuni ating the onsidered pa ket from the sour es S ⊆ N to the sink t ∈ T , T ⊆ N .For a onform representation of the node inputs and the node outputs, we rep-resent the olle tion of the odewords w1, w2, . . . , w|L| by an equivalent representationthrough the single odeword index w. Assuming that wm ∈ Wm,Wm = 0, 1, . . . , Km−

1 for m = 1, 2, . . . , |L|, the odeword index w ∈ 0, 1, . . . , (∏|L|m=1 Km)−1 is al ulateda ording to

w =

|L|∑

m=1

(m−1∏

i=1

Ki

)

wm, (4.10)whi h orrresponds to the linear index of the odewords w1, w2, . . . , w|L| in an arrayof size K1 × K2 × . . . × K|L| a ording to the olumn-major order (e.g., as used inMatlab).The node's output xk is then given by the tuple(w, b), (4.11)allowing for a homogeneous representation of the node inputs and outputs throughoutthe network while eliminating the need for intermediate de oding at traversed networknodes.4.5.3 Open ProblemsIn this preliminary treatment of the problem, we proposed a method for the systemati design of joint sour e-network odes using a ombination of the Diophantine indexassignments introdu ed in Chapter 3 and standard linear network odes [LYC03.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 131Although it seems that the proposed approa h is appli able in general s enarios andgeneral settings, more work is needed to settle this issue.Furthermore, although the Diophantine index assignments are (at least partly)treated in an analyti al fashion, the proposed sour e-network oding s heme is purelyheuristi so far. In parti ular, given the input alphabet sizes |Yl|, l ∈ L, L ⊆ N , andthe output alphabet size |Xk|, k ∈ N , it is not yet understood what amount of datashould be allo ated for network oding and what amount of data should be allo atedfor sour e oding, i.e., it is un lear on how to hoose the eld size 2q with respe t tothe ardinality of the odeword alphabets given by the produ t ∏|L|m=1 Km su h thatsystem performan e is maximized. For example, this problem ould be approa hed by onsidering similar sour e models and by employing similar te hniques as des ribed inChapter 3 to provide the means for an analyti al design. However, this was beyondthe time frame of this thesis.4.6 Proof-of-Con eptIn order to underline the versatility and ee tiveness of our joint sour e-network oding solution, we now provide a proof-of- on ept in form of numeri al results. Twodistin t s enarios are onsidered. The rst s enario fo uses on evaluating the de oderperforman e in a s enario where separation between sour e and network oding doesnot hold in general. The se ond s enario aims at providing a proof-of-s alability, i.e,we want to show that our methods are pra ti able for s enarios with a large numberof nodes, by onsidering a sensor network s enario where data from several spatially orrelated sour es is to be transmitted over a network to several sinks. Sin e theperformed simulations only represent a rst step towards a better understanding of thebehavior, the restri tions, and the performan e of the proposed joint sour e-network oding solutions, we on lude this se tion with an outline of possible experiments tobe ondu ted in the future.4.6.1 De oder EvaluationWe rst onsider a s enario where separation between sour e and network oding doesnot hold in general. In parti ular, we onsider the s enario with three sour es and twosinks introdu ed in [RJCE06 and shown in Figure 4.1. Furthermore, we shall fo uson the ase of blo k length L = 1.The (dis rete-valued) sour e symbols Is are assumed to be quantized versions ofthe ontinuous-valued sour e samples Us, s = 1, 2, 3, where the ve tor (U1, U2, U3)

T is

CHAPTER 4. JOINT SOURCE-NETWORK CODING 132Table 4.1: Simulation results - Counterexample where separation does not hold ingeneral [RJCE06ǫ [bit 0 1

3h = 0.942 2

3h = 1.88 h = 2.83(ρ) (1) (0.988) (0.881) (0)

δ [bit 0 0 0.585 1 0 0.322 0.585 0Rh [bit 3 3 3 3 3 3 3 3Rǫ [bit 0 1 1.59 2 2 2.32 2.59 3Rh+ǫ [bit 3 4 4.59 5 5 5.32 5.59 6P1,6 =P2,6 0 0 0 0 0 0 0 0P3,6 0 82.6e-3 43e-6 18e-6 28.2e-3 5.20e-3 430e-6 0P1,7 =P2,7 0 65.8e-3 43e-6 11e-6 23.7e-3 4.50e-3 468e-6 0P3,7 0 0 0 0 0 0 0 0SNR1,6 [dB 14.6 14.6 14.6 14.6 14.6 14.6 14.6 14.6SNR2,6 [dB 14.6 14.6 14.6 14.6 14.6 14.6 14.6 14.6SNR3,6 [dB 14.6 12.6 14.6 14.6 9.40 12.3 14.2 14.6SNR1,7 [dB 14.6 8.29 14.6 14.6 9.17 13.0 14.5 14.6SNR2,7 [dB 14.6 12.8 14.6 14.6 9.87 12.5 14.2 14.6SNR3,7 [dB 14.6 14.6 14.6 14.6 14.6 14.6 14.6 14.6Case (a) (b) (b1) (b2) ( ) ( 1) ( 2) (d)distributed a ording to a multivariate Gaussian distribution. To emulate the s enarioin [RJCE06, where I2 and I3 are orrelated and I1 is independent of them, the jointdistribution of (U1, U2, U3)

T is hosen su h that p(u1, u2, u3) fa tors into p(u1)· (u2, u3)where U1 ∼ N (1, 0) and (U2, U3) ∼ N (Σ, µ) with Σ =[

1 ρρ 1

], µ = (0, 0)T and orrelation oe ient ρ. We are able to express this orrelation model of the sour esI1, I2 and I3 in terms of ( onditional) entropies su h that H(I1)=H(I2)=H(I3)=h,H(I2|I3) = H(I3|I2) = ǫ and H(I2, I3) = h + ǫ where h (under afore mentionedassumptions) depends of the hosen quantizer and ǫ (additionally) on the orrelation oe ient ρ.After introdu ing the onstants Rh, Rǫ, Rh+ǫ and using the results in [SY01,it an be easily veried that the rates R1,6 = R1,4 = R2,4 = R3,7 = R5,6 = Rh ≥ h,R3,6 =Rǫ≥ ǫ and R4,5 =R5,7 =Rh+ǫ≥h + ǫ are admissible. Following this results, we hoose Rh = ⌈h⌉, Rǫ = ⌈ǫ⌉ + δ and Rh+ǫ = ⌈h + ǫ⌉ + δ, where δ orresponds to someadditional rate we might be willing to utilize to improve the de oding result.The oding fun tions at node v1, v2 and v3 are hosen as follows. No network oding is performed sin e the nodes only have a single (sour e) input. Sour e oding

CHAPTER 4. JOINT SOURCE-NETWORK CODING 133is performed based on the Diophantine index assignments introdu ed in Chapter 3.Whenever the output rate is Rh, the Diophantine index assignments have 2Rh inputsymbols and 2Rh output symbols, i.e., a simple bije tive mapping is performed. In the ase where the output rate is Rǫ, the Diophantine index assignments have 2Rh inputsymbols and 2Rǫ output symbols, i.e., a simple surje tive mapping is performed.At node v4 a ombination of sour e and network oding is performed using asystemati ode as introdu ed in Se tion 4.5. Network oding is performed on thenode input of node v1 and node v2 with unit oding oe ients and eld size 2Rh. Nosour e oding is performed on the input of node v1. The input of node v2 is sour e oded using a Diophantine index assignment with 2Rh input symbols and 2Rǫ outputsymbols.Node v5 simply forwards the network oded data to node v6 and node v7, orresponding to an addition with the neutral element 0. The sour e oded data isonly forwarded to node v7, i.e., a bije tive mapping is performed based on Diophantineindex assignments with 2Rǫ input symbols and 2Rǫ output symbols.For our numeri al results, we hoose a 8 level Lloyd-Max quantizer leading toh = 2.83 bit and hoose ǫ = 0 · h, 1

3· h, 2

3· h, h orresponding to ρ = 1, 0.988, 0.881and 0 (derived by experiment). We used 1 · 106 samples for ea h sour e and ea hsimulation. For the dis rete-valued ase, we quantify the system performan e in termsof the symbol error probability Ps,t between the sour e symbol Is and its estimate

Is,t. For the ontinuous-valued ase, the system performan e is evaluated based on theoutput signal-to-noise ratioSNRs,t = 10 log10

(

EU2s

E(Us,t − Us)2

)

[dB] (4.12)approximated by the sample average. Numeri al results for the presented setup andseveral values of δ are summarized in Table 4.1. In Case (a) where U2 and U3 are fully orrelated (i.e., u2 = u3) and Case (d) where U2 and U3 are statisti ally independent,we obtain optimal results with a error probability Ps,t = 0 and an output SNRs,t =

14.6 dB (distortion of Lloyd-Max quantization alone). The optimality of the resultsis expe ted, sin e in ea h of those ases the system degrades and an be representedby an equivalent system that does not require sour e orrelation for de oding. ForCase (b) and ( ) with δ = 0, we need the sour e orrelation for de oding. We observethat our joint sour e-network de oding approa h already a hieves reasonably goodperforman e for δ = 0. Furthermore, onsidering Case (b1), (b2), ( 1) and ( 2), weobserve that if we are willing to in rease the transmission rate by a small amountδ > 0 then the overall performan e improves rapidly, whi h underlines the apabilitiesof the de oder to ee tively exploit additional redundan y (in the re eived pa kets)

CHAPTER 4. JOINT SOURCE-NETWORK CODING 134to improve the overall de oding result.4.6.2 Proof-Of-S alabilityWe onsider an exemplary sensor network s enario with 30 nodes, uniformly dis-tributed in an unit square of size 1×1, in luding 6 sour e nodes and 3 sink nodes (seeFigure 4.10 (a)). In parti ular, we have the network nodes vn, n ∈ N , whereN = 1, 2, . . . , 30, the sour e nodes vs, s ∈ S, where S = 1, 2, . . . , 6, and thesink nodes vt, t ∈ T , where T = 28, 29, 30. We assume that the nodes have atransmission radius of r = 0.3 (see Figure 4.10 (a)).The sour es U = (U1, U2, . . . , U6)

T are spatially distributed and produ e outputsu = (u1, u2, . . . , u6)

T distributed a ording to a multivariate Gaussian distributionN (µ,Σ) with

Σ =

1 ρ1,2 · · · ρ1,6

ρ2,1 1 · · · ρ2,6... ... . . . ...ρ6,1 ρ6,2 · · · 1

and µ = 0T

6 = (0, 0, 0, 0, 0, 0)T . In parti ular, we assume that the orrelation oe ientρk,l for k, l ∈ S, k 6= l, de reases exponentially with the Eu lidean distan e dk,l,su h that ρk,l = exp(−β · dk,l). Sin e the performan e of our te hniques depend onthe orrelations between the sensors, we onsider two dierent s enarios, one withβ = 0.25 (strongly orrelated sensor measurements) and one with β = 1 (weakly orrelated measurements).We fo us on the ase of blo k length L = 1, i.e., the ase where ea h ea h sour esymbol is en oded and transmitted separately. All s alar quantizers at the en odersare Lloyd-Max optimized to minimize the mean squared error in the sensor readingsus, s ∈ S, using an identi al resolution of 3 bits, orresponding to 8 levels.In order to obtain an overlay-network suitable for joint sour e-network odingand in order to obtain the required transmission rates, we apply the onstru tionmethod des ribed in [CMB11b, CMB11a. This onstru tion method onsists of twomajor steps it rst applies the sour e-optimized lustering method des ribed inSe tion 2.4 with a maximum lustersize of two to the sour es, and se ondly it arriesout a generalized version of the overlay onstru tion method des ribed in [LMH+07.In our s enario (see Figure 4.10 (a)) the sour e-optimized lustering method produ esa total of three lusters, grouping together node v1 and node v5, node v2 and nodev4, as well as node v3 and node v6. To initialize the overlay network onstru tionmethod, we set the edge apa ities to 6 bits ( orresponding to an alphabet size of 64

CHAPTER 4. JOINT SOURCE-NETWORK CODING 135

Figure 4.10: Considered s enario - Original and simplied network: (a) Originalnetwork with N = 30 randomly distributed nodes with a transmission range of r = 0.3,and (b) the same network after overlay-network onstru tion for a apa ity onstraintof 6 bits per edge and using a strong orrelation s enario with β = 0.25.symbols), and the edge weights are hosen linearly proportional (with fa tor 10) tothe ommuni ation distan e between the nodes (see Figure 4.10 (b) for β = 0.25).The oding fun tions are designed as des ribed in Se tion 4.5.2 using the pre-viously obtained transmission rates (rounded up) to determine the required inputand output alphabet sizes. The eld size is hosen (individually) for ea h pa ketmt ∈ 1, 2, . . . , Mt a ording to the minimum transmission rate over all edges E (mt)

tthat are involved in ommuni ating the data from the sour es S to the sink t ∈ T .The system performan e is evaluated based on the output signal-to-noise ratio(SNR) SNRs,t = 10 · log10

(

EU2s

E(Us,t − Us)2

)

[dB]approximated by the sample average over 20000 samples for ea h individual pair ofthe sour es s ∈ S and the sinks t ∈ T . Furthermore, we shall also onsider the averageSNR SNRt = 10 · log10

(

E||U||2

E||U−U||2

)

[dB](over all sour es in S) of the individual sinks t ∈ T .

CHAPTER 4. JOINT SOURCE-NETWORK CODING 136Table 4.2: Simulation results - System performan e for β = 0.25 (strongly orrelatedsour es).SNRs,t [dB Sour e SNRt [dB1 2 3 4 5 628 14.1 12.9 14.7 14.6 14.6 11.9 13.7Sink 29 14.1 12.9 12.3 14.6 14.6 14.6 13.730 14.1 12.9 12.3 14.6 14.6 14.6 13.7Table 4.3: Simulation results - System performan e for β = 1.00 (weakly orrelatedsour es).SNRs,t [dB Sour e SNRt [dB1 2 3 4 5 628 14.7 14.7 14.7 14.6 14.7 14.6 14.7Sink 29 14.7 14.7 14.7 14.6 14.7 14.6 14.730 14.7 14.7 14.7 14.6 14.7 14.6 14.7The following performan e evaluation is based on numeri al results obtainedthrough simulation in Matlab. Table 4.2 shows the results for β = 0.25 (strongly orrelated sour es) and Table 4.3 shows the results for β = 1.00 (weakly orrelatedsour es), using joint sour e-network odes based on Diophantine index assignmentsand linear network odes. Assuming the distribution of the simulated distortions isapproximately Gaussian, the onden e intervals an be al ulated; we obtain intervalsof approximately 0.25 dB. In both ases, i.e., for the strong and the weak orrelations enario, we observe that obtained performan e is lose to the maximum possibleperforman e of 14.6 dB that an be attained when using 3 bit Lloyd-Max quantizersat the sour es. It is important to point out that the obtained performan e results forthe strong and weak orrelation s enarios annot be ompared dire tly, sin e also theoverall osts, and therefore the obtained transmission rates, of the resulting overlay-networks are (generally) dierent in both ases. In parti ular, for the strong orrelations enario the overall ost of the onstru ted overlay-network is 195.9 and for the weak orrelation s enario the overall ost is 226.5 and, thus, the the total rate required to ommuni ate the sour e data to the sinks is potentially smaller in the strong orrelation ase than in the weak orrelation s enario. Sin e there are 6 sour es in our s enario,ea h using a 3 bit quantizer, we would need to transmit with a total rate of 6 ×

3 bit = 18 bit for ea h sampling interval and ea h re eiver when no sour e andnetwork oding is employed. Using our oding approa h des ribed in Se tion 4.5.2, we

CHAPTER 4. JOINT SOURCE-NETWORK CODING 137are able to lower this total rate to 16 bit in the strong orrrelation ase; in the weak orrelation ase, and under above assumptions, we do not a hieve rate improvementsand the total rate remains equal to 18 bit. However, it is important to point out thatin both ases a good system performan e is a hieved using our joint oding solution.Although the simulation results obtained for this parti ular s enario are by far notrepresentative in generality, the important on lusion is that the proposed en odingand de oding te hniques an be applied, in a pra ti able fashion, to s enarios witha medium to large number of nodes whi h to a hieve was the main obje tive of thiswork.4.6.3 Open IssuesIn the previous experiments we showed that our joint sour e-network oding solution an be su essfully applied to ases were a modular approa h based on separatesour e and network oding would fail. Furthermore, we showed that our methodsare pra ti able in large-s ale settings and that a good system performan e an bea hieved. The ondu ted experiments are however far from on lusive so far andfurther experiments are required. Possible experiments in lude the following.Representative evaluation: So far we only presented simulation results fortwo distin t sour e-network oding s enarios. Although further experiments have been ondu ted in the s ope of this thesis (not expli itly presented here), and although thoseexperiments seem to onrm that our oding solutions indeed a hieve a good systemperforman e in the majority of the onsidered s enarios, a more exhaustive analysisis required here to guarantee some degree of representativeness. This is parti ularlytrue for the ase where a quantitative analysis of the system performan e is desired,but also holds for the ase where we may want to identify possible onstraints of our oding te hniques.Separate ode and de oder evaluation: The previously presented resultsshow the performan e of the overall system, whi h is generally ae ted by the proper-ties of the used ode, and partly inuen ed by the de oder performan e. As pointedout in Chapter 2, the de oder an only be optimal, if the used fa tor graph does not ontain y les. However, when onne ting the sour e model with the pa ket model,the resulting fa tor graph usually has y les. Therefore the de oder performan e and,thus, the system performan e will not be optimal. In order to determine the portionof performan e loss in urred by the ode properties and the portion resulting from thede oder's suboptimality, a more sophisti ated set of experiments needs to be devised.Impa t of ode parameters: As pointed out in Se tion 4.5.3, it is not yet lear how the ode parameters should be hosen to optimize the system performan e.

CHAPTER 4. JOINT SOURCE-NETWORK CODING 138Although it might be possible to partly resolve this issue by an analyti al investigationit seems likely that numeri al simulation results might be required, or at least mightbe useful, to identify strategies for xing those parameters.4.7 Con lusionsThe goal of this work was to provide a pra ti al oding solution to ost-ee tivelydeliver orrelated data in large-s ale sensor networks. Optimal solutions all for jointsour e-network oding that allows for ooperative in-network data pro essing as thepa kets traverse the network. Implementing joint sour e-network oding in reality ishowever rather hallenging mostly due to the inherent de oding omplexity [CME05.Motivated by this observation, we proposed a s alable de oding solution feasible for alarge number of nodes. Given the network topology and the orrelation stru ture ofthe data, we showed how to probabilisti ally represent the overall system (in ludingthe pa kets' pathes) by a fa tor graph. The obtained fa tor graph was then usedfor an e ient de oder implementation using the iterative sum-produ t algorithmfor de oding. We showed that the resulting de oding omplexity is mostly governedby the maximum node degree, and not by the number of network nodes, allowingfor a omputationally tra table de oder implementation even in large-s ale s enarios.Furthermore, we proposed a simple lass of memory-less but (at the same time) sour e-optimized ode onstru tions that are parti ularly suitable for joint sour e-network oding s enarios with multivariate Gaussian sour es. Our experiments onrmed thatour s alable solution an be su essfully applied to large-s ale s enarios, and that agood performan e an be a hieved, even in s enarios where a separate sour e andnetwork oding approa h would fail.

Chapter 5Con lusions5.1 SummaryThis thesis provided basi theory and pra ti al oding solutions for data gathering indistributed sensor networks. To rea h this goal, we put into pra ti e the followingthree strategies:• Sour e-optimized lustering: We presented a s alable solution for distributedsour e oding in large-s ale sensor networks. The proposed methods rely on the ombination of a low- omplexity en oding stagea s alar quantizer followed bya distortion-optimized index assignment stageand a sour e-optimized luster-ing algorithm. We further showed how the obtained lusters an be in orporatedinto a sour e-optimized fa tor graph for an e ient de oder implementation.Despite the simpli ity of the proposed te hniques, our results show signi antperforman e gains as ompared to standard s alar quantization alone (i.e., with-out exploiting the sour e orrelations).• Diophantine ode design: We presented a onstru tive framework for thedesign of sour e-optimized distributed sour e odes. Using insights from fun-damental number theory, we proposed a new lass of memory-less index as-signments that an be tailored to exploit sour e properties ommon to manyimportant sour e models. Using Diophantine equations as a powerful designtool, we were able to formulate algorithms for a stru tured ode design whi hgenerally work for more than two sour es. We showed how the memory-lessindex assignments an be used at the ore of trellis-based distributed sour e odes. Thus, we were able to obtain odes for short as well as long blo k lengths enarios, allowing a exible trade-o between the system performan e and its139

CHAPTER 5. CONCLUSIONS 140 omplexity. Using numeri al and analyti al results, we showed that the proposed odes are apable of a hieving a good end-to-end performan e for a wide rangeof possible orrelation setups and more than two sour es.• S alable joint sour e-network oding: We presented a joint sour e-network oding solution for large-s ale networks. Given the network topology and the orrelation stru ture of the data, we showed how to probabilisti ally representthe overall system (in luding the pa kets' pathes) by a fa tor graph model whi h an be used for an e ient de oder implementation using the iterative sum-produ t algorithm. Furthermore, we briey addressed the problem of possible ode onstru tions potentially suitable for joint sour e-network oding. Ourexperiments onrmed that our s alable solution an be su essfully applied tolarge-s ale s enarios, and that a good performan e an be a hieved, even ins enarios where a separate sour e and network oding approa h would fail.Those three strategies have in ommon that ea h of them systemati ally exploits,to a ertain degree, the properties of the systemthe orrelation stru ture of thesour e data, the properties of the sour es' joint probability distribution, or the networktopology in order to allow for a ( omputationally) tra table de oder implementation.This, of ourse, also means that both the ode design and the de oder implementationare rather spe ialized for the given s enario or a lass of potential s enarios. Therefore,when designing the system, it needs to be de ided whether or not, and to what degree,the system properties an be taken into a ount to optimize the system performan e,its omplexity, or both. This mainly needs to be de ided individually, from aseto ase, although it is possible to formulate and to implement automatized ways toobtain pra ti al oding solutions for general sensor network s enarios. Generally, thisspe ialized mode of operation might be seen as a severe restri tion, however, sin e mostsensor networks are in any ase strongly dependant on the needs of the appli ation athand, this is likely to be a reasonable option for the network designer. Consideringour numeri al and analyti al results, we indeed believe that the adopted strategiesare a valid, maybe even ru ial, rst step towards providing pra ti al oding solutionsfor data gathering in large-s ale sensor networks. Follow up work by other resear hgroups [YY09, Yah10 might on ede that we are on the right path.5.2 Future WorkAs mentioned above, in order to su essfully put into pra ti e the proposed odingstrategies sour e-optimized lustering, Diophantine ode design, and joint sour e-

CHAPTER 5. CONCLUSIONS 141network oding in general sensor network s enarios it might be worthwhile, maybeeven ne essary, to opt for automati ways to design the odes and the fa tor graphmodels, maybe as part of a s heduled alibration stage, without extensive interventionof a network designer. Investigating in this dire tion learly provides ample oppor-tunities for future work in this area equally on erning ea h of the proposed odingstrategies. Considering the individual strategies, possible extensions and future workmight in lude the following.Sour e-optimized lustering: The sour e-optimized lustering method pre-sented in this work is based on the Kullba k-Leibler distan e (KLD) [CT91. Themain reasons for the hoi e of this optimization riterion was that, on one hand, theKLD is a standard measure to ompare the similarity of probability distributions and,on the other hand, the KLD was deemed to be a suitable measure in prior work relatedto this thesis [BT06. However, it would be equally possible to hoose other riteriasu h as the mutual information [CT91. This parti ular hoi e might be of spe ialinterest sin e it was shown in [GSV05, PS06 that there exists a dire t onne tionbetween the mutual information and the (gradient of the) mean squared error (MSE)in multiple input multiple output (MIMO) systems. Sin e the MSE is often onsideredas performan e riterion in sensor network appli ations, as assumed at several parts ofthis thesis, it might be of parti ular interest to optimize the lusters and the de oderfa tor graph based on the mutual information instead of the KLD, provided that asimilar result as in [GSV05, PS06 an be established for our sensor network s enario.Furthermore, it is surely worth onsidering the ase where the distribution ofthe sensor observations is not known beforehand, or where it is not of multivariateGaussian shape, or both (as assumed at several o asions in this thesis in orderto obtain losed-form expressions). Thus, generally dierent oding and lusteringstrategies need to be adopted possibly based on training sets or similar approa hes.Finding oding solution for su h problems still remains a hallenging task and requiresfurther resear h eorts.Diophantine ode design: We showed in this thesis how (memory-less) Dio-phantine index assignments an be used for the stru tured design of trellis-based odes (having memory) similar as done for a distributed sour e oding based onsyndromes (DISCUS) in [PR99, PR03. However, it also should be possible to applythe same ideas and on epts in the design of stru tured turbo- odes similar as donein [BM01 possibly laying the groundwork for future work in this area.Considering dualities between multiterminal oding problems [SCX06, the iden-tied ode onstru tions might also nd appli ations in other areas. The possible setof known dual problems en ompasses oding for the (deterministi ) broad ast han-

CHAPTER 5. CONCLUSIONS 142nel [SCX06, CT91, data hiding appli ations [CSPR99, loss-less multiple des ription oding [SCX06 and hannel oding [SCX06. Furthermore, there might be potentialappli ations in video ompression [PBAT08 and, although this might seem a bitdistant, in multiterminal quantum ommuni ation settings [ADHW06. Whether ornot the proposed odes an be applied to su h s enarios will however largely dependon the fa t to what extent the onsidered sour e models nd their ounterpart in the orresponding dual problem formulations.S alable joint sour e-network oding: Given the timeframe of this thesis,we only were able to briey address the problem of the a tual ode design for jointsour e-network oding s enarios. Here a broader and more thorough treatment of theproblem would be more than desireable.Furthermore, it would be interesting to identify algorithms to onstru t anoverlay network suitable for joint sour e-network oding in general setting similardone as in [LMH+07 for the ase of two orrelated sour es. A rst step towards thisgoal was already a hieved in [CMB11b, CMB11a, however, a more general approa hsupporting arbitrary luster sizes would be an wel ome extension here.Naturally, we also would like to be able to ompare the performan e againstthe fundamental rate-distortion limits for networks with orrelated sour es, but at thetime su h limits appear to be a distant goal.

Appendix ASour e-Optimized ClusteringA.1 Optimal De oding RuleIn this se tion we derive a simple expression for the optimal de oding rule.Let k ∈ N be the index identifying the sour e for whi h the estimate has to be al ulated. Let T ⊆ N be a set of indi es identifying the en oders whose odewords, olle ted in the ve tor wT ∈WT , WT =

∏

∀n∈T Wn, are available for de oding.Spe i ally, the goal is to show that EUk|wT =∑|Ik|−1

ik=0 EUk|ik · p(ik|wT ).We start by using the denition of the onditional expe tation and apply the Bayesrule su h thatEUk|wT =

∫ +∞

uk=−∞

uk · p(uk|wT ) duk =1

p(wT )·

∫ +∞

uk=−∞

uk · p(wT |uk) · p(uk) duk.(A.1)Furthermore, we an state thatp(wT |uk) =

|Ik|−1∑

ik=0

p(wT |ik) · p(ik|uk) =

p(wT |ik), if qk(uk) = ik,

0, otherwise.Sin e the index ik = qk(uk) is onstant for all uk that fall into the quantizer regionBk(ik) su h that bk(ik) < uk ≤ bk(ik + 1), the integral in (A.1) an be splitted intoseparate parts and we obtain

EUk|wT =1

p(wT )·

|Ik|−1∑

ik=0

p(wT |ik)

∫ bk(ik+1)

uk=bk(ik)

uk · p(uk) duk. (A.2)We observe that p(wT |ik) = p(ik |wT )·p(wT )p(ik)

and that1

p(ik)·

∫ bk(ik+1)

uk=bk(ik)

uk · p(uk) duk

(a)=

∫ +∞

uk=−∞

uk · p(uk|ik) duk = EUk|ik, (A.3)143

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 144where the equality (a) holds sin e p(ik|uk) is either unity or zero, depending on whetherqk(uk) = ik or not (i.e., if uk falls into the quantizer region Bk(ik) su h that bk(ik) <

uk ≤ bk(ik + 1), then p(ik|uk) = 1). We therefore on lude thatp(uk|ik) =

p(uk, ik)

p(ik)=

p(ik|uk) · p(uk)

p(ik)=

p(uk)p(ik)

, if qk(uk) = ik

0, otherwise. (A.4)The desired equality an then be established using above results together with (A.2).A.2 E ient Marginalization and its ComplexityIn this se tion we hara terize the omplexity of the marginalization operation requiredat several points of this work, e.g., onsider the al ulation of the optimal estimatein (2.3) where the marginalization in (2.4) has to be performed using the argumentin (2.5).For a general treatment of the problem, we shall employ the same denitionsas provided in Appendix A.1. Let furthermore S = k ∪ T be a set of indi esidentifying the sour es whose dis rete representations, olle ted in the ve tor iS ∈ IS ,IS =

∏

∀n∈S In, are involved within the al ulation. Spe i ally, we shall onsider the al ulation of p(ik = l|wT ) out of p(wT , iS) through the following marginalizationp(ik = l|wT ) = γ ·

∑

∀iS∈IS :ik=l

p(wT , iS) (A.5)with γ = 1/p(wT ) andp(wT , iS) = p(iS) · p(wT |iS)

(a)= p(iS) · p(wT |iT ) = p(iS) ·

∏

∀n∈T

p(wn|in), (A.6)where equality (a) obviously holds for k ∈ T , sin e in this ase S = T , and also fork /∈ T , sin e in this ase ik does not provide any information about wT due to thefa t that iT is known and wn = mn(in) for all n ∈ T su h that p(wT |iS) = p(wT |iT ).In the most straightforward implementation of the marginalization in (A.5),the summation over p(wT , iS) has to be performed over all possible realizations ofiS ∈ IS with ik = l where the a tual value of p(wT , iS) an be al ulated using theprodu t representation in (A.6). It is worth pointing out that p(wT |iT ) in (A.6) anbe ome either zero or unity depending on the urrent onguration of the transitionprobabilities p(wn|in) for all n ∈ T . This an be used to restri t the number of indextuples iS ∈ IS that have to be onsidered throughout the marginalization in (A.5), asshown in the following.

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 145For brevity, we shall restri t ourselves to the ase where k ∈ T , i.e., whereS = T .1 Let QT (wT ) be the set of index tuples iT ∈ IT that are mapped ontowT ∈WT .2 Then, the marginalization in (A.5) an be expressed as follows:

p(ik = l|wT ) = γ ·∑

∀iT ∈QT (wT ):ik=l

p(iT )

=

γ ·∑

∀iT ∈ik=l×QT \k(wT \k)

p(iT ), if mk(ik = l) = wk,

0, otherwise. (A.7)Noti e that the marginalization a ording to (A.7) has to be performed, if it hasto be performed at all, only over the members of the set QT \k(wT \k). Sin e the ardinality of this set is mu h smaller than the ardinality of IT in (A.5) the omplexityof the marginalization an be redu ed onsiderably.For a more detailed dis ussion of the omplexity, the ardinality of the setQT (wT ) shall be hara terized in the following. Noti e that Qn(wn) denotes theset of indi es in ∈ In that are mapped onto the odeword wn ∈ Wn. The followingresult is usefulLemma A.2.1. For any surje tive mapping fun tion mn : In → Wn and any wn ∈

Wn, |Qn(wn)| ≤ |In| − |Wn|+ 1.Proof. Sin e mn is a fun tion ea h in ∈ In is mapped to exa tly one wn ∈ Wn.From this we on lude that (a) there are no in ∈ In that are mapped to more thanone wn ∈ Wn ⇒⋂

∀wn∈WnQn(wn) = ∅ (mutual ex lusivity) ⇒ |⋃∀wn∈Wn

Qn(wn)| =∑

∀wn∈Wn|Qn(wn)| and (b) ea h in ∈ In is mapped to some wn ∈ Wn

⇒ |⋃

∀wn∈WnQn(wn)| = |In|. Sin e mn is a surje tive fun tion there exists an in ∈ Infor any wn ∈ Wn su h that wn = mn(in) and we on lude that ( ) |Qn(wn)| ≥

1 for all wn ∈ Wn. From (a) and (b) we obtain that ∑∀wn∈Wn|Qn(wn)| = |In|whi h an be solved for an arbitrarily hosen wn ∈ Wn, e.g. wn = a, and weobtain |Qn(wn = a)| = |In| −

∑

∀wn∈Wn:wn 6=a |Qn(wn)|. Be ause of ( ) we know that|Qn(wn = a)| is maximal if |Qn(wn)| = 1 for all wn ∈ Wn : wn 6= a and we obtain that|Qn(wn = a)| ≤ |In| − (|Wn| − 1) for any a ∈ Wn establishing the laim.Using this result, the omplexity of the marginalization in (A.5) an be hara -terized. For the sake of a simple dis ussion, we assume that |In| = L for all n ∈ T andthat |Wn| = K for all n ∈ T . Using Lemma A.2.1 and after dening the system spe i 1The results for the ase k /∈ T an be derived a ordingly.2It is worth pointing out that QT (wT ) an be onstru ted easily sin e the mapping fun tions mnare assumed to be known for all n ∈ T .

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 146parameter F = L−K + 1, we are able to on lude that in our ase |Qn(wn)| ≤ F forany n ∈ T . We furthermore assume that the elementary operations (like additions,multipli ations, omparisons, look-ups, et .) are of onstant omplexity, i.e., of O(1).Spe i ally, we assume that p(iT ) an be determined with a omplexity of O(1), e.g.,that it an be approximated or simulated with onstant omplexity, or that it an belooked-up.In the following, we onsider the newly derived expression for the marginaliza-tion (A.7). In the ase where mk(ik = l) = wk holds, it is easy to see that, in the worst- ase, F |T |−1 instan es of p(iT ) are required throughout the al ulation and that aroundF |T |−1 additions have to be performed. Thus, around 2 ·F |T |−1 elementary operationshave to be performed orresponding to a omputational omplexity of O(F |T |−1). Inthe ase where mk(ik = l) = wk does not hold, the result of the marginalizationbe omes zero, without any further al ulations, and the omputational omplexityderives to be of O(1) for testing the ase alone.A.3 Complexity of Optimal De odingIn this se tion, whi h uses the same denitions as the previous appendi es, we dis ussthe omplexity of optimal de oding as required, e.g., in (2.3). Spe i ally, we want to onsider the al ulation

uk(wT ) =

|Ik|−1∑

ik=0

uk,ik · p(ik|wT ). (A.8)We observe that the al ulation in (A.8) requires that uk,ik and p(ik|wT ) have to bedetermined, multiplied and summed-up for all possible realizations of ik ∈ Ik wherep(ik|wT ) an be derived from p(iS) using the e ient marginalization des ribed inAppendix A.2.In order to use the results derived in Appendix A.2, we restri t ourselves tothe ase where k ∈ T , i.e., where S = T .3 For a simplied omplexity analysis, wefurthermore assume that |In| = L, |Wn| = K and |Qn(wn)| ≤ F for all n ∈ T whereF = L−K + 1. Elementary operations (like additions, multipli ations, omparisons,look-ups, et .) are assumed to be of onstant omplexity, i.e., of O(1). Spe i ally, weassume that uk,ik and p(iT ) an be determined with a omplexity of O(1), e.g., thatthey an be approximated or simulated with onstant omplexity, or that they an belooked-up.3The results for the ase k /∈ T an be derived a ordingly.

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 147Using the results of Appendix A.2, we are able to state that the omputational omplexity for deriving p(ik|wT ), as required in (A.8), is either of O(F |T |−1), or ofO(1), depending on whether ik is mapped onto wk, or not. In order to determine theoverall omplexity, we noti e that the summation in (A.8) has to be performed over allik ∈ Ik. Therefore, we an employ Lemma A.2.1 in Appendix A.2 to determine howoften it will be true (at most) that mk(ik) = wk and, thus, how often (at most) the al ulation of p(ik|wT ) in (2.4) has to be performed. We on lude that this al ulationhas to be performed (at most) F times. The test if mk(ik) = wk is true, the look-up of uk,ik as well as the multipli ation in (A.8) an be negle ted ompared to the omplexity of al ulating p(ik|wT ) in total F times. Therefore, we are able to on ludethat al ulating one estimate has a omputational omplexity of O(F |T |).A.4 Distortion Cal ulationIn this se tion, whi h again uses the same denitions as the previous appendi es,we shall show that the overall distortion asso iated with ea h sour e k ∈ N an bedes ribed by the sum

E(Uk − Uk)2 = E(Uk − Uk)

2+ E(Uk − Uk)2 (A.9)where E(Uk −Uk)

2 is the distortion aused by the quantization stage and E(Uk −

Uk)2 is the distortion aused by the index assignment stage.To do so, we start with the denition of the expe tation value and obtain

E(Uk − Uk)2 =

∑

∀wT ∈WT

p(wT )E(Uk − Uk)2|wT

(a)=∑

∀iT ∈IT

p(iT )E(Uk − Uk)2|iT ,where the equality (a) holds due to the fa t that the index assignments mn aresurje tive fun tions for all n ∈ T and, thus, the summation over all wT ∈ WT overs the same observation spa e as the summation over all iT ∈ IT . Based onthis observation, equation (A.9) an easily be established by showing that

E(Uk − Uk)2|iT = E(Uk − Uk)

2|iT + E(Uk − Uk)2|iT . (A.10)The denition of the onditional expe tation allows us to rewrite

E(Uk − Uk)2|iT

(a)=

∫ +∞

uk=−∞

(uk(wT )− uk)2p(uk|iT )duk

(b)=

∫ +∞

uk=−∞

(uk(wT )− uk)2p(uk|ik)duk

(c)=

1

p(ik)

∫ bk(ik+1)

uk=bk(ik)

(uk(wT )− uk)2p(uk)duk, (A.11)

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 148where the denition of the onditional expe tation is used in (a) together with the fa tthat wn = mn(in) for all n ∈ T , equality (b) is valid sin e ik is known if iT is knownand, thus, p(uk|iT ) = p(uk|ik), and equality ( ) holds due to (A.4). Assuming thatuk,ik =

∫ bk(ik+1)

uk=bk(ik)uk · p(uk)duk

∫ bk(ik+1)

uk=bk(ik)p(uk)duk

=

∫ bk(ik+1)

uk=bk(ik)uk · p(uk)duk

p(ik), (A.12)i.e., that the re onstru tion value of the quantizer is the entroid of the quantizationregion, it is possible to show that the required integration an be split into two parts4su h that

∫ bk(ik+1)

uk=bk(ik)

(uk(wT )− uk)2p(uk)duk =

∫ bk(ik+1)

uk=bk(ik)

(uk,ik − uk)2p(uk)duk

+(uk(wT )− uk,ik)2p(uk,ik),

(A.13)where p(uk,ik) = p(ik). Plugging (A.13) into (A.11), we obtainE(Uk − Uk)

2|iT =1

p(ik)

∫ bk(ik+1)

uk=bk(ik)

(uk,ik − uk)2p(uk)duk + (uk(wT )− uk,ik)

2

= E(Uk − Uk)2|ik+ E(Uk − Uk)

2|iT (A.14)dire tly establishing (A.10) and, thus, the desired result in (A.9).A.5 Complexity of the Index-Reuse OptimizationThe omputational omplexity of the index-reuse algorithm presented in Algorithm 1 an be bounded by evaluating how often, in the worst- ase, the operations within theinnermost of the nested loops have to be performed.The outermost loop is exe uted for ea h of the L −K merging that have to beperformed to obtain mappings with K output odewords from the initial mapping withL output odewords. The se ond loop is exe uted for all onsidered en oders n ∈ Ω,i.e., in total |Ω| times. Finally, the innermost loop runs through all possibilities of hoosing 2 out of k odewords for the merging, i.e., we have (k

2

)= k!

2!(k−2)!= 1

2(k2 − k)possibilities. In the worst- ase, i.e., in the initial ase where k = L, we obtain 1

2(L2−L)possibilities. Thus, the operations within the innermost loop have to be performed

|Ω|(L−K)12(L2 − L) = 1

2|Ω|(L−K)L2 − 1

2|Ω|(L−K)L times in the worst- ase.Now, to determine the overall omplexity of the algorithm the omplexities ofthe merging operation en = g(fn, a, b), the omplexity of the distortion al ulation4This an be a hieved by substituting uk(wT ) = uk,ik

+ dk, where dk = uk(wT )− uk,ik, su h that

(uk(wT )−uk)2 an be expressed as (uk,ik+dk−uk)2 whi h derives to (uk,ik

−uk)2+2dk(uk,ik−uk)+d2

k.

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 149d = dd(Ψ, En) and the test if d < d∗ have to be determined. Assuming that themerging and the test an be performed with a onstant omputational omplexity ofO(1), it remains to determine the omplexity of al ulating dd(Ψ) given the urrent setof mapping fun tions En. This al ulation requires the al ulation of dd(n) a ordingto (2.8) whi h, in turn, requires the al ulation of the estimate un(wΩ) a ordingto (2.3) for all L|Ψ| possible realizations of iΨ ∈ IΨ.5 Assuming that n ∈ Ω, i.e.,that |Ψ| = |Ω|, the result of Appendix A.3 dire tly applies here and we an statethat al ulating one estimate has a omputational omplexity of O(F |Ω|).6 Thus, inthe most straightforward implementation, the omputational omplexity of al ulatingdd(n) a ording to (2.8) is of O((LF )|Ω|).Based on the presented results and after substituting F by L−K+1, as derived inAppendix A.2 for surje tive mapping fun tions, we are able to on lude that the overall omputational omplexity of the index-reuse algorithm is of O(|Ω|L|Ω|+2(L−K)|Ω|+1)showing an exponential growth with |Ω|.A.6 E ient Sub-Optimal De oding and its Com-plexityIn this se tion, whi h uses the same denitions as the previous appendi es, we shallelaborate on how a sour e fa torization based on CCREs a ording to (2.10) and (2.11) an be used for e ient de oding. This shall be a hieved by assuming that thefa torization (2.10) also holds for the dis rete ase7 su h that

p(i) =

M∏

m=1

fm(iSm) =

M∏

m=1

p(iAm|iBm

), (A.15)whi h an be exploited for s alable de oding as shown in [BT06 for a similar systemsetup. Sin e the de oder design onsidered in this work follows the same prin iples,we shall fo us on the dieren es resulting from system spe i properties.In Se tion 2.2 we have shown that the al ulation of the optimal estimate un(w)a ording to (2.3) for n ∈ N requires the al ulation of the probabilitiesp(in = l|w) = γ ·

∑

∀i∈I:in=l

p(w, i)(a)= γ ·

∑

∀i∈Q(w):in=l

p(i) (A.16)5There are more e ient ways to al ulate dd(n) based on intermediate results. However, due tola k of spa e, the dis ussion is negle ted here.6The results for the ase where n /∈ Ω an be derived a ordingly.7This assumption is plausible sin e Un → In forms a Markov hain, n ∈ N .

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 150where the equality (a) is due to the result derived in Appendix A.2. Repla ing p(i) byits approximation p(i) as given by the fa torization in (A.15), we obtain the followingapproximationp(in = l|w) = γ ·

∑

∀i∈Q(w):in=l

p(i) = γ ·∑

∀i∈Q(w):in=l

(M∏

m=1

fm(iSm)

)

, (A.17)whi h an be al ulated e iently for all l ∈ In and for all n ∈ N by running the sum-produ t algorithm on the fa tor graph representation of the fa torization in (A.15).For a general treatment of fa tor graphs and the sum-produ t algorithm please referto [KFL01 or to [BT06 where a similar system setup is dis ussed.In order to provide the fundamentals, we in lude a brief review here. A fa torgraph is a bipartite graph that onsists of variable and fun tion nodes and expresseshow a (global) fun tion fa tors into (lo al) fun tions. The variable nodes representthe arguments of the fun tions and the fun tion nodes the (lo al) fun tions itself. Thesum-produ t algorithm allows us to perform the (global) marginalization in (A.17)based on (lo al) marginalization of the following typeµm→n(l) =

∑

∀iSm∈QSm(wSm ):in=l

(

fm(iSm)

∏

g∈Sm:g 6=n

µg→m

)

, (A.18)whi h are performed in a stru tured way for all n ∈ Sm and m ∈ M. Follow-ing the intuition in [KFL01, the results of the marginalization in (A.18) for l =

0, 1, . . . , |In| − 1 an be seen as messages represented a ve tor µm→n = (µm→n(0),

µm→n(1), . . . , µm→n(|In| − 1)) that are sent from the fun tion node m ∈ M to thevariable node n ∈ N for further pro essing. Similarly, the inputs of the marginalizationin (A.18) an also be seen as messages µg→m that were re eived at the fun tion nodem originating from some variable nodes g ∈ N . Those messages represent the produ t

µg→m(k) =∏

∀h∈M:g∈Sh,h 6=m

µh→g(k) (A.19)for k = 0, 1, . . . , |Ig| − 1 and, thus, µg→m = (µg→m(0), µg→m(1), . . . , µg→m(|Ig| −

1)). Using this abstra tion, the te hniques des ribed in [KFL01 an be dire tlyapplied here giving rise to an e ient al ulation of (A.17).8 In parti ular this is8It is worth pointing out that the expression in (A.18) is optimized to minimize the marginalization omplexity by using knowledge about the re eived odewords. This in turn means that in thisparti ular setup the fun tion nodes have to be initialized and not the variable nodes as in onventionalimplementations. Spe i ally, the fun tion nodes are initialized by dening the set QSm(wSm

) usingknowledge of wSmfor m = 1, 2, . . . , M and the variable nodes are initialized with trivial messages.

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 151a hieved by running an appropriate message passing algorithm9 along the fa tor graphrepresentation of (A.15); and depending on whether the message passing pro edureterminates or not, i.e., on whether the fa tor graph is y le-free or not,10 the exa t oran approximated value of p(in = l|w) is obtained simultaneously for all l ∈ In andn ∈ N .Sin e the presented de oding s heme is based on message passing, its overall omplexity an be analyzed by onsidering all messages that are reated during thede oding pro ess and jointly evaluating their omplexity. We noti e that the al u-lation of the messages at ea h fun tion node m ∈ M a ording to (A.18) requiresa marginalization of the same type as dis ussed in Appendix A.2. Assuming thatfurthermore Ln = L and Kn = K for n = 1, 2, . . . , N and that the omplexity ofelementary operations are the same as stated in Appendix A.2, the derived resultsdire tly apply here and we are able to on lude that the messages at the fun tion nodesm ∈ M an be reated with a omputational omplexity of O(F |Sm|). Consideringthe messages reated at the variable nodes n ∈ N a ording to (A.19), it is easy tosee that the messages an be derived with a omputational omplexity of O(L). Wenoti e that the omplexity of al ulating the messages at the fun tion nodes is higherthan at the variable nodes sin e generally |Sm| > 1 for all (but maybe one) m ∈ M,i.e., the omplexity of al ulating the messages at the variable nodes an be negle tedhere. In order to provide an expression for the omplexity, we have to distinguishbetween two ases.In the ase where the fa tor graph is y le-free, the e ient forward-ba kwardalgorithm, see [KFL01, an be used for message passing and only one message needsto be passed along ea h edge and in ea h (edge) dire tion within the graph. Assumingthat |Sm| ≤ S for all m ∈ M, the al ulation in (A.18) has to be performed at mostM ·S times leading to a omputational omplexity of O(MSF S). In the ase where thegraph has y les, the message passing has to be performed in an iterative way for anreasonable amount of iterations T >> 1, see [KFL01, and we obtain a omputational omplexity of O(TMSF S).

9For fa tor graphs without y les the e ient forward-ba kward algorithm an be employed,see [KFL01.10Using the result in [BT06 this an be ensured if |Bm| = 1 for m = 1, 2, . . . , M .

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 152A.7 Complexity of Sour e-Optimized ClusteringThe omplexity of sour e-optimized lustering des ribed in Se tion 2.4.2 shall bedis ussed next.11 In [JMF99 it is shown that hierar hi al lustering, upon whi hthe presented pro edure is based on, has a omputational omplexity of O(N2 log N).However, these results do not dire tly hold for sour e-optimized lustering sin e forea h step of the pro edure, i.e., for ea h merging performed, the dierential KLDbenet ∆D′(Λ′k, Λ

′l) a ording to (2.16) has to be al ulated. Looking at (2.16) inmore detail, we observe that merging luster Λ′

k and Λ′l requires the al ulation of theKLD benet ∆D(Λ′

k∪Λ′l, ∅) a ording to (2.14) whi h, in turn, requires the al ulationof the determinant for the orresponding ovarian e matrix RΛ′

k∪Λ′

l. Using the generaldenition of determinants, it is easy to see that it an be al ulated using Gaussianelimination. Assuming that the matrix, whose determinant has to be derived, is of size

N ×N , then the omplexity of the Gaussian elimination and, thus, also of al ulatingthe determinant is of O(N3), e.g., see [Str69.12 Sin e |Λ′k ∪ Λ′

l| is always smaller orequal to N , the matrix RΛ′k∪Λ′

lis (at most) of size N × N and, thus, the al ulationof |RΛ′

k∪Λ′

l| is of O(N3). Assuming that the omplexity of performing one mergingstep within the lassi al hierar hi al lustering algorithm is of O(1), i.e., the minimumpossible, then the number of merging operations to be performed an be boundedby O(N2 log N). For the sour e-optimized lustering pro edure this means that its omputational omplexity is of O(N5 log N).A.8 Complexity of Sour e-Optimized LinkingThis se tion addresses the omplexity of onstru ting the sour e-optimized fa toriza-tion presented in Se tion 2.4.3. Using the result in [Nep01, we are able to on lude thatthe dire ted spanning tree algorithm, upon whi h the presented pro edure is based on, an be implemented with a omplexity of O(C log C). However, beside this, also the omplexity for prepro essing the data required to initialize the dire ted spanning treealgorithm needs to be onsidered. In parti ular, this means that the link osts have tobe determined before the dire ted spanning tree algorithm an be run. In total thereare C2 link osts ck,l representing the KLD benet asso iated with establishing a linkbetween luster Λk, k ∈ Γ, and Λl, l ∈ Γ, that have to be al ulated a ording to (2.25).11The main goal at this point is to show that sour e-optimized lustering is of polynominal omplexity ( onsidering the number of sour es N) and not to nd an exa t expression for the omplexity of Algorithm 2. This would ex eed the s ope of this work.12There are more e ient ways to al ulate the determinant of a matrix, e.g., see [Str69, but it issu ient for our purposes to assume that the Gaussian elimination is used.

APPENDIX A. SOURCE-OPTIMIZED CLUSTERING 153In order to determine this link ost all possible ombinations of P ′k ∈ T (A, Λk) and

Q′l ∈ T (B, Λl) have to be evaluated as stated in (2.24). It is easy to see that thenumber of su h ombinations is given by the produ t between |T (A, Λk)| and |T (B, Λl)|where |T (A, Λk)| =

(|Λk|A

) and |T (B, Λl)| =(|Λl|B

). For simpli ity, we assume in thefollowing that A = B = S2and that |Λc| = S for c = 1, 2, . . . , C.13 After simplemathemati al manipulation we are able to on lude that there are at most 2S log2 Ssu h ombinations. It remains to derive the omplexity of al ulating the argumentin (2.24), i.e., the omplexity of al ulating ∆D∗(P ′

k,Q′l) a ording to (2.22) whi h is learly determined by the omplexity of al ulating ∆D(P ′

k∪Q′l, ∅) a ording to (2.14).Using the same arguments as in Appendix A.7, we an state that the omplexity of al ulating the determinant |RP ′

k∪Q′

l| in (2.14) is of O(|P ′

k ∪ Q′l|

3), i.e., it is of O(S3)using the same assumptions as before. Putting everything together, we are able to on lude that the al ulation of all link ost is of O(C22S log2 SS3) = O(C22(3+S) log2 S).The omputational omplexity of the overall sour e-optimized linking pro edure isthen given by the sum of the derived omplexities, i.e., of the dire ted spanning treealgorithm and the link ost al ulation. Sin e the omplexity of the algorithm anbe negle ted here, we on lude that the omplexity of sour e-optimized linking is ofO(C22(3+S) log2 S) only being feasible for small values of S.

13For other ongurations the following results an be derived a ordingly.

Appendix BDiophantine Distributed Sour eCodingB.1 De odabilityThe su ient part is established by onsidering the following two laims: (a) If thede oded set I(w) does not have more than a single member for all odeword tuplesw ∈W that result from an index tuple i ∈ A, then A is de odable. (b) A su ient ondition for the requirement in (a) is that all index tuples i in the admissible set Aneed to be en oded onto dierent odeword tuples.We rst prove laim (a). Let w(0) ∈W be an arbitrary odeword tuple and letI−1(w(0)) be the set of index tuples mapped onto it. As required above, we assume that|I(w(0))| ≤ 1. Using the denition of the de oded set, we obtain |I−1(w(0))∩A| ≤ 1.This means that |I−1(w(0))∩A| is either zero, or one. If |I−1(w(0))∩A| = 0, it mustfollow that I−1(w(0)) ∩A = ∅. This an only happen if there is no index tuple in theadmissible set A whi h is mapped onto the onsidered odeword w(0), a ase, whi hdoes not ae t the de odability of A. For the ase where |I−1(w(0))∩A| = 1, we knowthat there is exa tly one index tuple i(0) in the admissible set A whi h is also mappedonto w(0). This index tuple is de odable sin e it an be re overed by the interse tioni(0) = I−1(w(0)) ∩ A. The same holds for any other w(0) ∈W establishing laim (a).Then, we prove laim (b). A ording to the laim, all index tuples in A aremapped onto dierent odeword tuples W . This in turns means that there exist nopair (or any larger subset) of indi es in A that are mapped onto the same odewordtuple w(0) ∈W . Therefore I(w(0)) = I−1(w(0)) ∩ A will always be (stri tly) smallerthan two establishing laim (b). Together with the proof of laim (a) this proves the154

APPENDIX B. DIOPHANTINE DISTRIBUTED SOURCE CODING 155su ient part.The ne essary part is obvious sin e, in this ase, some of the index tuples in theadmissible set are mapped onto the same odeword and, thus, the de oded set for the orresponding odewords has more than one member. This establishes Lemma 3.4.1.B.2 Main Diagonal Distan e d(u, δ)Let u = (u1, u2, . . . , uN)T ∈ RN . A ording to (3.3), the main diagonal is des ribed bythe equation δ : u = 0N + t · 1N , t ∈ R. The minimum distan e d(u, δ) between thepoint u and the main diagonal δ an be al ulated if the point h = (h1, h2, . . . , hN)T ∈

RN lo ated on δ is known that minimizes the distan e d(u,h). For the sake of brevity,we fo us on the ase where the standard Eu lidean distan e is used to quantify d(u, δ).1Sin e h lies on δ, we rst observe that h an be expressed as h = h · 1T

N =

(h, . . . , h)T , h ∈ R. Furthermore, it is possible to argue that h needs to lie ona line through u perpendi ularly interse ting δ. One observes that ve tors v =

(v1, v2, . . . , vN)T ∈ RN perpendi ular to δ must result in a zero s alar ve tor produ t

v · 1N = 0. We on lude that vN = −∑N−1

n=1 vn is veried by su h lines. Using bothobservations, it be omes possible to state that(h, . . . , h)T = (u1, u2, . . . , uN)T + t · (v1, . . . , vN−1,−

N−1∑

n=1

vn)T (B.1)must hold for some t ∈ R. Reformulating (B.1), we obtain the following equationsystem: (a) h = un + t · vn , for n = 1, . . . , N − 1,(b) h = uN − t ·∑N−1

j=m vm.(B.2)After solving equation (a) for t and subsequent substitution in (b), we obtain a systemof N − 1 equations that an be added-up resulting in the single equation:

N−1∑

n=1

h =N∑

n=1

(

un +1

∑N−1m=1 vm

(uN − h) · vN

)

. (B.3)Solving equation (B.3) for h, we obtainh =

1

N·

N∑

n=1

un. (B.4)1The following results an also be established for the ase where an wrapped ve tor spa e asdes ribed in Se tion 3.3.2 is onsidered.

APPENDIX B. DIOPHANTINE DISTRIBUTED SOURCE CODING 156At this point it is worth pointing out that, interestingly, h orresponds to the averagevalue of the ve tor oordinates in u providing an alternative interpretation for thepoint h = h · 1TN . Using the denition of the distan e (3.4), we obtain

d(u, δ) =

√√√√

N∑

n=1

(

un −1

N

N∑

m=1

um

)2

= . . . =

√√√√

N∑

n=1

u2n −

1

N

(N∑

n=1

un

)2 (B.5)establishing the desired result.For the ase of N = 2, i.e., given the symbol ve tor u = (u1, u2)T , it is possibleestablish an important onne tion between the Eu lidean distan e ||u1−u2||2 = (u1−

u2)2 of the ve tor's omponents and the diagonal distan e d(u, δ). In parti ular, byreformulating (B.5) it an be shown that

d(u, δ) =||u1 − u2||2

2, (B.6)whi h makes the diagonal distan e d(u, δ) linear proportional to the Eu lidean distan e

||u1−u2||2. This is remarkable be ause ||u1−u2||2, whi h is ommonly used as a design riterion (e.g., see [PR99, PR03), gets a dire t geometri interpretation whi h, in ourwork, is a tively used for the ode design and de oding purposes.B.3 The Segment Distan e d(c) and its PropertiesUsing the denition of the diagonal segment D(c) at position c = (c1, c2, . . . , cN−1)T ∈

ZN−1 in (3.8), it be omes evident that all index tuples i = (i1, i2, . . . , iN)T ∈ D(c)have the same diagonal distan e d(i, δ). We denote this distan e as d(c). Parti ularly onsidering the referen e point b = (0, c1, c2, . . . , cN−1)

T in the plane with i1 = 0, weare able to determine the distan e d(c) by al ulating the distan e d(b, δ) a ordingto (B.5). Thus, we obtaind(c) = d(b, δ) =

√√√√

N−1∑

n=1

c2n −

1

N

(N−1∑

n=1

cn

)2

. (B.7)Considering a s enario with N en oders and setting d(c) = r, for some (positive) onstant r ∈ R, the equation (B.7) des ribes a (N−1)−dimensional quadrati surfa e,a quadri , whi h in our parti ular ase be omes a (N−1)−dimensional ellipsoid ( om-pare Figure 3.9). We observe, without further proof, that the ellipsoid is rotationsymmetri around its semi-major axis with referen e point 0TN−1 and dire tion ve tor

1TN−1.22For example, this an be shown by representing d(c) in normal matrix form and analyzing theresulting Eigenve tors.

Referen es[ACLY00 R. Ahlswede, N. Cai, S.-Y. R. Li, and R. W. Yeung. Network informationow. IEEE Transa tions on Information Theory, 46(4):12041216, July2000.[ADHW06 C. Ahn, A. C. Doherty, P. Hayden, and A. J. Winter. On the distributed ompression of quantum information. IEEE Transa tions on InformationTheory, 52(10):43494357, O tober 2006.[AMB09 J. Almeida, G. Maierba her, and J. Barros. Low- omplexity indexassignments for se ure quantization. In Pro eedings of the 43rd AnnualConferen e on Information S ien es and Systems (CISS'09), BaltimoreMD, USA, Mar h 2009.[ASSC02 I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayir i. A surveyon sensor networks. IEEE Communi ations Magazine, 40(8):102114,August 2002.[AU97 A. V. Aho and J. D. Ullman. Foundations of Computer S ien e, C Edition.W. H. Freeman & Co., New York, NY, USA, 1997.[BM01 J. Baj sy and P. Mitran. Coding for the slepian-wolf problem withturbo odes. In Pro eedings of the IEEE Global Tele ommuni ationsConferen e (GLOBECOM'01), pages 14001404, San Antonio, Texas,USA, November 2001.[BT06 J. Barros and M. Tue hler. S alable de oding on fa tor trees: apra ti al solution for wireless sensor networks. IEEE Transa tions onCommuni ations, 54(2):284294, February 2006.[BZV96 T. Berger, Z. Zhang, and H. Viswanathan. The CEO problem. IEEETransa tions on Information Theory, 42:887902, May 1996.157

REFERENCES 158[CK03 C.-Y. Chong and S. P. Kumar. Sensor networks: evolution, opportunities,and hallenges. Pro eedings of the IEEE, 91(8):12471256, August 2003.[CL65 Y. J. Chu and T. H. Liu. On the shortest arbores en e of a dire ted graph.S ien e Sini a, 14:13961400, 1965.[CMB11a S. B. Cruz, G. Maierba her, and J. Barros. Cooperative sour e-network oding, July 2011. Submitted to the IEEE Transa tions on SignalPro essing.[CMB11b S. B. Cruz, G. Maierba her, and J. Barros. Joint sour e-network odingfor large-s ale sensor networks, August 2011. A epted for publi ation inthe IEEE International Symposium on Information Theory (ISIT'11).[CME05 T. P. Coleman, M. Medard, and M. Eros. Towards pra ti al minimum-entropy universal de oding. In Pro eedings of the Data CompressionConferen e (DCC'05), pages 3342, Snowbird, Utah, USA, Mar h 2005.[Coh80 H. Cohn. Advan ed Number Theory. Dover Publi ations, In ., Mineola,NY, USA, 1st edition, 1980.[Csi82 I. Csiszar. Linear odes for sour es and sour e networks: Error exponents,universal oding. IEEE Transa tions on Information Theory, 28(4):585592, July 1982.[CSPR99 J. Chou, S. Sandeep Pradhan, and K. Ram handran. On the dualitybetween distributed sour e oding and data hiding. In Conferen eRe ord of the Thirty-Third Asilomar Conferen e on Signals, Systems, andComputers, volume 2, pages 15031507, Pa i Grove, CA , USA, O tober1999.[CT91 T. M. Cover and J. Thomas. Elements of Information Theory. John Wileyand Sons, In ., 1991.[CVA02 J. Cardinal and G. Van Ass he. Joint entropy- onstrained multiterminalquantization. In Pro eedings of the IEEE International Symposium onInformation Theory (ISIT'02), Lausanne, Switzerland, June 2002.[CZBW04 J. Chen, X. Zhang, T. Berger, and S. B. Wi ker. An upper bound onthe sum-rate distortion fun tion and its orresponding rate allo ations hemes for the CEO problem. Spe ial Issue of JSAC, On FundamentalPerforman e of Wireless Sensor Networks, 22(6):977 987, August 2004.

REFERENCES 159[DN97 C. R. Dietri h and G. N. Newsam. Fast and exa t simulation of stationaryGaussian pro esses through ir ulant embedding of the ovarian e matrix.SIAM Journal on S ienti Computing, 18(4):10881107, July 1997.[Edm67 J. Edmonds. Optimum bran hings. J. Resear h of the National Bureauof Standards, 71B:233240, 1967.[FG87 T. J. Flynn and R. M. Gray. En oding of orrelated observations. IEEETransa tions on Information Theory, IT-33(6):773787, November 1987.[G. 73 G. D. Forney, Jr. The Viterbi algorithm. Pro eedings of the IEEE,61(3):268278, Mar h 1973.[Geo03 L. Georgiadis. Arbores en e optimization problems solvable by Edmonds'algorithm. Theor. Comput. S i., 301(1-3):427437, May 2003.[GF01 J. Gar ia-Frias. Compression of orrelated binary sour es using turbo odes. Communi ations Letters, IEEE, 5(10):417 419, O tober 2001.[GSV05 D. Guo, S. Shamai, and S. Verdú. Mutual information and minimummean-square error in Gaussian hannels. IEEE Transa tions on Informa-tion Theory, 51:12611283, April 2005.[HKM+03 T. Ho, R. Koetter, M. Medard, D. R. Karger, and M. Eros. The benetsof oding over routing in a randomized setting. In Pro eedings of theIEEE International Symposium on Information Theory (ISIT'03), page442, Kanagawa, Japan, June 2003.[HMEK04 T. Ho, M. Medard, M. Eros, and R. Koetter. Network oding for orrelated sour es. In Pro eedings of the Conferen e on InformationS ien es and Systems (CISS'04), Prin eton, NJ, USA, Mar h 2004.[JAI97 A. Kh. Al Jabri and S. Al-Issa. Zero-error odes for orrelated informationsour es. In Pro eedings of the 6th IMA International Conferen e onCryptography and Coding, pages 1722, London, UK, 1997. Springer-Verlag.[JMF99 A. K. Jain, M. N. Murty, and P. J. Flynn. Data lustering: a review.ACM Computing Surveys, 31(3):264323, September 1999.[JN84 N. Jayant and P. Noll. Digital Coding of Waveforms. Prenti e Hall, 1984.

REFERENCES 160[KFL01 F. R. Ks his hang, B. Frey, and H.-A. Loeliger. Fa tor graphs andthe sum-produ t algorithm. IEEE Transa tions on Information Theory,47(2):498519, February 2001.[KM03 R. Koetter and M. Médard. An algebrai approa h to network oding.IEEE/ACM Transa tions on Networking, 11(5):782795, O tober 2003.[Knu05 D. E. Knuth. The Art of Computer Programming, Volume 4, Fas i le3: Generating All Combinations and Partitions. Addison-Wesley Profes-sional, 1st edition, 2005.[LGS04 K. Lajnef, C. Guillemot, and P. Siohan. Distributed oding of threesour es using pun tured turbo odes. In Pro eedings of the 2004 IEEE 6thWorkshop on Multimedia Signal Pro essing, pages 307310, Siena, Italy,September 2004.[LMH+07 A. Lee, M. Médard, K. Z. Haigh, S. Gowan, and P. Rubel. Minimum- ostsubgraphs for joint distributed sour e and network oding. In Pro eedingsof the Third Workshop on Network Coding, Theory, and Appli ations(NETCOD'07), San Diego, CA, USA, January 2007.[LRM+06 D. S. Lun, N. Ratnakar, M. Medard, R. Koetter, D. R. Karger, T. Ho,E. Ahmed, and F. Zhao. Minimum- ost multi ast over oded pa ketnetworks. IEEE Transa tions on Information Theory, 52(6):26082623,June 2006.[LXG02 A. D. Liveris, Zixiang Xiong, and C.N. Georghiades. Compression ofbinary sour es with side information at the de oder using LDPC odes.IEEE Communi ations Letters, 6(10):440442, O tober 2002.[LYC03 S.-Y. R. Li, R. W. Yeung, and N. Cai. Linear network oding. IEEETransa tions on Information Theory, 49(2):371381, February 2003.[MB05 G. Maierba her and J. Barros. Low- omplexity oding for the CEOproblem with many en oders. In Twenty-sixt Syposium on InformationTheory in the Benelux, Brussels, Belgium, May 2005.[MB06a G. Maierba her and J. Barros. On the Kullba k-Leibler distan e and themean square distortion of mismat hed distributed quantizers. In Pro eed-ings of the 3rd International Workshop on Mathemati al Te hniques andProblems in Tele ommuni ations, Leiria, Portugal, September 2006.

REFERENCES 161[MB06b G. Maierba her and J. Barros. Sour e-optimized lustering for distributedsour e oding. In Pro eedings of the IEEE Global Tele ommuni ationsConferen e (GLOBECOM'06), San Fran is o, CA, USA, November 2006.[MB07a G. Maierba her and J. Barros. Code design for the distributed s alarquantization problem based on diophantine analysis. In Pro eedings ofthe IEEE Winter S hool on Coding and Information Theory (WSIT'007),La Colle sur Loup, Fran e, Mar h 2007.[MB07b G. Maierba her and J. Barros. Diophantine index assignments fordistributed sour e oding. In Pro eedings of the 2007 IEEE InformationTheory Workshop (ITW 2007) - Frontiers in Coding, Lake Tahoe,California, USA, September 2007.[MB09 G. Maierba her and J. Barros. Low- omplexity oding and sour e-optimized lustering for large-s ale sensor networks. ACM Transa tionson Sensor Networks, 5(3):132, May 2009.[MB11 G. Maierba her and J. Barros. Diophantine distributed sour e oding,July 2011. Submitted to the IEEE Transa tions on Information Theory.[MBM09 G. Maierba her, J. Barros, and M. Médard. Pra ti al sour e-networkde oding. In Pro eedings of the IEEE International Symposium onWireless Communi ation Systems (ISWCS'09), Siena, Italy, September2009.[MF90 M. Mar ellin and T. Fis her. Trellis oded quantization of memoryless andGauss-Markov sour es. IEEE Transa tions on Communi ations, 38(1):8293, January 1990.[Nep01 A. Nepomnias haya. E ient implementation of Edmonds' algorithmfor nding optimum bran hings on asso iative parallel pro essors. InPro eedings of the Eighth Intern. Conf. on Parallel and DistributedSystems (ICPADS'01), KyongJu City, Korea, June 2001.[PBAT08 F. Pereira, C. Brites, J. As enso, and M. Tagliasa hi. Wyner-ziv video oding: A review of the early ar hite tures and further developments. InPro eedings of the 2008 IEEE International Conferen e on Multimediaand Expo, pages 625628, April 2008.[Poo94 H. V. Poor. An Introdu tion to Signal Dete tion and Estimation. Springer-Verlag, 1994.

REFERENCES 162[PR99 S. S. Pradhan and K. Ram handran. Distributed sour e oding usingsyndromes (DISCUS): design and onstru tion. In Pro eedings of theIEEE Data Compression Conferen e (DCC'99), pages 158167, Snowbird,UT, USA, Mar h 1999.[PR03 S. S. Pradhan and K. Ram handran. Distributed sour e oding usingsyndromes (DISCUS): design and onstru tion. IEEE Transa tions onInformation Theory, 49(3):626643, Mar h 2003.[PR05 S. S. Pradhan and K. Ram handran. Generalized oset odes for dis-tributed binning. IEEE Transa tions on Information Theory, 51(10):3457 3474, O tober 2005.[PS06 Palomar D. P. and Verdú S. Gradient of mutual information in linearve tor Gaussian hannels. IEEE Transa tions on Information Theory,52(1):141 154, January 2006.[RJCE06 A. Ramamoorthy, K. Jain, P.A. Chou, and M. Eros. Separatingdistributed sour e oding from network oding. IEEE Transa tions onInformation Theory, 52(6):27852795, June 2006.[RMZG03 D. Rebollo-Monedero, R. Zhang, and B. Girod. Design of optimalquantizers for distributed sour e oding. In Pro eedings of the DataCompression Conferen e (DCC'03), pages 13 22, Mar h 2003.[Ros10 K. H. Rosen. Elementary Number Theory. Pearson Edu ation, In .,Boston, MA, USA, 6th edition, 2010.[SCX06 V. M. Stankovi , S. Cheng, and Z. Xiong. On dualities in multiterminal oding problems. IEEE Transa tions on Information Theory, 52(1):307315, January 2006.[Ser00 S. D. Servetto. Latti e quantization with side information. In Pro eedingsof the IEEE Data Compression Conferen e (DCC'00), Snowbird, UT,USA, Mar h 2000.[Sha56 C. Shannon. The zero error apa ity of a noisy hannel. IRE Transa tionson Information Theory, 2(3):819, September 1956.[SS02 A. S aglione and S. D. Servetto. On the interdependen e of routing anddata ompression in multi-hop sensor networks. In Pro eedings of theACM MobiCom, Atlanta, GA, USA, September 2002.

REFERENCES 163[Str69 V. Strassen. Gaussian elimination is not optimal. Numer. Math., 13:354356, 1969.[SW73 D. Slepian and J. K. Wolf. Noiseless oding of orrelated informationsour es. IEEE Transa tions on Information Theory, IT-19(4):471480,July 1973.[SY01 L. Song and R. W. Yeung. Network information ow-multiple sour es. InPro eedings of the IEEE International Symposium on Information Theory(ISIT'01), Washington, DC, USA, June 2001.[Ung82 G. Ungerboe k. Channel oding with multilevel/phase signals. IEEETransa tions on Information Theory, 28(1):5567, January 1982.[VB97 H. Viswanathan and T. Berger. The quadrati Gaussian CEO problem.IEEE Transa tions on Information Theory, 43(5):15491559, September1997.[War63 J. Ward. Hierar hi al grouping to optimize an obje tive fun tion. Journalof the Ameri an Statisti al Asso iation, 58:236244, Mar h 1963.[Wit76 H. Witsenhausen. The zero-error side information problem and hromati numbers. IEEE Transa tions on Information Theory, 22(5):592593,September 1976.[WSXK09 Y. Wu, V. Stankovi , Z. Xiong, and S. Y. Kung. On pra ti al designfor joint distributed sour e and network oding. IEEE Transa tions onInformation Theory, 55(4):1709 1720, April 2009.[WTV06 A. B. Wagner, S. Tavildar, and P. Viswanath. Rate region of the quadrati Gaussian two-en oder sour e- oding problem. In Pro eedings of the IEEEInternational Symposium on Information Theory (ISIT'06), pages 14041408, Seattle, Washington, USA, July 2006.[WTV08 A. B. Wagner, S. Tavildar, and P. Viswanath. Rate region of the quadrati Gaussian two-en oder sour e- oding problem. IEEE Transa tions onInformation Theory, 54(5):19381961, May 2008.[WZ76 A. D. Wyner and J. Ziv. The rate-distortion fun tion for sour e odingwith side information at the de oder. IEEE Transa tions on InformationTheory, IT-22(1):110, January 1976.

REFERENCES 164[XLC04 Z. Xiong, A. D. Liveris, and S. Cheng. Distributed sour e oding for sensornetworks. IEEE Signal Pro essing Magazine, 21(5):8094, September2004.[Yah10 P. Yahampath. Joint sour e-de oding in large s ale sensor networks usingMarkov random eld models. Signal Pro essing, 90:31343146, De ember2010.[YB00 Y.-O. Yan and T. Berger. On instantaneous odes for zero-error odingof two orrelated sour es. In Pro eedings of the IEEE InternationalSymposium on Information Theory (ISIT'00), page 344, Sorrento, Italy,June 2000.[YX08 Y. Yang and Z. Xiong. The supremum sum-rate loss of quadrati Gaussiandire t multiterminal sour e oding. In Pro eedings of the InformationTheory and Appli ations Workshop, pages 449453, San Diego, CA, USA,February 2008.[YY09 R. Yasaratna and P. Yahampath. Design of s alable de oders forsensor networks via bayesian network learning. IEEE Transa tions onCommuni ations, 57(10):28682871, O tober 2009.[ZE03 Q. Zhao and M. Eros. Lossless and near-lossless sour e oding formultiple a ess networks. IEEE Transa tions on Information Theory,49(1):112128, January 2003.[ZSE02 R. Zamir, S. Shamai, and U. Erez. Nested linear/latti e odes forstru tured multiterminal binning. IEEE Transa tions on InformationTheory, 48(6):12501276, June 2002.