introduction to information theory...references • eugene chiu, jocelyn lin, brok mcferron,...
TRANSCRIPT
![Page 1: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/1.jpg)
Introduction to Information Theory
Part 4
![Page 2: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/2.jpg)
A General Communication System
CHANNEL
• Information Source• Transmitter• Channel• Receiver• Destination
210/2/2012
![Page 3: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/3.jpg)
Information Channel
InputX Channel Output
Y
10/2/2012 3
![Page 4: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/4.jpg)
Information Channel
InputX Channel Output
YCholesterol Levels Condition
of Arteries
10/2/2012 4
![Page 5: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/5.jpg)
Information Channel
InputX Channel Output
YSymptoms
or Test resultsDiagnosis
10/2/2012 5
![Page 6: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/6.jpg)
Information Channel
InputX Channel Output
YGeological Structure Presence
of oil deposits
10/2/2012 6
![Page 7: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/7.jpg)
Information Channel
InputX Channel Output
YOpinion Poll Next President
10/2/2012 7
![Page 8: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/8.jpg)
Perfect Communication(Discrete Noiseless Channel)
XTransmittedSymbol
YReceivedSymbol
0
1
0
1
810/2/2012
![Page 9: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/9.jpg)
9
NOISE
10/2/2012
![Page 10: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/10.jpg)
Motivating Noise…
XTransmittedSymbol
YReceivedSymbol
0
1
0
1
1010/2/2012
![Page 11: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/11.jpg)
Motivating Noise…
0
1
0
1
f = 0.1, n = ~10,000
1 ‐ f
1 ‐ f
f
11
f
10/2/2012
![Page 12: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/12.jpg)
Motivating Noise…
12
Message: $5213.75Received: $5293.75
1. Detect that an error has occurred.
2. Correct the error.
3. Watch out for the overhead.
10/2/2012
![Page 13: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/13.jpg)
Error Detection by Repetition
In the presence of 20% noise…
Message : $ 5 2 1 3 . 7 5Transmission 1: $ 5 2 9 3 . 7 5Transmission 2: $ 5 2 1 3 . 7 5Transmission 3: $ 5 2 1 3 . 1 1Transmission 4: $ 5 4 4 3 . 7 5Transmission 5: $ 7 2 1 8 . 7 5
There is no way of knowing where the errors are.
10/2/2012 13
![Page 14: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/14.jpg)
Error Detection by RepetitionIn the presence of 20% noise…
Message : $ 5 2 1 3 . 7 5Transmission 1: $ 5 2 9 3 . 7 5Transmission 2: $ 5 2 1 3 . 7 5Transmission 3: $ 5 2 1 3 . 1 1Transmission 4: $ 5 4 4 3 . 7 5Transmission 5: $ 7 2 1 8 . 7 5Most common: $ 5 2 1 3 . 7 5
1. Guesswork is involved.2. There is overhead.
10/2/2012 14
![Page 15: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/15.jpg)
Error Detection by Repetition
In the presence of 50% noise…
Message : $ 5 2 1 3 . 7 5…Repeat 1000 times!
1. Guesswork is involved.But it will almost never be wrong!
2. There is overhead.A LOT of it!
10/2/2012 15
![Page 16: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/16.jpg)
Binary Symmetric Channel (BSC)(Discrete Memoryless Channel)
TransmittedSymbol
ReceivedSymbol
0
1
0
1
16
1
1
10/2/2012
![Page 17: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/17.jpg)
Binary Symmetric Channel (BSC)(Discrete Memoryless Channel)
TransmittedSymbol
ReceivedSymbol
0
1
0
1
17
1
1
Defined by a set of conditional probabilities (aka transitional probabilities)
∈ ∈
The probability of occurring at the output when is the input to the channel.
10/2/2012
![Page 18: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/18.jpg)
A General Discrete Channel
p |
p |p | p |
input symbols output symbols
transition probabilities
10/2/2012 18
![Page 19: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/19.jpg)
Channel With Internal Structure
0
1
0
1
1 0
1
1 0
1
0
1
1 1
1 1
1 1
10/2/2012 19
![Page 20: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/20.jpg)
The Weather Channel
SUNNY
CLOUDY
RAINY
HOT
WARM
COLD
0.5
0.25
0.25
p(HOT|SUNNY)=0.5
p(WARM|SUNNY)=0.5
p(WARM|CLOUDY)=0.5
p(COLD|CLOUDY)=0.5
p(WARM|RAINY)=0.5
p(COLD|RAINY)=0.5
10/2/2012 20
![Page 21: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/21.jpg)
Entropy• , random variables with entropy and
• Conditional Entropy: Average entropy in , given knowledge of .
,∈∈ |
where ,
• Joint Entropy: ,Entropy of the pair ,
10/2/2012 21
![Page 22: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/22.jpg)
The Weather ChannelSUNNY
CLOUDY
RAINY
HOT
WARM
COLD
0.5
0.25
0.25
p(HOT|SUNNY)=0.5
p(WARM|SUNNY)=0.5
p(WARM|CLOUDY)=0.5
p(COLD|CLOUDY)=0.5
p(WARM|RAINY)=0.5
p(COLD|RAINY)=0.5
Q. What is the Entropy, ?
10/2/2012 22
![Page 23: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/23.jpg)
The Weather ChannelSUNNY
CLOUDY
RAINY
HOT
WARM
COLD
0.5
0.25
0.25
p(HOT|SUNNY)=0.5
p(WARM|SUNNY)=0.5
p(WARM|CLOUDY)=0.5
p(COLD|CLOUDY)=0.5
p(WARM|RAINY)=0.5
p(COLD|RAINY)=0.5
Q. What is the Entropy, ?
0.5 log 2 0.25 log 4 0.25 log 40.5 0.5 0.51.5bits
10/2/2012 23
![Page 24: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/24.jpg)
Example
• ∑ ||
• Entropy of a toss of die, is log 6 2.59
• If outcome is HIGH (either 5 or 6):2 1
• If outcome is LOW (either 1, 2, 3, or 4):4 2
• Conditional Entropy:13 2
23 4
53 1.67
10/2/2012 24
![Page 25: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/25.jpg)
Example
• ∑ ||
• Entropy of a toss of die, is log 6 2.59
• If outcome is HIGH (either 5 or 6):2 1
• If outcome is LOW (either 1, 2, 3, or 4):4 2
• Conditional Entropy:13 2
23 4
53 1.67
Entropy Reduction:
Entropy of a variable is,on average, never increasedby knowledge of anothervariable .
10/2/2012 25
![Page 26: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/26.jpg)
The Weather ChannelSUNNY
CLOUDY
RAINY
HOT
WARM
COLD
0.5
0.25
0.25
p(HOT|SUNNY)=0.5
p(WARM|SUNNY)=0.5
p(WARM|CLOUDY)=0.5
p(COLD|CLOUDY)=0.5
p(WARM|RAINY)=0.5
p(COLD|RAINY)=0.5
Q. What is the Entropy, | ?
| log1| log
1|
0.5 log 2 0.5log 2 1
∴ 1
10/2/2012 26
![Page 27: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/27.jpg)
The Weather ChannelSUNNY
CLOUDY
RAINY
HOT
WARM
COLD
0.5
0.25
0.25
p(HOT|SUNNY)=0.5
p(WARM|SUNNY)=0.5
p(WARM|CLOUDY)=0.5
p(COLD|CLOUDY)=0.5
p(WARM|RAINY)=0.5
p(COLD|RAINY)=0.5
Q. What is the Entropy, ?
0.5 ∗ 0.5 0.250.5 ∗ 0.5 0.25 ∗ 0.5 025 ∗ 0.5 0.50.25 ∗ 0.5 0.25 ∗ 0.5 0.25
∴ 1.5
10/2/2012 27
![Page 28: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/28.jpg)
Mutual Information
• The mutual information of a random variable given the random variable is
It is the information about transmitted by .
10/2/2012 28
![Page 29: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/29.jpg)
Mutual Information: Properties
•••• is symmetric in and
• ,
••10/2/2012 29
![Page 30: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/30.jpg)
The Weather ChannelSUNNY
CLOUDY
RAINY
HOT
WARM
COLD
0.5
0.25
0.25
p(HOT|SUNNY)=0.5
p(WARM|SUNNY)=0.5
p(WARM|CLOUDY)=0.5
p(COLD|CLOUDY)=0.5
p(WARM|RAINY)=0.5
p(COLD|RAINY)=0.5
Q. What is the mutualinformation ; ?
; 1.5 1.0 0.5
Also
; ; 0.5
10/2/2012 30
![Page 31: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/31.jpg)
Entropy Concepts
• ; , – | |• ; |• ; |• ; is symmetric in and• ; ∑ ∑ , ,
• ; 0• ;
10/2/2012 31
![Page 32: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/32.jpg)
Channel Capacity
• The capacity of a channel is the maximum possible mutual information that can be achieved between input and output by varying the probabilities of the input symbols.
If X is the input channel and Y is the output, the capacity C is
10/2/2012 32
![Page 33: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/33.jpg)
Channel Capacity
Mutual information about X given Y is the information transmitted by the channel and depends on the probability structure
– Input probabilities– Transition probabilities– Output probabilities
10/2/2012 33
![Page 34: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/34.jpg)
Channel Capacity
Mutual information about X given Y is the information transmitted by the channel and depends on the probability structure
– Input probabilities– Transition probabilities: fixed by properties of channel– Output probabilities: determined by input and transition probabilities
10/2/2012 34
![Page 35: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/35.jpg)
Channel Capacity
Mutual information about X given Y is the information transmitted by the channel and depends on the probability structure
– Input probabilities: can be adjusted by suitable coding– Transition probabilities: fixed by properties of channel– Output probabilities: determined by input and transition probabilities
10/2/2012 35
![Page 36: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/36.jpg)
Channel Capacity
;
Mutual information about X given Y is the information transmitted by the channel and depends on the probability structure
– Input probabilities: can be adjusted by suitable coding– Transition probabilities: fixed by properties of channel– Output probabilities: determined by input and transition
probabilities
That is, input probabilities determine mutual information and can be varied by coding. The maximum mutual information with respect to these input probabilities is the channel capacity.
10/2/2012 36
![Page 37: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/37.jpg)
Shannon’s Second Theorem
• Suppose a discrete channel has capacity C and the source has entropy H
If H < C there is a coding scheme such that the source can be transmitted over the channel with an arbitrarily small frequency of error.
If H > C, it is not possible to achieve arbitrarily small error frequency.
10/2/2012 37
![Page 38: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/38.jpg)
Detailed Communication Model
10/2/2012 38
informationsource
datareduction
sourcecoding encipherment channel
coding modulation
destination datareconstruction
sourcecoding
decipherment channeldecoding
demodulation
distortion(noise)
TRANSMITTERDISCRETECHANNEL
RECEIVER
![Page 39: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/39.jpg)
Error Correcting Codes: Checksum
• ISBN: 0‐691‐12418‐3
• 1*0+2*6+3*9+4*1+5*1+6*2+7*4+8*1+9*8= 168 mod 11 = 3
• This is a staircase checksum
10/2/2012 39
![Page 40: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/40.jpg)
Error Correcting Codes
• Hamming Codes (1950)
• Linear Codes
• Low Density Parity Codes (1960)
• Convolutional Codes
• Turbo Codes (1993)
10/2/2012 40
![Page 41: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/41.jpg)
MTC: Summary
• Information• Entropy• Source Coding Theorem
• Redundancy• Compression• Huffman Encoding• Lempel‐Ziv Coding
10/2/2012 41
• Channel• Conditional Entropy• Joint Entropy• Mutual Information• Channel Capacity• Shannon’s Second Theorem
• Error Correction Codes
![Page 42: Introduction to Information Theory...References • Eugene Chiu, Jocelyn Lin, Brok Mcferron, NoshirwanPetigara, SatwiksaiSeshasai: Mathematical Theory of Claude Shannon: A study of](https://reader033.vdocuments.net/reader033/viewer/2022042312/5edaeab709ac2c67fa68841f/html5/thumbnails/42.jpg)
References• Eugene Chiu, Jocelyn Lin, Brok Mcferron, Noshirwan Petigara, Satwiksai Seshasai: Mathematical Theory
of Claude Shannon: A study of the style and context of his work up to the genesis of information theory. MIT 6.933J / STS.420J The Structure of Engineering Revolutions
• Luciano Floridi, 2010: Information: A Very Short Introduction, Oxford University Press, 2011.• Luciano Floridi, 2011: The Philosophy of Information, Oxford University Press, 2011.• James Gleick, 2011: The Information: A History, A Theory, A Flood, Pantheon Books, 2011.• Zhandong Liu , Santosh S Venkatesh and Carlo C Maley, 2008: Sequence space coverage, entropy of
genomes and the potential to detect non‐human DNA in human samples, BMC Genomics 2008, 9:509• David Luenberger, 2006: Information Science, Princeton University Press, 2006.• David J.C. MacKay, 2003: Information Theory, Inference, and Learning Algorithms, Cambridge University
Press, 2003.• Claude Shannon & Warren Weaver, 1949: The Mathematical Theory of Communication, University of
Illinois Press, 1949.• W. N. Francis and H. Kucera: Brown University Standard Corpus of Present‐Day American English, Brown
University, 1967.• Edward L. Glaeser: A Tale of Many Cities, New York Times, April 10, 2010. Available at:
http://economix.blogs.nytimes.com/2010/04/20/a‐tale‐of‐many‐cities/• Alan Rimm‐Kaufman, The Long Tail of Search. Search Engine Land Website, September 18, 2007.
Available at: http://searchengineland.com/the‐long‐tail‐of‐search‐12198
42