universal schemes in information theory introductory lecture … · 2011. 9. 27. · universal...
TRANSCRIPT
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
EE477
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
(cont.)
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
(cont.)
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
• LZ77 (sliding window) • LZ78 (incremental parsing)
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
(cont.)
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
Tuesday, September 27, 11
-
Universal Schemes in Information Theory
Introductory lecture
immediate
September 16, 2011
Abstract
bla
1 Examples
1.1 Lossless compression
Problem: Losslessly compress a string x = (x1, x2, . . .)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. entropy rate
2. finite-state compressibiity
Yeah, but can they be attained without
1. knowledge of P
2. knowledge of x
Ziv-Lempel CompressionAchieves:
• The fundamental limits from the non-universal settings:
– The entropy rate of any stationary source
– The finite-state compressibility of any individual sequence
• Linear complexity
• Cuteness
• State of the art performance on real data
(cont.)
Of wide current use:
Gif, Zip, Gzip, PNG, ...
Tuesday, September 27, 11
-
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
Tuesday, September 27, 11
-
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
(cont.)
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
Tuesday, September 27, 11
-
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
Tuesday, September 27, 11
-
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
(cont.)
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
Tuesday, September 27, 11
http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/
-
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
(cont.)
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
1.2 Prediction
Problem: Predict xi on the basis of (x1, x2, . . . xi�1), for i � 1, so as to minimize the per-symbol prediction loss, asmeasured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” predictor
2. Finite-state predictability
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Feder-Merhav-Gutman PredictorAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” predictor
– Finite-state predictability
• Linear complexity
• Cuteness
• Will beat you in a game of rock-paper-scissors any day, see
http://www.mit.edu/~emin/writings/lz_rps/
Note: universal predictor induced by universal compressor
2
Tuesday, September 27, 11
http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/http://www.mit.edu/~emin/writings/lz_rps/
-
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn), so as to minimize theper-symbol prediction loss, as measured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser
– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn), so as to minimize theper-symbol prediction loss, as measured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser
– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given `Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser
– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
Goal:
Tuesday, September 27, 11
-
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn), so as to minimize theper-symbol prediction loss, as measured by a given loss function `
Questions: what are the fundamental limits assuming:
1. x ⇠ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser
– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
(cont.)1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
Tuesday, September 27, 11
-
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
Tuesday, September 27, 11
-
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
(cont.)
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP, cf.
http://www.hpl.hp.com/research/info_theory/dude/index.htm
and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP, cf.
http://www.hpl.hp.com/research/info_theory/dude/index.htm
and maybe Google...
3
Tuesday, September 27, 11
http://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htm
-
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP and maybe Google...
3
(cont.)
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP, cf.
http://www.hpl.hp.com/research/info_theory/dude/index.htm
and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP, cf.
http://www.hpl.hp.com/research/info_theory/dude/index.htm
and maybe Google...
3
1.3 Discrete Denoising
Problem: Recover (x1, x2, . . . xn) on the basis of its DMC-corrupted version (z1, z2, . . . zn),minimize per-symbol loss, as measured by given ⌅Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” denoiser
2. Sliding-window “denoisability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
DUDE: Discrete Universal DEnoiserAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” denoiser– Sliding-window “denoisability”
• Linear complexity
• Cuteness?
• In use @ HP, cf.
http://www.hpl.hp.com/research/info_theory/dude/index.htm
and maybe Google...
3
Tuesday, September 27, 11
http://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htmhttp://www.hpl.hp.com/research/info_theory/dude/index.htm
-
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality, i.e., the estimate of xi may depend only on(z1, z2, . . . zi)
Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
Tuesday, September 27, 11
-
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality, i.e., the estimate of xi may depend only on(z1, z2, . . . zi)
Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
(cont.)
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
Tuesday, September 27, 11
-
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
Tuesday, September 27, 11
-
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
(cont.)
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter
– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
Tuesday, September 27, 11
-
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x ⇥ P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
(cont.)
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter
– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
1.4 Filtering
Problem: Like the discrete denoising problem, but with causality:the estimate of xi may depend only on (z1, z2, . . . zi)Questions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. Performance of the “Bayes-optimal” filter
2. Finite-state “filterability”
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
LZ + DUDE - based filterAchieves:
• The fundamental limits from the non-universal settings:
– Performance of the “Bayes-optimal” filter
– Finite-state “filterability”
• Linear complexity
• Cuteness
Note: universal filter induced by universal predictor
4
Tuesday, September 27, 11
-
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibiity with distortion
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibiity with distortion
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibiity with distortion
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals, i.e., without
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals?
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
Tuesday, September 27, 11
-
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals?
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
Tuesday, September 27, 11
-
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals?
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
introduce noise such that resulting signal is• corrupted• more compressible
Tuesday, September 27, 11
-
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals?
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
introduce noise such that resulting signal is• corrupted• more compressible
Idea:
Tuesday, September 27, 11
-
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals?
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
1.5 Lossy compression
Problem: Lossily compress a string x = (x1, x2, . . .) under a given distortion criterionQuestions: what are the fundamental limits assuming:
1. x � P is stochastic
2. x is an individual sequence
Answers:
1. rate distortion curve
2. finite-state compressibility with distortion
Yeah, but can they be attained by us mortals?
1. knowledge of P
2. knowledge of x
Lossy compression via MCMCShow movieAchieves:
• The fundamental limits from the non-universal setting: the rate distortion curve for any stationary source
• O(1) complexity per iteration...
• Cuteness?
• “State of the art” performance on discrete (small alphabet) data
References
[1] A. Banerjee, X. Guo, and H. Wang, “On the optimality of conditional expectation as a Bregman predictor,”IEEE Trans. Inf. Theory, vol. IT-51, no. 7, pp. 2664–2669, July 2005.
[2] A. D. Barbour, O. Johnson, I. Kontoyiannis and M. Madiman, “Compound Poisson Approximation via Infor-mation Functionals,” Electronic Journal of Probability, vol. 15, no. 42, pp. 1344-1368, 2010.
[3] P. Billingsley. Convergence of Probability Measures. Second edition. Wiley, New York, 1999.
[4] P. Brémaud, Point Processes and Queues, Martingale Dynamics, Springer-Verlag, New York, 1981.
[5] I. Csiszár and J. Körner, Information Theory: Coding theorems for discrete memoryless systems, AcademicPress, New York, 1981.
[6] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, New York, 1998.
[7] T. E. Duncan, “On the calculation of mutual information,” SIAM J. Appl. Math., vol. 19, pp. 215 – 220, July1970.
[8] P. Dupuis and R. S. Ellis. The large deviation principle for a general class of queueing systems. I. Trans. Amer.Math. Soc. 347 (1995), no. 8, 2689–2751
[9] W. H. Fleming. Logarithmic transformations and stochastic control. Advances in Filtering and Optimal Stochas-tic Control; Lecture Notes in Control and Information Sciences, 1982, Vol. 42, 131–141
[10] R. G. Gallager, “Source coding with side information and universal coding,” M.I.T. LIDS-P-937, 1976 (revised1979).
5
introduce noise such that resulting signal is• corrupted• more compressible
Idea:
Tuesday, September 27, 11
http://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://www.youtube.com/watch?v=v2VSveJ-RDIhttp://ww