Elements of information theory

Başlık:

Yazar:

Cover, T. M., 1938-2012.

ISBN:

9780471241959

Ek Yazar:

Cover, T. M., 1938-2012.

Edition:

2nd ed.

Yayım Bilgisi:

Hoboken, N.J. : Wiley-Interscience, 2006.

Fiziksel Tanım:

xxiii, 748 pages : illustrations ; 25 cm

Contents:

Entropy, relative entropy, and mutual information -- Asymptotic equipartition property -- Entropy rates of a stochastic process -- Data compression -- Gambling and data compression -- Channel capacity -- Differential entropy -- Gaussian channel -- Rate distortion theory -- Information theory and statistics -- Maximum entropy -- Universal source coding -- Kolmogorov complexity -- Network information theory -- Information theory and portfolio theory -- Inequalities in information theory.

Konu Terimleri:

Information theory.

Bilgi teorisi

Added Author:

Thomas, Joy A.

Mevcut:*

Library	Materyal Türü	Barkod	Yer Numarası	Durum
Searching... Pamukkale Merkez Kütüphanesi	Kitap	0097804	Q360 C68 2006	Searching... Unknown

Bound With These Titles

On Order

Özet

The latest edition of this classic is updated with new problem sets and material

The Second Edition of this fundamental textbook maintains the book's tradition of clear, thought-provoking instruction. Readers are provided once again with an instructive mix of mathematics, physics, statistics, and information theory.

All the essential topics in information theory are covered in detail, including entropy, data compression, channel capacity, rate distortion, network information theory, and hypothesis testing. The authors provide readers with a solid understanding of the underlying theory and applications. Problem sets and a telegraphic summary at the end of each chapter further assist readers. The historical notes that follow each chapter recap the main points.

The Second Edition features:

Chapters reorganized to improve teaching 200 new problems New material on source coding, portfolio theory, and feedback capacity Updated references

Now current and enhanced, the Second Edition of Elements of Information Theory remains the ideal textbook for upper-level undergraduate and graduate courses in electrical engineering, statistics, and telecommunications.

Author Notes

Thomas M. Cover, PhD, is Professor in the departments of electrical engineering and statistics, Stanford University
Joy A. Thomas, PhD, is the Chief Scientist at Stratify, Inc., a Silicon Valley start-up specializing in organizing unstructured information

Preface to the Second Edition	p. xv
Preface to the First Edition	p. xvii
Acknowledgments for the Second Edition	p. xxi
Acknowledgments for the First Edition	p. xxiii
1 Introduction and Preview	p. 1
1.1 Preview of the Book	p. 5
2 Entropy, Relative Entropy, and Mutual Information	p. 13
2.1 Entropy	p. 13
2.2 Joint Entropy and Conditional Entropy	p. 16
2.3 Relative Entropy and Mutual Information	p. 19
2.4 Relationship Between Entropy and Mutual Information	p. 20
2.5 Chain Rules for Entropy, Relative Entropy, and Mutual Information	p. 22
2.6 Jensen's Inequality and Its Consequences	p. 25
2.7 Log Sum Inequality and Its Applications	p. 30
2.8 Data-Processing Inequality	p. 34
2.9 Sufficient Statistics	p. 35
2.10 Fano's Inequality	p. 37
Summary	p. 41
Problems	p. 43
Historical Notes	p. 54
3 Asymptotic Equipartition Property	p. 57
3.1 Asymptotic Equipartition Property Theorem	p. 58
3.2 Consequences of the AEP: Data Compression	p. 60
3.3 High-Probability Sets and the Typical Set	p. 62
Summary	p. 64
Problems	p. 64
Historical Notes	p. 69
4 Entropy Rates of a Stochastic Process	p. 71
4.1 Markov Chains	p. 71
4.2 Entropy Rate	p. 74
4.3 Example: Entropy Rate of a Random Walk on a Weighted Graph	p. 78
4.4 Second Law of Thermodynamics	p. 81
4.5 Functions of Markov Chains	p. 84
Summary	p. 87
Problems	p. 88
Historical Notes	p. 100
5 Data Compression	p. 103
5.1 Examples of Codes	p. 103
5.2 Kraft Inequality	p. 107
5.3 Optimal Codes	p. 110
5.4 Bounds on the Optimal Code Length	p. 112
5.5 Kraft Inequality for Uniquely Decodable Codes	p. 115
5.6 Huffman Codes	p. 118
5.7 Some Comments on Huffman Codes	p. 120
5.8 Optimality of Huffman Codes	p. 123
5.9 Shannon-Fano-Elias Coding	p. 127
5.10 Competitive Optimality of the Shannon Code	p. 130
5.11 Generation of Discrete Distributions from Fair Coins	p. 134
Summary	p. 141
Problems	p. 142
Historical Notes	p. 157
6 Gambling and Data Compression	p. 159
6.1 The Horse Race	p. 159
6.2 Gambling and Side Information	p. 164
6.3 Dependent Horse Races and Entropy Rate	p. 166
6.4 The Entropy of English	p. 168
6.5 Data Compression and Gambling	p. 171
6.6 Gambling Estimate of the Entropy of English	p. 173
Summary	p. 175
Problems	p. 176
Historical Notes	p. 182
7 Channel Capacity	p. 183
7.1 Examples of Channel Capacity	p. 184
7.1.1 Noiseless Binary Channel	p. 184
7.1.2 Noisy Channel with Nonoverlapping Outputs	p. 185
7.1.3 Noisy Typewriter	p. 186
7.1.4 Binary Symmetric Channel	p. 187
7.1.5 Binary Erasure Channel	p. 188
7.2 Symmetric Channels	p. 189
7.3 Properties of Channel Capacity	p. 191
7.4 Preview of the Channel Coding Theorem	p. 191
7.5 Definitions	p. 192
7.6 Jointly Typical Sequences	p. 195
7.7 Channel Coding Theorem	p. 199
7.8 Zero-Error Codes	p. 205
7.9 Fano's Inequality and the Converse to the Coding Theorem	p. 206
7.10 Equality in the Converse to the Channel Coding Theorem	p. 208
7.11 Hamming Codes	p. 210
7.12 Feedback Capacity	p. 216
7.13 Source-Channel Separation Theorem	p. 218
Summary	p. 222
Problems	p. 223
Historical Notes	p. 240
8 Differential Entropy	p. 243
8.1 Definitions	p. 243
8.2 AEP for Continuous Random Variables	p. 245
8.3 Relation of Differential Entropy to Discrete Entropy	p. 247
8.4 Joint and Conditional Differential Entropy	p. 249
8.5 Relative Entropy and Mutual Information	p. 250
8.6 Properties of Differential Entropy, Relative Entropy, and Mutual Information	p. 252
Summary	p. 256
Problems	p. 256
Historical Notes	p. 259
9 Gaussian Channel	p. 261
9.1 Gaussian Channel: Definitions	p. 263
9.2 Converse to the Coding Theorem for Gaussian Channels	p. 268
9.3 Bandlimited Channels	p. 270
9.4 Parallel Gaussian Channels	p. 274
9.5 Channels with Colored Gaussian Noise	p. 277
9.6 Gaussian Channels with Feedback	p. 280
Summary	p. 289
Problems	p. 290
Historical Notes	p. 299
10 Rate Distortion Theory	p. 301
10.1 Quantization	p. 301
10.2 Definitions	p. 303
10.3 Calculation of the Rate Distortion Function	p. 307
10.3.1 Binary Source	p. 307
10.3.2 Gaussian Source	p. 310
10.3.3 Simultaneous Description of Independent Gaussian Random Variables	p. 312
10.4 Converse to the Rate Distortion Theorem	p. 315
10.5 Achievability of the Rate Distortion Function	p. 318
10.6 Strongly Typical Sequences and Rate Distortion	p. 325
10.7 Characterization of the Rate Distortion Function	p. 329
10.8 Computation of Channel Capacity and the Rate Distortion Function	p. 332
Summary	p. 335
Problems	p. 336
Historical Notes	p. 345
11 Information Theory and Statistics	p. 347
11.1 Method of Types	p. 347
11.2 Law of Large Numbers	p. 355
11.3 Universal Source Coding	p. 357
11.4 Large Deviation Theory	p. 360
11.5 Examples of Sanov's Theorem	p. 364
11.6 Conditional Limit Theorem	p. 366
11.7 Hypothesis Testing	p. 375
11.8 Chernoff-Stein Lemma	p. 380
11.9 Chernoff Information	p. 384
11.10 Fisher Information and the Cramer-Rao Inequality	p. 392
Summary	p. 397
Problems	p. 399
Historical Notes	p. 408
12 Maximum Entropy	p. 409
12.1 Maximum Entropy Distributions	p. 409
12.2 Examples	p. 411
12.3 Anomalous Maximum Entropy Problem	p. 413
12.4 Spectrum Estimation	p. 415
12.5 Entropy Rates of a Gaussian Process	p. 416
12.6 Burg's Maximum Entropy Theorem	p. 417
Summary	p. 420
Problems	p. 421
Historical Notes	p. 425
13 Universal Source Coding	p. 427
13.1 Universal Codes and Channel Capacity	p. 428
13.2 Universal Coding for Binary Sequences	p. 433
13.3 Arithmetic Coding	p. 436
13.4 Lempel-Ziv Coding	p. 440
13.4.1 Sliding Window Lempel-Ziv Algorithm	p. 441
13.4.2 Tree-Structured Lempel-Ziv Algorithms	p. 442
13.5 Optimality of Lempel-Ziv Algorithms	p. 443
13.5.1 Sliding Window Lempel-Ziv Algorithms	p. 443
13.5.2 Optimality of Tree-Structured Lempel-Ziv Compression	p. 448
Summary	p. 456
Problems	p. 457
Historical Notes	p. 461
14 Kolmogorov Complexity	p. 463
14.1 Models of Computation	p. 464
14.2 Kolmogorov Complexity: Definitions and Examples	p. 466
14.3 Kolmogorov Complexity and Entropy	p. 473
14.4 Kolmogorov Complexity of Integers	p. 475
14.5 Algorithmically Random and Incompressible Sequences	p. 476
14.6 Universal Probability	p. 480
14.7 Kolmogorov complexity	p. 482
14.8 [Omega]	p. 484
14.9 Universal Gambling	p. 487
14.10 Occam's Razor	p. 488
14.11 Kolmogorov Complexity and Universal Probability	p. 490
14.12 Kolmogorov Sufficient Statistic	p. 496
14.13 Minimum Description Length Principle	p. 500
Summary	p. 501
Problems	p. 503
Historical Notes	p. 507
15 Network Information Theory	p. 509
15.1 Gaussian Multiple-User Channels	p. 513
15.1.1 Single-User Gaussian Channel	p. 513
15.1.2 Gaussian Multiple-Access Channel with m Users	p. 514
15.1.3 Gaussian Broadcast Channel	p. 515
15.1.4 Gaussian Relay Channel	p. 516
15.1.5 Gaussian Interference Channel	p. 518
15.1.6 Gaussian Two-Way Channel	p. 519
15.2 Jointly Typical Sequences	p. 520
15.3 Multiple-Access Channel	p. 524
15.3.1 Achievability of the Capacity Region for the Multiple-Access Channel	p. 530
15.3.2 Comments on the Capacity Region for the Multiple-Access Channel	p. 532
15.3.3 Convexity of the Capacity Region of the Multiple-Access Channel	p. 534
15.3.4 Converse for the Multiple-Access Channel	p. 538
15.3.5 m-User Multiple-Access Channels	p. 543
15.3.6 Gaussian Multiple-Access Channels	p. 544
15.4 Encoding of Correlated Sources	p. 549
15.4.1 Achievability of the Slepian-Wolf Theorem	p. 551
15.4.2 Converse for the Slepian-Wolf Theorem	p. 555
15.4.3 Slepian-Wolf Theorem for Many Sources	p. 556
15.4.4 Interpretation of Slepian-Wolf Coding	p. 557
15.5 Duality Between Slepian-Wolf Encoding and Multiple-Access Channels	p. 558
15.6 Broadcast Channel	p. 560
15.6.1 Definitions for a Broadcast Channel	p. 563
15.6.2 Degraded Broadcast Channels	p. 564
15.6.3 Capacity Region for the Degraded Broadcast Channel	p. 565
15.7 Relay Channel	p. 571
15.8 Source Coding with Side Information	p. 575
15.9 Rate Distortion with Side Information	p. 580
15.10 General Multiterminal Networks	p. 587
Summary	p. 594
Problems	p. 596
Historical Notes	p. 609
16 Information Theory and Portfolio Theory	p. 613
16.1 The Stock Market: Some Definitions	p. 613
16.2 Kuhn-Tucker Characterization of the Log-Optimal Portfolio	p. 617
16.3 Asymptotic Optimality of the Log-Optimal Portfolio	p. 619
16.4 Side Information and the Growth Rate	p. 621
16.5 Investment in Stationary Markets	p. 623
16.6 Competitive Optimality of the Log-Optimal Portfolio	p. 627
16.7 Universal Portfolios	p. 629
16.7.1 Finite-Horizon Universal Portfolios	p. 631
16.7.2 Horizon-Free Universal Portfolios	p. 638
16.8 Shannon-McMillan-Breiman Theorem (General AEP)	p. 644
Summary	p. 650
Problems	p. 652
Historical Notes	p. 655
17 Inequalities in Information Theory	p. 657
17.1 Basic Inequalities of Information Theory	p. 657
17.2 Differential Entropy	p. 660
17.3 Bounds on Entropy and Relative Entropy	p. 663
17.4 Inequalities for Types	p. 665
17.5 Combinatorial Bounds on Entropy	p. 666
17.6 Entropy Rates of Subsets	p. 667
17.7 Entropy and Fisher Information	p. 671
17.8 Entropy Power Inequality and Brunn-Minkowski Inequality	p. 674
17.9 Inequalities for Determinants	p. 679
17.10 Inequalities for Ratios of Determinants	p. 683
Summary	p. 686
Problems	p. 686
Historical Notes	p. 687
Bibliography	p. 689
List of Symbols	p. 723
Index	p. 727

Mevcut:*

Bound With These Titles

On Order

Özet

Özet

Author Notes

Table of Contents