I have started Information Theory classes just recently and was wondering what would be a standard book to purchase. I know I can go for basic introductory books but I also like to purchase standard books that I can use throughout my career for future reference purposes. I get some threads in forums about introductory books or basic books on information theory, but I can't find any about standard books on information theory. P.S.:By standard book I mean like Cormen is for Algorithms
This is a list of recommended books, videos and web sites copied from the Further Readings section of my book on information theory (given at the end of this post).
Applebaum D (2008). Probability and Information: An Integrated Approach. A thorough introduction to information theory, which strikes a good balance between intuitive and technical explanations.
Avery J (2012). Information Theory and Evolution. An engaging account of how information theory is relevant to a wide range of natural and man-made systems, including evolution, physics, culture and genetics. Includes interesting background stories on the development of ideas within these different disciplines.
Baeyer HV (2005). Information: The New Language of Science Erudite, wide-ranging, and insightful account of information theory. Contains no equations, which makes it very readable.
Cover T and Thomas J (1991). Elements of Information Theory. Comprehensive, and highly technical, with historical notes and an equation summary at the end of each chapter.
Ghahramani Z (2002). Information Theory. Encyclopedia of Cognitive Science. An excellent, brief overview of information.
Gleick J (2012). The Information. An informal introduction to the history of ideas and people associated with information theory.
Guizzo EM (2003). The Essential Message: Claude Shannon and the Making of Information Theory. Master’s Thesis, Massachusetts Institute of Technology. One of the few accounts of Shannon’s role in the development of information theory. See http://dspace.mit.edu/ bitstream/handle/1721.1/39429/54526133.pdf.
Laughlin, SB (2006). The Hungry Eye: Energy, Information and Retinal Function, Excellent lecture on the energy cost of Shannon information in eyes. See http://www.crsltd.com/guest-talks/ crs-guest-lecturers/simon-laughlin.
MacKay DJC (2003). Information Theory, Inference, and Learning Algorithms. The modern classic on information theory. A very readable text that roams far and wide over many topics. The book’s web site (below) also has a link to an excellent series of video lectures by MacKay. Available free online at http://www.inference.phy. cam.ac.uk/mackay/itila/.
Pierce JR (1980). An Introduction to Information Theory: Symbols, Signals and Noise. Second Edition. Pierce writes with an informal, tutorial style of writing, but does not flinch from presenting the fundamental theorems of information theory. This book provides a good balance between words and equations.
Reza FM (1961). An Introduction to Information Theory. A more comprehensive and mathematically rigorous book than Pierce’s book, it should be read only after first reading Pierce’s more informal text.
Seife C (2007). Decoding the Universe: How the New Science of Information Is Explaining Everything in the Cosmos, From Our Brains to Black Holes. A lucid and engaging account of the relationship between information, thermodynamic entropy and quantum computing. Highly recommended.
Shannon CE and Weaver W (1949). The Mathematical Theory of Communication. University of Illinois Press. A surprisingly accessible book, written in an era when information theory was known only to a privileged few. This book can be downloaded from http://cm. bell-labs.com/cm/ms/what/shannonday/paper.html
For the complete novice, the videos at the online Kahn Academy provide an excellent introduction.
Additionally, the online Scholarpedia web page by Latham and Rudi provides a lucid technical account of mutual information: http://www.scholarpedia.org/article/Mutual_information.
Finally, some historical perspective is provided in a long interview with Shannon conducted in 1982: http://www.ieeeghn.org/wiki/ index.php/Oral-History:Claude_E._Shannon.
Finally, my book is called Information Theory: A Tutorial Introduction (2015). Chapter 1 can be downloaded from here: http://jim-stone.staff.shef.ac.uk/BookInfoTheory/InfoTheoryBookMain.html