Introduction to information theory

Information theory is a branch of the mathematical theory of probability and mathematical statistics, that quantifies the concept of information. It is concerned with information entropy, communication systems, data transmission and rate distortion theory, cryptography, data compression, error correction, and related topics.

Information theory was formulated by Shannon while at Bell Labs, who was working on the problem of efficiently transmitting information over a noisy communication channel. His approach employed probability and ergodic theory to study the statistical characteristics of communication systems. Since then information theory has been used in a variety of disciples, well beyond telecommunications, ranging from physics to medicine to computer science.