Information Theory

Mutual Information

One way to measure statistical independence between two random variables is to look at their mutual information content. This is defined as the Kullback-Leibler distance between the joint probability of these variables and the product of the individual probabilities. The mutual information between two random variables x and y from the sets X and J, is given by:

for the continuous case and:

for the discrete case, where P(·) is the probability density function. Mutual information gives us the amount of information that y contains about x.

From the definition we can derive some of the properties of mutual information:

I (x,y) = I (y,x)
I (x,x) = H (x)
I (x,y) = H (x) - H (x|y) = H (y) - H (y|x) = H (x) + H (y) - H (x,y)

Since mutual information is itself a Kullback-Leibler distance it is always positive and zero only if the two variables it measures are independent.