Information Theory

Joint and Conditional Entropy

If we have a set of random variables we can also define their joint entropy and their conditional entropy. For the random variables x and y from the sets X and J respectively, the joint entropy is defined for the continuous case as:

and for the discrete case as:

.
Joint entropy is a measure of overall uncertainty of a set of variables. Taking the entropy of a random vector as H(x) we mean the joint entropy between all of the vector elements.

The conditional entropy of x and y is defined as:

.
for the continuous case and:

.
for the discrete case, and is a measure of uncertainty of y given certainty of x.