Entropy of a probability distribution

The entropy of a probability distribution is a number that describes the degree of uncertainty or disorder the distribution represents.

Examples

Assume we have a set of two mutually exclusive propositions (or equivalently, a random experiment with two possible outcomes). Assume all two possiblities are equally likely.

Then our advance uncertainty about the eventual outcome is rather small - we know in advance it will be one of exactly two known alternatives.

Assume now we have a set of a million alternatives - all of them equally likely - rather than two.

It seems clear that our uncertainty now about the eventual outcome will be much bigger.

Formal definitions

1. Given a discrete probability distribution function f, the entropy H of the distribution is given by $H=-\sum _{i=-Inf}^{i=Inf}f(x_{i})log_{2}f(x_{i})$
2. Given a continuous probability distribution function f, the entropy H of the distribution is given by $H=-\int _{i=-Inf}^{i=Inf}f(x_{i})log_{2}f(x_{i})dx$

Note that some authors prefer to use the natural logarithm rather than base two.