Principle of maximum entropy

The principle of maximum entropy is a postulate about a universal feature of any probability assignment on a given set of propositions (events, hypotheses, indices, etc.). Let some testable information about a probability distribution function be given. Consider the set of all trial probability distributions which encode this information. Then the probability distribution which maximizes the information entropy is the true probability distribution, with respect to the testable information prescribed.

In most practical cases, the testable information is given by a set of conserved quantities (average values of some moment functions), associated with the probability distribution in question. In this way the maximum entropy principle is most often used in statistical thermodynamics. Another possibility is to prescribe some symmetries of the probability distribution. An equivalence between the conserved quantities and corresponding symmetry groups implies the same level of equivalence for both these two ways of specifying the testable information in the maximum entropy method.

The maximum entropy principle is also needed to guarantee the uniqueness and consistency of probability assignments obtained by different methods, statistical mechanics and logical inference in particular. Strictly speaking, the trial distributions, which do not maximize the entropy, are actually not probability distributions.

The principle was first expounded by E.T. Jaynes in his seminal papers of 1957 where he emphasized a natural correspondence between statistical mechanics and information theory. In particular, Jaynes offered a new and very general rationale why the Gibbsian method of statistical mechanics works. He suggested that the entropy in statistical mechanics, and the information entropy in information theory, are principally the same thing. Consequently, statistical mechanics should be seen just as a particular application of a general tool of logical inference and information theory.

The maximum entropy principle makes explicit our freedom in using different forms of prior information. As a special case, a uniform prior probability density (Laplace's principle of indifference) may be adopted. Thus, the maximum entropy principle is not just an alternative to the methods of inference of classical statistics, but its important conceptual generalization and correction.

Testable information

The principle of maximum entropy is useful explicitly only when applied to testable information. A piece of information is testable if it can be determined whether a given distribution is consistent with it. For example, the statements

The expectation of the variable x is 2.87

and

p₂ + p₃ > 0.6

are statements of testable information.

Given testable information, the maximum entropy procedure consists of seeking the probability distribution which maximizes information entropy, subject to the constraints of the information. This constrained optimization problem is typically solved using the method of Lagrange multipliers.

Entropy maximization with no testable information takes place under a single constraint: the sum of the probabilities must be one. Under this constraint, the maximum entropy probability distribution is the uniform distribution,

$p\_i=frac\{1\}\{n\}\; \{rm\; for\; all\}\; iin\{,1,dots,n,\}.$

The principle of maximum entropy can thus be seen as a generalization of the classical principle of indifference, also known as the principle of insufficient reason

General solution for the maximum entropy distribution with linear constraints

Discrete case

We have some testable information I about a quantity x ∈ {x₁, x₂,..., x_n}. We express this information as m constraints on the expectations of the functions f_k; that is, we require our epistemic probability distribution to satisfy

$sum\_\{i=1\}^n\; Pr(x\_i|I)f\_k(x\_i)\; =\; F\_k\; qquad\; k\; =\; 1,\; cdots,m.$

Furthermore, the probabilities must sum to one, giving the constraint

$sum\_\{i=1\}^n\; Pr(x\_i|I)\; =\; 1.$

The probability distribution with maximum information entropy subject to these constraints is

$Pr(x\_i|I)\; =\; frac\{1\}\{Z(lambda\_1,cdots,\; lambda\_m)\}\; expleft[lambda\_1\; f\_1(x\_i)\; +\; cdots\; +\; lambda\_m\; f\_m(x\_i)right]$

It is sometimes called the Gibbs distribution. The normalization constant is determined by

$Z(lambda\_1,cdots,\; lambda\_m)\; =\; sum\_\{i=1\}^n\; expleft[lambda\_1\; f\_1(x\_i)\; +\; cdots\; +\; lambda\_m\; f\_m(x\_i)right].$

and is conventionally called the partition function. (Interestingly, the Pitman-Koopman theorem states that the necessary and sufficient condition for a sampling distribution to admit sufficient statistics of bounded dimension is that it have the general form of a maximum entropy distribution.)

The λ_k parameters are Lagrange multipliers whose particular values are determined by the constraints according to

$F\_k\; =\; frac\{partial\}\{partial\; lambda\_k\}\; log\; Z(lambda\_1,cdots,\; lambda\_m).$

These m simultaneous equations do not generally possess a closed form solution, and are usually solved by numerical methods.

Continuous case

For continuous distributions, the simple definition of Shannon entropy ceases to be so useful (see differential entropy). Instead what is more useful is the relative entropy of the distribution, (Jaynes, 1963, 1968, 2003),

$H\_c=-int\; p(x)logfrac\{p(x)\}\{m(x)\},dx$

where m(x), which Jaynes called the "invariant measure", is proportional to the limiting density of discrete points. For now, we shall assume that it is known; we will discuss it further after the solution equations are given.

Relative entropy is usually defined as the Kullback-Leibler divergence of m from p (although it is sometimes, confusingly, defined as the negative of this). The inference principle of minimizing this, due to Kullback, is known as the Principle of Minimum Discrimination Information.

We have some testable information I about a quantity x which takes values in some interval of the real numbers (all integrals below are over this interval). We express this information as m constraints on the expectations of the functions f_k, i.e. we require our epistemic probability density function to satisfy

$int\; p(x|I)f\_k(x)dx\; =\; F\_k\; qquad\; k\; =\; 1,\; cdots,m$

And of course, the probability density must integrate to one, giving the constraint

$int\; p(x|I)dx\; =\; 1$

The probability density function with maximum H_c subject to these constraints is

$p(x|I)\; =\; frac\{1\}\{Z(lambda\_1,cdots,\; lambda\_m)\}\; m(x)expleft[lambda\_1\; f\_1(x)\; +\; cdots\; +\; lambda\_m\; f\_m(x)right]$

with the partition function determined by

$Z(lambda\_1,cdots,\; lambda\_m)\; =\; int\; m(x)expleft[lambda\_1\; f\_1(x)\; +\; cdots\; +\; lambda\_m\; f\_m(x)right]dx$

As in the discrete case, the values of the λ_k parameters are determined by the constraints according to

$F\_k\; =\; frac\{partial\}\{partial\; lambda\_k\}\; log\; Z(lambda\_1,cdots,\; lambda\_m)$

The invariant measure function m(x) can be best understood by supposing that x is known to take values only in the bounded interval (a, b), and that no other information is given. Then the maximum entropy probability density function is

where A is a normalization constant. The invariant measure function is actually the prior density function encoding 'lack of relevant information'. It cannot be determined by the principle of maximum entropy, and must be determined by some other logical method, such as the principle of transformation groups or marginalization theory.

Refer to Cover and Thomas for excellent explanation of the ideas .

Examples

For several examples of maximum entropy distributions, see the article on maximum entropy probability distributions.

Justifications for the principle of maximum entropy

Proponents of the principle of maximum entropy justify its use in assigning epistemic probabilities in several ways, including the following two arguments. These arguments take the use of epistemic probability as given, and thus have no force if the concept is itself under question.

Information entropy as a measure of 'uninformativeness'

Consider a discrete epistemic probability distribution among m mutually exclusive propositions. The most informative distribution would occur when one of the propositions was known to be true. In that case, the information entropy would be equal to zero. The least informative distribution would occur when there is no reason to favor any one of the propositions over the others. In that case, the only reasonable probability distribution would be uniform, and then the information entropy would be equal to its maximum possible value, log m. The information entropy can therefore be seen as a numerical measure which describes how uninformative a particular probability distribution is from zero (completely informative) to log m (completely uninformative).

By choosing to use the distribution with the maximum entropy allowed by our information, the argument goes, we are choosing the most uninformative distribution possible. To choose a distribution with lower entropy would be to assume information we do not possess; to choose one with a higher entropy would violate the constraints of the information we do possess. Thus the maximum entropy distribution is the only reasonable distribution.

The Wallis derivation

The following argument is the result of a suggestion made by Graham Wallis to E. T. Jaynes in 1962 (Jaynes, 2003). It is essentially the same mathematical argument used for the Maxwell-Boltzmann statistics in statistical mechanics, although the conceptual emphasis is quite different. It has the advantage of being strictly combinatorial in nature, making no reference to information entropy as a measure of 'uncertainty', 'uninformativeness', or any other imprecisely defined concept. The information entropy function is not assumed a priori, but rather is found in the course of the argument; and the argument leads naturally to the procedure of maximizing the information entropy, rather than treating it in some other way.

Suppose an individual wishes to make an epistemic probability assignment among m mutually exclusive propositions. She has some testable information, but is not sure how to go about including this information in her probability assessment. She therefore conceives of the following random experiment. She will distribute N quanta of epistemic probability (each worth 1/N) at random among the m possibilities. (One might imagine that she will throw N balls into m buckets while blindfolded. In order to be as fair as possible, each throw is to be independent of any other, and every bucket is to be the same size.) Once the experiment is done, she will check if the probability assignment thus obtained is consistent with her information. If not, she will reject it and try again. Otherwise, her assessment will be

$p\_i\; =\; frac\{n\_i\}\{N\}$

where n_i is the number of quanta that were assigned to the i^th proposition.

Now, in order to reduce the 'graininess' of the epistemic probability assignment, it will be necessary to use quite a large number of quanta of epistemic probability. Rather than actually carry out, and possibly have to repeat, the rather long random experiment, our protagonist decides to simply calculate and use the most probable result. The probability of any particular result is the multinomial distribution,

$Pr(mathbf\{p\})\; =\; W\; cdot\; m^\{-N\}$

where

$W\; =\; frac\{N!\}\{n\_1\; !n\_2\; !...n\_m!\}$

is sometimes known as the multiplicity of the outcome.

The most probable result is the one which maximizes the multiplicity W. Rather than maximizing W directly, our protagonist could equivalently maximize any monotonic increasing function of W. She decides to maximize

$begin\{matrix\}frac\{1\}\{N\}log\; W$

&=& frac{1}{N}log frac{N!}{n_1 !n_2 !...n_m!}qquadqquadqquadqquadqquad &=& frac{1}{N}log frac{N!}{Np_1 !Np_2 !...Np_m!} qquadqquadqquadqquad &=& frac{1}{N}left(log N! - sum_{i=1}^m log Np_i! right) qquadqquadend{matrix}

At this point, in order simplify the expression, our protagonist takes the limit as N → ∞, i.e. as the epistemic probability levels go from grainy discrete values to smooth continuous values. Using Stirling's approximation, she finds

$begin\{matrix\}lim\_\{N\; to\; infty\}left(frac\{1\}\{N\}log\; Wright)$

&=& frac{1}{N}left(Nlog N - sum_{i=1}^m Np_ilog Np_i right)qquadqquadqquadqquad &=& log N - sum_{i=1}^m p_ilog Np_i qquadqquadqquadqquadqquadqquad &=& log N - log N sum_{i=1}^m p_i - sum_{i=1}^m p_ilog p_i qquadqquadqquad &=& left(1 - sum_{i=1}^m p_i right)log N - sum_{i=1}^m p_ilog p_i qquadqquadqquad &=& - sum_{i=1}^m p_ilog p_i qquadqquadqquadqquadqquadqquadqquadqquad &=& H(mathbf{p}) qquadqquadqquadqquadqquadqquadqquadqquadqquadqquad end{matrix}

All that remains for our protagonist to do is to maximize entropy under the constraints of her testable information. She has found that the maximum entropy distribution is the most probable of all "fair" random epistemic distributions, in the limit as the probability levels go from discrete to continuous.

Compatibility with Bayes Rule

Recently, it has been shown that Bayes' Rule and the Principle of Maximum Entropy (MaxEnt) are completely compatible and can be seen as special cases of the Method of Maximum (relative) Entropy (Giffin 2007). This method reproduces every aspect of orthodox Bayesian inference methods. In addition this new method opens the door to tackling problems that could not be addressed by either the MaxEnt or orthodox Bayesian methods individually. Moreover, recent contributions (Lazar 2003, and Schennach 2005) show that frequentist relative-entropy-based inference approaches (such as Empirical Likelihood and Exponentially Tilted Empirical Likelihod - see e.g. Owen 2001 and Kitamura 2006) can be combined with prior information to perform Bayesian posterior analysis.

See also

• Entropy maximization
• Maximum entropy classifier
• Maximum entropy spectral estimation

References

• Jaynes, E.T., 1957, 'Information Theory and Statistical Mechanics', in Physical Review May 1957 Volume 106, #4 (pages 620-630).
• Jaynes, E. T., 1963, 'Information Theory and Statistical Mechanics', in Statistical Physics, K. Ford (ed.), Benjamin, New York, p. 181.
• Jaynes, E. T., 1968, 'Prior Probabilities', IEEE Trans. on Systems Science and Cybernetics, SSC-4, 227.
• Jaynes, E. T., 2003, Probability Theory: The Logic of Science, Cambridge University Press.
• Giffin, A. and Caticha, A., 2007, Updating Probabilities with Data and Moments
• Guiasu, S. and Shenitzer, A., 1985, 'The principle of maximum entropy', The Mathematical Intelligencer, 7(1).
• Kapur, J. N.; and Kesevan, H. K., 1992, Entropy optimization principles with applications, Boston: Academic Press. ISBN 0-12-397670-7
• Kitamura, Y., 2006, “ Empirical Likelihood Methods in Econometrics: Theory and Practice,”Cowles Foundation Discussion Papers 1569, Cowles Foundation, Yale University.
• Lazar, N., 2003, “"Bayesian Empirical Likelihood", ”Biometrika, 90, 319–-326.
• Owen, A. B., Empirical Likelihood, Chapman and Hall.
• Schennach, S. M., 2005, "“Bayesian Exponentially Tilted Empirical Likelihood", Biometrika, 92(1), 31-–46.
• Uffink, Jos, 1995, 'Can the Maximum Entropy Principle be explained as a consistency requirement?', Studies in History and Philosophy of Modern Physics 26B, 223-261.

External links

• An easy-to-read introduction to maximum entropy methods in the context of natural language processing is:

Ratnaparkhi A. "A simple introduction to maximum entropy models for natural language processing" Technical Report 97-08, Institute for Research in Cognitive Science, University of Pennsylvania, 1997. Available here

• Maximum Entropy Modeling

This page contains pointers to various papers and software implementations of Maximum Entropy Model on the net.