MATLAB Function Reference

information

Estimate the mutual information of two stationary signal with independent pairs of samples

Syntax

Description

Mutual information estimation is like entropy estimation a two stage process; first a two demensional histogram2 is estimated and thereafter the mutual information is calculated. For the explanation of the usage of the descriptor of the histogram see histogram2 .

In case of a disrete stochastic variable i and j in the integer subranges lowerx <= i < upperx and lowery <= j < uppery the descriptor should be selected as [lowerx,upperx,upperx-lowerx;lowery,uppery,uppery-lowery]. The R(epresentation)-unbiased entropy will be estimated.

In case of a continuous stochastic variable the descriptor can be left unspecified. In this case the default descriptor of histogram2 will be used.

The estimate depends on the value of approach

The base of the logarithm determines the unit of measurement. Default base e (nats) is used, alternative choises are 2 (bit) and 10 (Hartley).

As a result the function returns the estimate, the N-bias (Nbias) of the estimate, the estimated standard error sigma and the used descriptor.

Note: due to samples outside the histogram, which are excuded from the estimate, the relation entropy(x)+entropy(y)-estropy2(x,y) == information(x,y) is only approximately valid.

Example

The mutual information of binormal distributed pairs of samples with correlation coefficient rho is 0.5 equals 0.1438 nat. Estimate this mutual information:

>> rho=0.5;
>> x=normrnd(0,1,1,1000);
>> n=normrnd(0,1,1,1000);
>> y=rho*x+sqrt(1-rho^2)*n;
>> [estimate,nbias,sigma,descriptor] =information(x,y,[-3,3,12;-3,3,12])
estimate = 0.1209
nbias = 0
sigma = 0.0173
descriptor = -3 3 12
-3 3 12

See Also

information
histogram

See Also

entropy
histogram2

Literature

Moddemeijer, R. On Estimation of Entropy and Mutual Information of Continuous Distributions, Signal Processing, 1989, vol. 16, nr. 3, pp. 233-246, abstract , BibTeX ,

For the principle of Minimum Mean Square Error estimation see:

Moddemeijer, R. An efficient algorithm for selecting optimal configurations of AR-coefficients, Twentieth Symp. on Information Theory in the Benelux, May 27-28, 1999, Haasrode (B), pp 189-196, eds. A. Barbé et. al., Werkgemeenschap Informatie- en Communicatietheorie, Enschede (NL), and IEEE Benelux Chapter on Information Theory, ISBN: 90-71048-14-4, abstract , BibTeX ,

Source code

information.m

MATLAB Function Reference


Copyright R. Moddemeijer