MATLAB Function Reference

information

Estimate the mutual information of two stationary signal with independent pairs of samples

Syntax

```[estimate,Nbias,sigma,descriptor] = information(x,y)
[estimate,Nbias,sigma,descriptor] = information(x,y,descriptor)
[estimate,Nbias,sigma,descriptor] = information(x,y,descriptor,base)
[estimate,Nbias,sigma,descriptor] = information(x,y,descriptor,base,approach)
```

Description

Mutual information estimation is like entropy estimation a two stage process; first a two demensional histogram2 is estimated and thereafter the mutual information is calculated. For the explanation of the usage of the `descriptor` of the histogram see histogram2 .

In case of a disrete stochastic variable `i` and `j` in the integer subranges `lowerx <= i < upperx` and `lowery <= j < uppery` the descriptor should be selected as `[lowerx,upperx,upperx-lowerx;lowery,uppery,uppery-lowery]`. The R(epresentation)-unbiased entropy will be estimated.

In case of a continuous stochastic variable the descriptor can be left unspecified. In this case the default descriptor of histogram2 will be used.

The estimate depends on the value of `approach`

• `'unbiased'`: a N(umber)-unbiased estimate (default),
• `'biased'`: a N(umber)-biased estimate and
• `'mmse'`: a minimum Mean Square Error estimate, obtained by balancing bias and variance after N-bias correction.
The `base` of the logarithm determines the unit of measurement. Default base e (nats) is used, alternative choises are 2 (bit) and 10 (Hartley).

As a result the function returns the `estimate`, the N-bias (`Nbias`) of the estimate, the estimated standard error `sigma` and the used `descriptor`.

Note: due to samples outside the histogram, which are excuded from the estimate, the relation ```entropy(x)+entropy(y)-estropy2(x,y) == information(x,y)``` is only approximately valid.

Example

The mutual information of binormal distributed pairs of samples with correlation coefficient `rho` is 0.5 equals 0.1438 nat. Estimate this mutual information:

``` >> rho=0.5; >> x=normrnd(0,1,1,1000); >> n=normrnd(0,1,1,1000); >> y=rho*x+sqrt(1-rho^2)*n; >> [estimate,nbias,sigma,descriptor] =information(x,y,[-3,3,12;-3,3,12]) estimate = 0.1209 nbias = 0 sigma = 0.0173 descriptor = -3 3 12 -3 3 12 ```

Literature

Moddemeijer, R. On Estimation of Entropy and Mutual Information of Continuous Distributions, Signal Processing, 1989, vol. 16, nr. 3, pp. 233-246, abstract , BibTeX ,

For the principle of Minimum Mean Square Error estimation see:

Moddemeijer, R. An efficient algorithm for selecting optimal configurations of AR-coefficients, Twentieth Symp. on Information Theory in the Benelux, May 27-28, 1999, Haasrode (B), pp 189-196, eds. A. Barbé et. al., Werkgemeenschap Informatie- en Communicatietheorie, Enschede (NL), and IEEE Benelux Chapter on Information Theory, ISBN: 90-71048-14-4, abstract , BibTeX ,

Source code

 MATLAB Function Reference