MATLAB Function Reference

entropy

Estimate the entropy of a stationary signal with independent samples

Syntax

```[estimate,Nbias,sigma,descriptor] = entropy(x)
[estimate,Nbias,sigma,descriptor] = entropy(x,descriptor)
[estimate,Nbias,sigma,descriptor] = entropy(x,descriptor,base)
[estimate,Nbias,sigma,descriptor] = entropy(x,descriptor,base,approach)
```

Description

Entropy estimation is a two stage process; first a histogram is estimated and thereafter the entropy is calculated. For the explanation of the usage of the `descriptor` of the histogram see histogram .

In case of a disrete stochastic variable `i` in the integer subrange `lower <= i < upper` the descriptor should be selected as `[lower,upper,upper-lower]`. The R(epresentation)-unbiased entropy will be estimated.

In case of a continuous stochastic variable the descriptor can be left unspecified. In this case the default descriptor of histogram will be used.

The estimate depends on the value of `approach`

• `'unbiased'`: a N(umber)-unbiased estimate (default),
• `'biased'`: a N(umber)-biased estimate and
• `'mmse'`: a minimum Mean Square Error estimate, obtained by balancing bias and variance after N-bias correction.
The `base` of the logarithm determines the unit of measurement. Default base e (nats) is used, alternative choises are 2 (bit) and 10 (Hartley).

As a result the function returns the `estimate`, the N-bias (`Nbias`) of the estimate, the estimated standard error `sigma` and the used `descriptor`.

Example

Estimate the entropy of a discrete stochastic variable with probability 0.25 on an observed 0 and probability 0.75 on an observed 1. The entropy is 0.5623 nat.

``` >> signal(1:250)=0; >> signal(251:1000)=1; >> [estimate,nbias,sigma,descriptor] =entropy(signal,[0,2,2],'u') estimate = 0.5628 nbias = 0 sigma = 0.0151 descriptor = 0 2 2 ```

Estimate the entropy of a Gaussian distributed continuous variable with zero mean and unit variance. The entropy is 1.4189 nat.

``` >> signal=normrnd(0,1,1,1000); >> [estimate,nbias,sigma,descriptor] =entropy(signal,[-3,3,30]) estimate = 1.3643 nbias = 0 sigma = 0.0208 descriptor = -3 3 30 ```

See Also

Literature

Moddemeijer, R. On Estimation of Entropy and Mutual Information of Continuous Distributions, Signal Processing, 1989, vol. 16, nr. 3, pp. 233-246, abstract , BibTeX ,

For the principle of Minimum Mean Square Error estimation see:

Moddemeijer, R. An efficient algorithm for selecting optimal configurations of AR-coefficients, Twentieth Symp. on Information Theory in the Benelux, May 27-28, 1999, Haasrode (B), pp 189-196, eds. A. Barbé et. al., Werkgemeenschap Informatie- en Communicatietheorie, Enschede (NL), and IEEE Benelux Chapter on Information Theory, ISBN: 90-71048-14-4, abstract , BibTeX ,

Source code

 MATLAB Function Reference

Copyright R. Moddemeijer