MATLAB Function Reference

entropy

Estimate the entropy of a stationary signal with independent samples

Syntax

Description

Entropy estimation is a two stage process; first a histogram is estimated and thereafter the entropy is calculated. For the explanation of the usage of the descriptor of the histogram see histogram .

In case of a disrete stochastic variable i in the integer subrange lower <= i < upper the descriptor should be selected as [lower,upper,upper-lower]. The R(epresentation)-unbiased entropy will be estimated.

In case of a continuous stochastic variable the descriptor can be left unspecified. In this case the default descriptor of histogram will be used.

The estimate depends on the value of approach

The base of the logarithm determines the unit of measurement. Default base e (nats) is used, alternative choises are 2 (bit) and 10 (Hartley).

As a result the function returns the estimate, the N-bias (Nbias) of the estimate, the estimated standard error sigma and the used descriptor.

Example

Estimate the entropy of a discrete stochastic variable with probability 0.25 on an observed 0 and probability 0.75 on an observed 1. The entropy is 0.5623 nat.

>> signal(1:250)=0;
>> signal(251:1000)=1;
>> [estimate,nbias,sigma,descriptor] =entropy(signal,[0,2,2],'u')
estimate = 0.5628
nbias = 0
sigma = 0.0151
descriptor = 0 2 2

Estimate the entropy of a Gaussian distributed continuous variable with zero mean and unit variance. The entropy is 1.4189 nat.

>> signal=normrnd(0,1,1,1000);
>> [estimate,nbias,sigma,descriptor] =entropy(signal,[-3,3,30])
estimate = 1.3643
nbias = 0
sigma = 0.0208
descriptor = -3 3 30

See Also

entropy2
histogram

Literature

Moddemeijer, R. On Estimation of Entropy and Mutual Information of Continuous Distributions, Signal Processing, 1989, vol. 16, nr. 3, pp. 233-246, abstract , BibTeX ,

For the principle of Minimum Mean Square Error estimation see:

Moddemeijer, R. An efficient algorithm for selecting optimal configurations of AR-coefficients, Twentieth Symp. on Information Theory in the Benelux, May 27-28, 1999, Haasrode (B), pp 189-196, eds. A. Barbé et. al., Werkgemeenschap Informatie- en Communicatietheorie, Enschede (NL), and IEEE Benelux Chapter on Information Theory, ISBN: 90-71048-14-4, abstract , BibTeX ,

Source code

entropy.m

MATLAB Function Reference


Copyright R. Moddemeijer