On Estimation of Entropy and Mutual Information of Continuous Distribution

R. Moddemeijer

University of Twente, Department of Electrical Engineering,
P.O. Box 217, NL-7500 AE Enschede, The Netherlands,

present addres

University of Groningen, Department of Computing Science,
P.O. Box 800, NL-9700 AV Groningen, The Netherlands,
phone: +31.50.363 3940 - fax: +31.50.363 38005 - e-mail: rudy@cs.rug.nl

Abstract

Mutual information is uded in a procedure to estimate time-delays between recordings of electroencephalogram (EEG) signals originating from epileptic animals or patients. We present a simple and reliable histogram-based method to estimate mutual information. The accuracies of this mutual information estimator and of a similar entropy estimator are discussed. The bias and variance calculations presented can also be applied to discrete valued systems. Finally we present some simulation results, which are compared with earlier work.

Keywords

Mutual information, entropy, bias, variance, estimator, delay estimation.

Published

Signal Processing, 1989, vol. 16, nr. 3, pp. 233-246, BibTeX

Related work

Thesis: Delay-Estimation with Application to Electroencephalograms in Epilepsy
Conference presentation: On estimation of entropy and mutual information of continuous distributions
Conference presentation: Estimation of entropy and mutual information of continuous distributions ,
Later work: A statistic to estimate the variance of the histogram based mutual information estimator based on dependent pairs of observations

Matlab sources:

entropy.m, Estimate the entropy of a stationary signal with independent samples, documentation.
entropy2.m, Estimate the entropy of two stationary signal with independent pairs of samples, documentation.
information.m, Estimate the mutual information of two stationary signal with independent pairs of samples, documentation.

other publications