differential entropy

Let $(X,\mathfrak{B},\mu)$ be a probability space, and let $f\in L^{p}(X,\mathfrak{B},\mu)$, $||f||_{p}=1$ be a function. The differential entropy $h(f)$ is defined as

 $h(f)\equiv-\int_{X}|f|^{p}\log|f|^{p}\ d\mu$ (1)

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}]=-\sum_{i}p_{i}\log p_{i}$. Consider first $u_{a}$, the uniform 1-dimensional distribution on $(0,a)$. The differential entropy is

 $h(u_{a})=-\int_{0}^{a}\frac{1}{a}\log\frac{1}{a}\ d\mu=\log a.$ (2)

Next consider probability distributions such as the function

 $g=\frac{1}{2\pi\sigma}e^{-\frac{(t-\mu)^{2}}{2\sigma^{2}}},$ (3)

the 1-dimensional Gaussian. This pdf has differential entropy

 $h(g)=-\int_{\mathbb{R}}g\log g\ dt=\frac{1}{2}\log 2\pi e\sigma^{2}.$ (4)

For a general $n$-dimensional Gaussian (http://planetmath.org/JointNormalDistribution) $\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$, $K_{ij}=\mathrm{cov}(x_{i},x_{j})$, we have

 $h(\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K}))=\frac{1}{2}\log(2\pi e)^{n}|% \mathbf{K}|$ (5)

where $|\mathbf{K}|=\det{\mathbf{K}}$.

Title differential entropy DifferentialEntropy 2013-03-22 12:18:48 2013-03-22 12:18:48 Mathprof (13753) Mathprof (13753) 16 Mathprof (13753) Definition msc 54C70 ShannonsTheoremEntropy ConditionalEntropy