Fisher Information

Definition

Fisher Information \(\mathcal{I}(\theta)\) quantifies the amount of information (a most mysterious word) a sequence of observation gives, on average, about a parameter, where \(\theta\) is the true parameter value.

Let \(f(x; \theta)\) be the likelihood/density function, then \(f(X; \theta)\) is a random variable. \(\mathcal{I}(\theta)\) can be computed in two ways:

Either

\[ \mathcal{I}(\theta) = \operatorname{E}[(\frac{\partial \log f(X; \theta)}{\partial \theta})^2] = \int_{-\infty}^{\infty} (\frac{\partial \log f(X; \theta)}{\partial \theta})^2 f(x; \theta) dx \]

or

\[ \mathcal{I}(\theta) = -\operatorname{E}[\frac{\partial^2 \log f(X; \theta)}{\partial \theta^2}] = -\int_{-\infty}^{\infty}\frac{\partial^2 \log f(X; \theta)}{\partial \theta^2} f(x; \theta) dx \]

Examples

Exponential

\[ \begin{split} \mathcal{I}(\theta) &= -\operatorname{E}[\frac{\partial^2 (n\log \lambda -\lambda s)}{\partial \lambda^2}] \\ &=\operatorname{E}[\frac{n}{\lambda^2}] \\ &=\int_0^\infty \int_0^\infty ..\int_0^\infty \frac{n}{\lambda^2} \lambda^n e^{-\lambda (x_1 +x_2 + ... + x_n)}dx_1dx_2...dx_n \\ &= \frac{n}{\lambda^2} \end{split} \]

Multi-parameter case

With multiple parameters, Fisher info will be in its matrix form. For example, in two-parameter case, $$ () = \[\begin{bmatrix} \operatorname{E}[(\frac{\partial}{\partial\theta_1}\log f(X; \theta))^2] & \operatorname{E}[\frac{\partial}{\partial\theta_1}\log f(X; \theta)\frac{\partial}{\partial\theta_2}\log f(X; \theta) \\ \operatorname{E}[\frac{\partial}{\partial\theta_2}\log f(X; \theta)\frac{\partial}{\partial\theta_1}\log f(X; \theta)] & \operatorname{E}[(\frac{\partial}{\partial\theta_2}\log f(X; \theta))^2] \end{bmatrix}\] \[ or, \] () = - \[\begin{bmatrix} \operatorname{E}[\frac{\partial^2}{\partial\theta_1^2}\log f(X; \theta)] & \operatorname{E}[\frac{\partial^2}{\partial\theta_1\partial\theta_2}\log f(X; \theta)] \\ \operatorname{E}[\frac{\partial^2}{\partial\theta_2\partial\theta_1}\log f(X; \theta)] & \operatorname{E}[\frac{\partial^2}{\partial\theta_2^2}\log f(X; \theta)] \end{bmatrix}\]

$$

Examples

Weibull