Moment-based approach for sampling optimization (general)

Parameter Variance

For a general distribution, it is infeasible to compute its posterior distribution because of the integration, let alone variance. However, for the moment-based approach, only mean and variance of parameters are of interest, not the distribution itself, which is only a means of finding mean and variance. Parameter means can be computed through point estimators. Variance is the difficult part. Luckily, Cramér–Rao bound gives a good estimation of variance. The theorem states: the variance of any unbiased estimator of \(\operatorname{Var}[\hat{\Theta}]\) is bounded by the reciprocal of the Fisher information \(\frac{1}{\mathcal{I}(\theta)}​\), i.e. \[ \operatorname{Var}[\hat\Theta] \geq \frac{1}{\mathcal{I}(\theta)} \]

Examples

Exponential (biased)

\[ P(S \leq s) = P(\hat{\Lambda} \geq \frac{n}{s}) \\ F_S(s) = 1- F_{\hat{\Lambda}}(\frac{n}{s}) \\ f_S(s) = f_{\hat\Lambda}(\frac{n}{s})\frac{n}{s^2} \\ f_{\hat\Lambda}(\frac{n}{s}) = f_S(s)\frac{s^2}{n} \\ f_{\hat\Lambda}(\hat\lambda) = f_S(\frac{n}{\hat\lambda})\frac{n}{\hat\lambda^2} \]

\[ \operatorname{E}(\hat\Lambda) = \int_0^\infty \hat\lambda f_{\hat\Lambda}(\hat\lambda)d\hat\lambda = \frac{n \lambda}{n-1} \]

\[ \operatorname{Var}[\hat\Lambda] = \int_0^{\infty} (\hat{\lambda} - \operatorname{E}(\hat\Lambda))^2 f(\hat\lambda)d\hat{\lambda} = \frac{\lambda ^2 n^2}{(n-2) (n-1)^2} \]