Using Sufficient Statistics for Bayesian Inference

Sufficient Statistics

Given a statistics $T(X_1, X_2, …, X_n) $, the following two definitions are equivalent:

  1. \(f(x_1, x_2, ..., x_n|\theta, T = t) = f(x_1, x_2, ..., x_n | T = t)\)
  2. \(f(x_1, x_2, ..., x_n) = h(x_1, x_2, ..., x_n)g(\theta, T(x_1, x_2, ..., x_n))\)

Consider exponentials \(X_1, X_2\) and \(T = X_1 + X_2\).

From definition 1:

\[ f(x_1, x_2 | T = t) = \frac{f(x_1)f(x_2)}{P(x_1 + x_2 = t)}\\ =\frac{\theta^2e^{-\theta(x_1 + x_2)}}{\sum_{x=0}^t P(x_1 =t - x | x_2 =x)P(x_2 =x)}\\ =\frac{\theta^2e^{-\theta t}}{\int_0^t f(x_1 = t - x)f(x_2 =x)dx} \\ =\frac{\theta^2e^{-\theta t}}{\int_0^t\theta^2 e^{-\theta(t -x )}e^{-\theta x}dx} \\ =\frac{e^{-\theta t}}{\int_0^t e^{-\theta t}dx} \\ =\frac{1}{t} \]

From definition 2:

\[ f(x_1, x_2) = \theta^2 e^{-\theta(x_1 +x_2)} = \theta^2 e^{-\theta T(x_1, x_2)} \]

where \(h(x_1, x_2) = 1\) and \(g(\theta, T(x_1, x_2)) =\theta^2 e^{-\theta T(x_1, x_2)}\).

Bayesian Inference

When doing Bayesian inference, using a sufficient statistics will yield the same result as if we used all samples.

Examples

Exponential

using all samples:

\[ f(\lambda|x_1, x_2, ..., x_n) =\frac{f(x_1, x_2, ..., x_n| \lambda)f(\lambda)}{\int_0^{\infty}f(x_1, x_2, ..., x_n)f(\lambda)d\lambda} \\ =\frac{\lambda^{n-1}e^{-s\lambda}}{\int_0^{\infty}\lambda^{n-1}e^{-s\lambda}d\lambda}\\ =\frac{\lambda^{n-1}s^n e^{-s\lambda}}{(n-1)!}\\ \]

\[ f(\lambda | x_1, x_2, ..., x_n) = \frac{\lambda ^n s^{n+1} e^{-\lambda s}}{n!} \]

using only \(S\), which is n-stage Erlang:

\[ f(\lambda | s) = \frac{f(s|\lambda)f(\lambda)}{\int_0^{\infty}f(s|\lambda)f(\lambda)d\lambda}\\ =\frac{\lambda ^{n-1} s^n e^{-\lambda s}}{(n-1)!} \]

\[ f(\lambda | s) = \frac{\lambda ^n s^{n+1} e^{-\lambda s}}{n!} \]

Bernoulli

\[ f(p|x_1, x_2, ..., x_n) = \frac{P(x_1, x_2, ..., x_n|p)f(p)}{\int_0^1 P(x_1, x_2, ..., x_n|p)f(p)dp} \\ = \frac{p^s(1-p)^{n-s}p^{-1/2}(1-p)^{-1/2}}{\int_0^1p^s(1-p)^{n-s}p^{-1/2}(1-p)^{-1/2}dp} = \frac{p^{s-1/2}(1-p)^{n-s-1/2}}{\frac{\Gamma \left(s+\frac{1}{2}\right) \Gamma \left(n-s+\frac{1}{2}\right)}{\Gamma (n+1)}} \]

Application

Gamma

\[ f(\alpha, \beta | s, p) \]