2.2 Conjugates for likelihoods from the exponential family

(Gamma sample.) Let $X_1,\mathrm{\dots }{X}_n$ be independent variables having the Gamma $(k,\theta )$ distribution, where $k$ is known. Note that the case $k=1$ corresponds to the exponential distribution. So

$$L(\theta ;x)\propto {\theta }^{nk}\exp \{-\theta \mathrm{\Sigma }{x}_i\}.$$

Now, studying this form, regarded as a function of $\theta$ suggests we could take a prior of the form

	$\pi (\theta )$	$\propto {\theta }^{p-1}\exp \{-q\theta \}$
	$⟹\theta$	$\sim \text{Gamma }(p,q)$

	$\pi (\theta \mid x)$	$\propto \pi (\theta )L(\theta ;x)$
		$\propto {\theta }^{p-1}\exp \{-q\theta \}{\theta }^{nk}\exp \{-\theta \mathrm{\Sigma }{x}_i\}$
		$\propto {\theta }^{p+nk-1}\exp \{-(q+\mathrm{\Sigma }{x}_i)\theta \},$

and so

$$\theta |x\sim \text{Gamma }(p+nk,q+\sum x_i)$$

.

It emerges that the only case where conjugates can be easily obtained is for data models within the exponential family.

That is,

$$f(x|\theta )=h(x)g(\theta )\exp \{t(x)c(\theta )\}$$

for functions $h,g,t$ and $c$.

This might seem restrictive, but in fact includes the exponential distribution, the Poisson distribution, the gamma distribution with known shape parameter, the binomial distribution and the normal distribution with known variance.

Given a random sample $x=(x_1,x_2,\mathrm{\dots },x_n)$ from this general distribution, the likelihood for $\theta$ is then

	$L(\theta ;x)$	$=$	${\displaystyle \prod _{i=1}^n}\{h(x_i)\}g{(\theta )}^n\exp \{{\displaystyle \sum _{i=1}^n}t(x_i)c(\theta )\}$
		$\propto$	$g{(\theta )}^n\exp \{{\displaystyle \sum _{i=1}^n}t(x_i)c(\theta )\}.$

Thus if we choose a prior of the form

$$\pi (\theta )\propto g{(\theta )}^d\exp \{bc(\theta )\},$$

we obtain

	$\pi (\theta |x)$	$\propto$	$\pi (\theta )L(\theta ;x)$
		$\propto$	$g{(\theta )}^d\exp \{bc(\theta )\}\times g{(\theta )}^n\exp \{{\displaystyle \sum _{i=1}^n}t(x_i)c(\theta )\}$
		$=$	$g{(\theta )}^{n+d}\exp \{[b+{\displaystyle \sum _{i=1}^n}t(x_i)]c(\theta )\}$
		$=$	$g{(\theta )}^D\exp \{Bc(\theta )\},$

where $D=n+d$ and $B=b+\sum t(x_i)$. This results in a posterior in the same family as the prior, but with modified parameters.

	$f(x|\theta )$	$=$	$\left({\displaystyle \genfrac{}{}{0pt}{}{n}{x}}\right){\theta }^x{(1-\theta )}^{n-x}$
		$=$	$\left({\displaystyle \genfrac{}{}{0pt}{}{n}{x}}\right){(1-\theta )}^n{\left({\displaystyle \frac{\theta }{1-\theta }}\right)}^x$
		$=$	$\left({\displaystyle \genfrac{}{}{0pt}{}{n}{x}}\right){(1-\theta )}^n\exp \{x\log ({\displaystyle \frac{\theta }{1-\theta }})\}.$

So, in exponential family notation we have $h(x)=\left(\genfrac{}{}{0pt}{}{n}{x}\right)$, $g(\theta )={(1-\theta )}^n$, $t(x)=x$, and $c(\theta )=\log (\frac{\theta }{1-\theta })$. Thus, we construct a conjugate prior with the form

	$\pi (\theta )$	$\propto$	${[{(1-\theta )}^n]}^d\exp \{b\log ({\displaystyle \frac{\theta }{1-\theta }})\}$
		$=$	${(1-\theta )}^{nd-b}{\theta }^b$

which is a member of the beta family of distributions.

Table 2.1 lists many of the standard prior–likelihood conjugate analysis.

Some of these have been given as examples; the others you should verify.