3.1 Examples

Suppose $y\sim \text{Binomial }(n,\pi )$ and our (conjugate) prior for $\pi$ is $\pi \sim \text{Beta }(p,q)$. The posterior for $\pi$ is given by:

$$\pi |y\sim \text{Beta }(p+y,q+n-y).$$

We let $y^{\star }$ be the number of successes in those $N$ trials, so that

$$f(y^{\star }|\pi )=\left(\genfrac{}{}{0pt}{}{N}{y^{\star }}\right){\pi }^{y^{\star }}{(1-\pi )}^{N-y^{\star }}.$$

	$f(y^{\star }|y)$	$=$	${\displaystyle {\int }_0^1}\left({\displaystyle \genfrac{}{}{0pt}{}{N}{y^{\star }}}\right){\pi }^{y^{\star }}{(1-\pi )}^{N-y^{\star }}\times {\displaystyle \frac{{\pi }^{p+y-1}{(1-\pi )}^{q+n-y-1}}{\text{B}(p+y,q+n-y)}}𝑑\pi$
		$=$	$\left({\displaystyle \genfrac{}{}{0pt}{}{N}{y^{\star }}}\right){\displaystyle \frac{1}{B(p+y,q+n-y)}}{\displaystyle {\int }_0^1}{\pi }^{p+y+y^{\star }-1}{(1-\pi )}^{N-y^{\star }+q+n-y-1}$
		$=$	$\left({\displaystyle \genfrac{}{}{0pt}{}{N}{y^{\star }}}\right){\displaystyle \frac{\text{B}(y^{\star }+p+y,N-y^{\star }+q+n-y)}{\text{B}(p+y,q+n-y)}}$

This is known as a Beta-binomial distribution. $y^{\star }\sim \text{Beta-Binomial }(N,P,Q)$ where $P=p+y$ and $Q=q+n-y$.

	$Y\mid \theta$	$\sim \text{Binomial }(n,\theta ),\theta >0$
	$\theta$	$\sim \text{Beta }(p,q)$
	$\theta \mid y$	$\sim \text{Beta }(p+y,q+n-y)$
	$y^{*}\mid \theta$	$\sim \text{Binomial }(N,\theta )$
	$y^{*}\mid y$	$\sim \text{Beta-Binomial }(N,p+y,q+n-y)$

This is the probability of a future observation $y^{\star }$ given counts $y_1,y_2,\mathrm{\dots },y_n$ have been observed. For the predictive, we integrate the likelihood over the posterior Gamma $(P,Q)$, Note $P=p+{\sum }_{i=1}^n{y}_i$ and $Q=q+n$.

	$f(y^{*}\mid y)$	$=$	${\displaystyle {\int }_{\theta =0}^{\mathrm{\infty }}}f(y^{*}|\theta )\pi (\theta \mid y)𝑑\theta$
		$=$	${\displaystyle {\int }_{\theta =0}^{\mathrm{\infty }}}{\displaystyle \frac{1}{y^{*}!}}{e}^{-\theta }{\theta }^{y^{*}}{\displaystyle \frac{Q^P}{\mathrm{\Gamma }(P)}}{\theta }^{P-1}{e}^{-Q\theta }𝑑\theta$
		$=$	${\displaystyle \frac{Q^P}{y^{*}!\mathrm{\Gamma }(P)}}{\displaystyle {\int }_{\theta =0}^{\mathrm{\infty }}}{\theta }^{y^{*}+P-1}{e}^{-\theta (Q+1)}𝑑\theta$
		$=$	${\displaystyle \frac{Q^P}{y^{*}!\mathrm{\Gamma }(P)}}{\displaystyle \frac{\mathrm{\Gamma }(y+P)}{{(Q+1)}^{y^{*}+P}}}$
		$=$	${\displaystyle \frac{Q^P}{y^{*}!\mathrm{\Gamma }(P)}}{\displaystyle \frac{\mathrm{\Gamma }(y^{*}+P)}{{(Q+1)}^{y^{*}+P}}}$
		$\propto$	${\displaystyle \frac{\mathrm{\Gamma }(y^{*}+P)}{y^{*}!}}{\left({\displaystyle \frac{Q}{Q+1}}\right)}^P{\left({\displaystyle \frac{1}{Q+1}}\right)}^{y^{*}}$

$$⟹y^{*}\mid y\sim \text{Negative-Binomial }(P,\frac{Q}{Q+1})$$

	$Y_i\mid \theta$	$\sim \text{Poisson }\left(\theta \right),\theta >0,y_i\ge 0,i=1,2,\mathrm{\dots },n$
	$\theta$	$\sim \text{Gamma }(p,q)$
	$\theta \mid y$	$\sim \text{Gamma }(p+{\sum }_{i=1}^n{y}_i,q+n)$
	$Y^{*}\mid \theta$	$\sim \text{Poisson }\left(\theta \right)$
	$⟹Y^{*}\mid y$	$\sim \text{Negative-Binomial }(p+{\sum }_{i=1}^n{y}_i,\frac{q+n-{\sum }_{i=1}^n{y}_i}{1+q+n-{\sum }_{i=1}^n{y}_i})$

The following identities are useful particularly with the Normal distribution because the Normal distribution can be described completely by it mean and variance.

	$\text{E }(y^{\star })$	$\equiv {\text{E }}_{\lambda }{\text{E }}_{y^{\star }\mid \lambda }$		(3.2)
	$\mathrm{Var}(y^{\star })$	$\equiv {\text{E }}_{\lambda }{\mathrm{Var}}_{y^{\star }\mid \lambda }+{\mathrm{Var}}_{\lambda }{\text{E }}_{y^{\star }\mid \lambda }.$		(3.3)

Show that the mean and variance of the predictive of a Poisson likelihood with a gamma prior can be expressed as (where $P=p+\sum y_i$ and $Q=q+n$.):

	$\text{E }(y^{\star })$	$={\displaystyle \frac{P}{Q}}$
	$\mathrm{Var}({\mathrm{y}}^{\star })$	$={\displaystyle \frac{P}{Q}}+{\displaystyle \frac{P}{Q^2}}$

Hint: Use the identities Equations (3.2) and (3.3). where $\text{E }(y^{\star }\mid \lambda )=\lambda$, $\mathrm{Var}(y^{\star }\mid \lambda )=\lambda$, $\text{E }(\lambda )=\frac{P}{Q}$ and $\mathrm{Var}(\lambda )=\frac{P}{Q^2}$.

Note that the uncertainty in a future observation is denoted by $\mathrm{Var}(y^{\star })$ and that

$$\underset{p,q\to 0}{lim}\mathrm{Var}(y^{\star })=\frac{\sum y_i}{n}+\frac{\sum y_i}{n^2}$$

• •

  The uncertainty of a future observation can be split up into two parts

  1. (1)

     Uncertainty of the sampling distribution : $\frac{\sum y_i}{n}$

  2. (2)

     Parameter uncertainty: $\frac{\sum y_i}{n^2}$

• •

  Predicting only from the MLE only gives (1) but fails to take into account (2).

• •

  Parameter uncertainly gets smaller for large samples: ${lim}_{n\to \mathrm{\infty }}\frac{\sum y_i}{n^2}=0$.

Recall that the sum of independent normal random variables is also normal. Therefore, since both $\mu$ and $\widetilde{ϵ}$, conditional on y and ${\sigma }^2$, are normally distributed, so is $Y^{\star }=\mu +\widetilde{ϵ}$. The predictive distribution is therefore

$$Y^{\star }|\text{𝐲},\sigma ,{\sigma }_p\sim \text{Normal }({\mu }_p,{\sigma }_p^2+{\sigma }^2)$$

It is worthwhile to have some intuition about the form of the variance of $Y^{\star }$ : In general, our uncertainty about a new sample $Y^{\star }$ is a function of our uncertainty about the center of the population ${\sigma }_p^2$ as well as how variable the population is $({\sigma }^2)$. As $n\to \mathrm{\infty }$ we become more and more certain about where $\mu$ is, and the posterior variance ${\sigma }_p^2$ of $\mu$ goes to zero. But certainty about $\mu$ does not reduce the sampling variability ${\sigma }^2$, and so our uncertainty about $Y^{\star }$ never goes below ${\sigma }^2$.

	$Y_i\mid \mu ,\tau$	$\sim \text{Normal }(\mu ,\frac{1}{\tau }),\tau >0,i=1,2,\mathrm{\dots },n$
	$\mu$	$\sim \text{Normal }({\mu }_0,\frac{1}{{\tau }_0})$
	$\mu \mid y,\tau$	$\sim \text{Normal }(\frac{{\mu }_0{\tau }_0+n\tau \overline{y}}{{\tau }_0+n\tau },\frac{1}{{\tau }_0+n\tau })$
	$Y^{*}\mid \mu ,\tau$	$\sim \text{Normal }(\mu ,\frac{1}{\tau })$
	$Y^{*}\mid y,\tau$	$\sim \text{Normal }(\frac{{\mu }_0{\tau }_0+n\tau \overline{y}}{{\tau }_0+n\tau },\frac{1}{{\tau }_0+n\tau }+\frac{1}{\tau })$

Note as $\tau \to 0Y^{*}\mid y,\tau \sim \text{Normal }(\overline{y},\left(\frac{1}{n}+1\right)\frac{1}{\tau })$ It can be shown that

$$Y^{*}\mid y\sim \text{t }(\overline{y},s^2\left(1+\frac{1}{n}\right),n-1)$$

where $s^2$ is the sample variance.