2.3 The evidence for the model

1. 1.

   Complex models with more parameters fit better and have a higher likelihood.

2. 2.

   However complex models can predict poorly. Occam’s razor applies to prediction.

3. 3.

   The marginal likelihood penalises the fit by a measure of complexity

Using Bayes theorem the posterior is

$$p(\theta \mid Y,M)=\frac{p(y\mid \theta ,M)p(\theta \mid M)}{p(y\mid M)}$$

Rewriting this with the marginal likelihood as the subject we get

$$p(y\mid M)=\frac{p(y\mid \theta ,M)p(\theta \mid M)}{p(\theta \mid y,M)}$$

Writing this in log form we have

	$\log p(y\mid M)$	$=\underset{\text{log-likelihood}}{\underbrace{\log p(y\mid \theta ,M)}}-\left(\underset{\text{penalty}}{\underbrace{\log p(\theta \mid y,M)-\log p(\theta \mid M)}}\right)$
	$-2\log p(y\mid M)$	$=\mathrm{Deviance}+2\mathrm{P}\mathrm{e}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{t}\mathrm{y}$

This is true for any $\theta$ including the MLE. $\log p(y\mid \theta ,M)$ increases with model complexity as does the penalty.

	$m(y)$	$=\left(\begin{array}{c}\hfill n\hfill \\ \hfill y\hfill \end{array}\right){\displaystyle \frac{{\int }_{\pi =0}^1{\pi }^y{(1-\pi )}^{n-y}{\pi }^{p-1}{(1-\pi )}^{q-1}𝑑\pi }{\text{B}(p,q)}}$
	$m(y)$	$=\left(\begin{array}{c}\hfill n\hfill \\ \hfill y\hfill \end{array}\right){\displaystyle \frac{\text{B}(y+p,n-y+q)}{\text{B}(p,q)}}$

This is the Beta-binomial distribution.

$$y\sim \text{Beta-Binomial }(n,p+y,q+n-y)$$

which has more variation than the binomial.

	$m(y)$	$=$	${\displaystyle {\int }_{\theta =0}^{\mathrm{\infty }}}f(y|\theta )\pi (\theta )𝑑\theta$
		$=$	${\displaystyle {\int }_{\theta =0}^{\mathrm{\infty }}}{\displaystyle \frac{1}{y!}}{e}^{-\theta }{\theta }^y{\displaystyle \frac{q^p}{\mathrm{\Gamma }(p)}}{\theta }^{p-1}{e}^{-q\theta }$
		$=$	${\displaystyle \frac{q^p}{\mathrm{\Gamma }(p)y!}}{\displaystyle {\int }_{\theta =0}^{\mathrm{\infty }}}{\theta }^{y+p-1}{e}^{-\theta (q+1)}$
		$=$	${\displaystyle \frac{q^p}{y!\mathrm{\Gamma }(p)}}{\displaystyle \frac{\mathrm{\Gamma }(y+p)}{{(q+1)}^{y+p}}}$
		$=$	${\displaystyle \frac{q^p}{y!\mathrm{\Gamma }(p)}}{\displaystyle \frac{\mathrm{\Gamma }(y+p)}{{(q+1)}^{y+p}}}$
		$\propto$	${\left({\displaystyle \frac{q}{q+1}}\right)}^p{\left({\displaystyle \frac{1}{q+1}}\right)}^y$

$$⟹y\sim \text{Negative-Binomial }(p,\frac{q}{q+1})$$

1. 1.

   Bayesian inference is obtained by updating our belief in a parameter by multiplying our current belief by the likelihood of the current observation(s) and then normalising. This yields the posterior distribution which can become the prior for the next observation.

2. 2.

   Updating belief in a model is no different, in principle, to updating a parameter.

3. 3.

   The marginal likelihood is the denominator in Bayes theorem.

4. 4.

   The ratio of marginal likelihoods give us a Bayes factor which compares the weight of evidence in favour of two given models.

   $$B{F}_{2,1}=\frac{P(y\mid {ℳ}_1)}{P(y\mid {ℳ}_2)}$$

5. 5.

   Bayes model selection is coherent and uses only the laws of probability.

A coin is tossed 10 times resulting in 9 heads and one tail. The question of interest is whether the coin is biased or not. The null hypothesis, $H_o$, is that the coin is fair, $(\pi =1/2)$, with the alternative hypothesis, $H_a$, is that the coin is biased ($\pi \ne 1/2$).

1. (i)

   Define the $p$-value used in classical hypothesis testing. What is the implication of a low $p$-value?

2. (ii)

   Use a classical hypothesis test to test whether the coin is fair with a 5% significance level.

3. (iii)

   If I assume (before the experiment) that each hypothesis is equally likely, calculate the Bayes Factor for $H_a$ relative to $H_o$ (If necessary state any further assumptions that you might need to make).

4. (iv)

   What is the probability that the coin is biased?

1. (i)

   A $p$-value is the probability of getting a test statistic at least as extreme as that observed under the null hypothesis by pure chance. A low p-value is evidence against the null hypothesis.

2. (ii)

   $p(X\in \{9,10\})={0.5}^{10}+10*{0.5}^{10}=0.01074$. The test is two tailed. Therefore the $p-$ value is $2\times 0.01074\approx 0.02$
   Since Reject $H_o$ in favour of $H_a$ with 95% certainty.

3. (iii)

   If we assume that under $H_a$, $\pi \sim \text{Beta }(\alpha ,\beta )$ where $\alpha =\beta =1$. The normalize constant for the binomial is $k=10$

   	${\displaystyle \frac{p(y\mid H_0)}{P(y\mid H_a)}}$	$={\displaystyle \frac{k\times {0.5}^{10}}{\int p(y\mid \pi )P(\pi )𝑑\pi }}$
   		$={\displaystyle \frac{k\times {0.5}^{10}}{k\times \text{𝐁}(y+\alpha ,n-y+\beta )}}$
   		$=0.1074$

1. (iv)

   Therefore if $P(H_o)=p(H_a)$ then

   ${\displaystyle \frac{1-p(H_a)}{p(H_a)}}=0.1074$

   $⟹p(H_a)=\frac{1}{1+0.1074}=0.903$ [4]