5.4 Evaluating the cost or utility of an experiment

Here we need calculate the expected utility or loss when both the parameters and the future data are unknown.

Recall that the expected loss of a decision $d\in 𝒟$ with respect to the state of nature $s\sim \pi (S)$ is

If we are interested in future data, call it $x$, but the data is as yet unknown then the Bayes risk, Equation 5.2, becomes

where ${\rho }^{*}(x)=\underset{d\in 𝒟}{inf}{\mathrm{E}}_{\theta \mid \mathrm{x}}\mathrm{L}(\mathrm{d},\{\mathrm{x},\theta \})$. In other words the Bayes risk over all possible samples is the marginal expectation of the posterior Bayes risk.

This Bayes risk or loss of the sampling procedure can be obtained by

1. 1.

   Choose a conjugate prior, $\pi (\theta )$ to the likelihood $f(x\mid \theta )$.

2. 2.

   Calculate Bayes rule, $d^{*}$, and Bayes risk using this prior, ${\rho }^{*}(\theta )$.

3. 3.

   Update the prior using the likelihood, $f(x\mid \theta )$, and re-calculate Bayes risk as a function of $x$:

   $${\rho }^{*}(x)={\mathrm{E}}_{\theta \mid \mathrm{x}}\mathrm{L}(\mathrm{d},\{\mathrm{x},\theta \}).$$

4. 4.

   The problem is now to calculate the marginal expectation, ${\mathrm{E}}_{\mathrm{x}}\rho (\mathrm{x},\theta )$. This can be done by using the identity

   $${\rho }^{*}={\mathrm{E}}_{\mathrm{x}}{\rho }^{*}(\mathrm{x})={\mathrm{E}}_{\theta }{\mathrm{E}}_{\mathrm{x}\mid \theta }{\rho }^{*}(\mathrm{x})$$

   .

The next example illustrates this procedure.

Suppose that we wish to estimate the parameter, $\theta$, of a Poisson distribution. Our prior for $\theta \sim \text{Gamma }(\alpha ,\beta )$. The loss function, for estimate $d$ and value $\theta$, is

1. 1.

   Find the Bayes rule and Bayes risk of an immediate decision.

2. 2.

   Find the Bayes rule and Bayes risk if we take a sample, $x$, of size $n$.

3. 3.

   Find the Bayes risk of the sampling procedure.

1. (1)

   We have already seen that for a squared loss, Bayes rule is

   $$d^{\star }={\text{E }}_{\pi }\theta =\frac{\alpha }{\beta }$$

   with corresponding Bayes risk

   ${\rho }^{\star }(\pi )$

   $={\text{var }}_{\pi }\theta ={\displaystyle \frac{\alpha }{{\beta }^2}}$

2. (2)

   As $\theta \sim \text{Gamma }(\alpha ,\beta )$ and $X_i\sim \text{Poisson }\left(\theta \right)$ then
   $\theta \mid x\sim \text{Gamma }(\alpha +{\sum }_{i=1}^n{x}_i,\beta +n)$ We observe that the Bayes rule and Bayes risk after sampling can be found from substituting $\alpha +{\sum }_{i=1}^n{x}_i$ for $\alpha$ and $\beta +n$ for $\beta$ to obtain the Bayes rule

   $d^{\star }={\displaystyle \frac{\alpha +{\sum }_{i=1}^n{x}_i}{\beta +n}}$

   ${\rho }^{\star }(f(\theta \mid x))={\displaystyle \frac{\alpha +{\sum }_{i=1}^n{x}_i}{{(\beta +n)}^2}}$

Bayes risk is

1. (3)

   The risk of the sampling procedure is the expected value of the posterior Bayes risk, when viewed as a random quantity,

   	${\text{E }}_X{\rho }^{\star }(f(\theta \mid x))$	$={\text{E }}_X\left[{\displaystyle \frac{(\alpha +{\sum }_{i=1}^n{X}_i)}{{(\beta +n)}^2}}\right]$
   		$={\displaystyle \frac{(\alpha +n\text{E }X)}{{(\beta +n)}^2}}$

   We use $\text{E }(X)={\text{E }}_{\theta }(\text{E }(X|\theta ))={\text{E }}_{\theta }(\theta )=\frac{\alpha }{\beta }$ leading to Bayes risk of

   $${\rho }^{\star }=\frac{(\alpha +n\frac{\alpha }{\beta })}{{(\beta +n)}^2}$$

A game of chance is being played with a possibly biased coin with an unknown probability $\theta$ of landing heads: The rules of the game allow you to win 2 pounds if two successive tosses of the coin land heads, and lose 3 pounds if the first toss lands tails. Consider a loss function, $L(\theta ,a)$ for a bet, where $L(\theta ,a)$ is the money you expect to lose conditional on $\theta$ for actions of betting, $a_1$, or not betting, $a_2$. The loss function in this case is

Your prior distribution for $\theta$ is $\text{Beta}(1,1)$.

1. (i)

   Calculate the prior expected loss of both actions.

2. (ii)

   Should you bet and why or why not?

   You now observe $n=3$ tosses of the coin, and of those tosses it lands heads every time.

3. (iii)

   Calculate the posterior distribution for $\theta$.

4. (iv)

   Calculate the posterior expected loss for each action, using the given loss function and state whether a bet should be made.

1. (i)

   $\text{E }(\theta )=\frac{1}{2}=1$, $\mathrm{Var}(\theta )=\frac{1}{12}$, so $\text{E }({\theta }^2)=\frac{1}{3}$

   	$\text{E }(L(\theta ,a_1))$	$=3-3\text{E }(\theta )-2\text{E }({\theta }^2)$
   		$={\displaystyle \frac{5}{6}}$
   	$\text{E }(L(\theta ,a_1))$	$=0$

2. (ii)

   No, you should not bet. Bayes rule is to minimise expected loss which is zero.

3. (iii)

   	$p(\theta \mid x)$	$\propto p(\theta )p(x\mid \theta )$
   		$\propto {\theta }^x{(1-\theta )}^{n-x}$
   	$\theta \mid x$	$\sim \text{Beta }(x+1,n-x+1)$
   	$\theta \mid x$	$\sim \text{Beta }(4,1)$

1. (iv)

   	$\text{E }(\theta \mid x)$	$={\displaystyle \frac{x+1}{n+2}}={\displaystyle \frac{4}{5}}$
   	$\text{Var }(\theta \mid x)$	$={\displaystyle \frac{(x+1)(n-x+1)}{{(n+2)}^2(n+3)}}={\displaystyle \frac{2}{75}}$
   	$\text{E }({\theta }^2\mid x)$	$={\displaystyle \frac{(x+1)(n-x+1)}{{(n+2)}^2(n+3)}}+{\left({\displaystyle \frac{x+1}{n+2}}\right)}^2={\displaystyle \frac{2}{3}}$

   	$\text{E }(L(\theta ,a_1))$	$=3-3\text{E }(\theta \mid x)-2\text{E }({\theta }^2\mid x)$
   		$=3-3{\displaystyle \frac{x+1}{n+2}}-2\left\{{\displaystyle \frac{(x+1)(n-x+1)}{{(n+2)}^2(n+3)}}-{\left({\displaystyle \frac{x+1}{n+2}}\right)}^2\right\}$
   		$=-{\displaystyle \frac{11}{15}}$
   	$\text{E }(L(\theta ,a_1))$	$=0\mathit{\hspace{1em}\hspace{1em}}\mathrm{Now}\mathrm{you}\mathrm{should}\mathrm{bet}.$

We wish to estimate the parameter, $\lambda$, of a Poisson distribution. Our prior for $\lambda$ is Gamma $(\alpha ,\beta )$. The loss function, for estimate $d$ and value $\lambda$, is

1. (i)

   Find the Bayes rule and show that Bayes risk of an immediate decision is

   $$\frac{4\alpha +3{\alpha }^2}{4{\beta }^2}.$$

2. (ii)

   Find the Bayes rule and Bayes risk when we have observed $X=x$.

3. (iii)

   Explain how you would calculate the integrated risk of the sampling procedure.

1. (i)

   Find the Bayes rule and show Bayes risk of an immediate decision is

   $$\frac{4\alpha +3{\alpha }^2}{4{\beta }^2}$$

   Bayes rule, $d^{*}$ is the decision that minimises the expected loss.

   	$\rho (d)$	$={\text{E }}_{\lambda }(L(\lambda ,d)$
   		$={\text{E }}_{\lambda }({\lambda }^2)-{\text{E }}_{\lambda }(\lambda )d+d^2$
   	${\rho }^{{}^{\prime }}(d)$	$=-{\text{E }}_{\lambda }(\lambda )+2d=0$
   	$d^{*}$	$={\displaystyle \frac{E_{\lambda }(\lambda )}{2}}$
   		$={\displaystyle \frac{\alpha }{2\beta }}$

Bayes risk is the expected loss at the optimal decision.

1. (ii)

   Find the Bayes rule and Bayes risk when we have observed $X=x$.
   We note that the posterior distribution of $\lambda$ is $\text{Gamma }({\alpha }_p,{\beta }_p)=\text{Gamma }(x+\alpha ,1+\beta )$.

   	$d^{*}$	$={\displaystyle \frac{{\alpha }_p}{2{\beta }_p}}={\displaystyle \frac{\alpha +x}{2(\beta +1)}}$
   	$\rho (d^{*})$	$={\displaystyle \frac{4{\alpha }_p+3{\alpha }_p^2}{4{\beta }_p^2}}={\displaystyle \frac{4(\alpha +x)+3{(\alpha +x)}^2}{4{(\beta +1)}^2}}$

2. (iiii)

   Explain how you would calculate the integrated risk of the sampling procedure.
   The risk of the sampling procedure an be found by taking the marginal expectation of the posterior risk

Consider the following loss function

• (i)

  Find the Bayes rule of an immediate decision.

• (ii)

  Let $X_1,\mathrm{\dots },X_n$ be exchangeable so that the $X_i$ are conditionally independent given parameter $\theta$. Suppose that $X_i|\theta \sim \text{Poisson }\left(\theta \right)$ and $\theta \sim \text{Gamma }(\alpha ,\beta )$ with $\alpha >1$.

  Under the loss given above, find the Bayes rule after observing $x=(x_1,\mathrm{\dots },x_n)$.

Find the Bayes rule of an immediate decision.

1. (i)

   	$\rho (d)$	$=d\text{E }\left({\displaystyle \frac{1}{\theta }}\right)-\log (d)+\log (\theta )-1$
   	${\rho }^{{}^{\prime }}(d)$	$=\text{E }({\displaystyle \frac{1}{\theta }})-{\displaystyle \frac{1}{d}}$
   	${\rho }^{{}^{\prime }}(d^{*})=0$	$⟹d^{*}=\left[\text{E }\left({\displaystyle \frac{1}{\theta }}\right)\right]$

   Can show for $\theta \sim \text{Gamma }(\alpha ,\beta )$
   

   	$\text{E }\left({\displaystyle \frac{1}{\theta }}\right)$	$={\displaystyle \frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}}{\displaystyle \frac{\mathrm{\Gamma }(\alpha -1)}{{\beta }^{\alpha -1}}}$
   		$={\displaystyle \frac{\beta }{\alpha -1}}$
   	$⟹d^{*}$	$={\displaystyle \frac{\alpha -1}{\beta }}$

Under the loss given above, find the Bayes rule after observing x.

1. (ii)

   $$\theta \mid x\sim \text{Gamma }(\alpha +\sum x_i,\beta +n)$$

   ,

   	$d^{*}$	$={\displaystyle \frac{{\alpha }^{*}-1}{{\beta }^{*}}}$
   		$={\displaystyle \frac{\alpha +\sum x_i}{\beta +n}}$

A certain coin has probability $\omega$ of landing heads. We assess that the prior for $\omega$ is a Beta distribution with parameters $\alpha ,\beta >1$. The loss function for estimate $d$ and value $\omega$ is

1. (a)

   Find the Bayes rule and Bayes risk for an immediate decision.

2. (b)

   Suppose that we toss the coin $n$ times before making our decision, and observe $k$ heads and $n-k$ tails. Find the new Bayes rule and Bayes risk.

1. (a)

   	$\rho (d)$	$=\text{E }(w)-2d+2d^2\text{E }(w^{-1})$
   	${\rho }^{{}^{\prime }}(d)$	$=-2+4d\text{E }(w^{-1}))$
   	${\rho }^{{}^{\prime }}(d^{*})=0$	$⟹d^{*}={\displaystyle \frac{{\text{E }}^{-1}(w^{-1})}{2}}$

   with Bayes risk $\rho (d^{*})=\text{E }(w)-2d^{*}+2d^{*}{}^2\text{E }(w^{-1})=\text{E }(w)-\frac{{\text{E }}^{-1}(w^{-1})}{2}$.

   Can show

   	$E(w^{-1})$	$={\displaystyle \frac{\mathrm{\Gamma }(\alpha +1)}{\mathrm{\Gamma }(\alpha +\beta )}}{\displaystyle \frac{\mathrm{\Gamma }(\alpha )}{\mathrm{\Gamma }(\alpha +\beta -1)}}$
   	$\mathrm{BayesRule}=d^{*}=$	$={\displaystyle \frac{\alpha +\beta -1}{\alpha -1}}$

Bayes risk can be found

1. (b)

   The posterior of $w$ is $w\mid k\sim \text{Beta }(\alpha +k,\beta +n-k)$.

   	$\mathrm{BayesRule}=d^{*}=$	${\displaystyle \frac{{\alpha }^{*}+{\beta }^{*}-1}{{\alpha }^{*}-1}}$
   		$={\displaystyle \frac{\alpha +k-1}{2(\alpha +\beta +n-1)}}$
   	$\mathrm{BayesRisk}$	$={\displaystyle \frac{{\alpha }^{*}({\alpha }^{*}+{\beta }^{*}-1)+{\beta }^{*}}{2({\alpha }^{*}+{\beta }^{*})({\alpha }^{*}+{\beta }^{*}-1)}}$
   		$={\displaystyle \frac{(\alpha +k)(\alpha +\beta +n-1)+\beta +n-k}{2(\alpha +\beta +n)(\alpha +\beta +n-1)}}$