Examples

Example 5.3:  Normal data, $\varphi \mathrm{=}\mu$.

$X_i\sim N(\mu ,{\sigma }^2)$. We saw previously that $\widehat{\mu }=\overline{x}$ and se$(\widehat{\mu })=\sigma /n^{1/2}$. Find the two forms of confidence interval for $\mu$.

The asymptotic normality approach gives a $(1-\alpha )\times 100\%$ confidence interval for $\mu$ as follows.

We have already obtained that

Here $\varphi =g(\overrightarrow{\theta })=\mu$. Then, $\widehat{\varphi }=\widehat{\mu }$ and $▽g{(\overrightarrow{\theta })}^T=[1,0]$, so

Therefore the confidence interval is

$$(\overline{x}-z_{\alpha /2}\frac{\widehat{\sigma }}{\sqrt{n}},\overline{x}+z_{\alpha /2}\frac{\widehat{\sigma }}{\sqrt{n}}),$$

where

$$\widehat{\sigma }={\left(\frac{1}{n}\sum _{i=1}^n{(x_i-\overline{x})}^2\right)}^{1/2},$$

is the maximum likelihood estimate of $\sigma$.

Alternatively, we can derive confidence intervals using the profile deviance. The first step is to evaluate the profile log-likelihood for $\mu$. Recall,

$$\mathrm{\ell }(\mu ,\sigma )=-\frac{n}{2}\log (2\pi )-n\log \sigma -\frac{1}{2{\sigma }^2}\sum _{i=1}^n{(x_i-\mu )}^2.$$

Fixing $\mu$ and maximising with respect to $\sigma$, the maximum occurs at ${\widehat{\sigma }}_{\mu }$, where

$$\frac{\partial \mathrm{\ell }(\mu ,{\widehat{\sigma }}_{\mu })}{\partial \sigma }=0.$$

Thus,

$$0=-\frac{n}{{\widehat{\sigma }}_{\mu }}+\frac{1}{{\widehat{\sigma }}_{\mu }^3}\sum _{i=1}^n{(x_i-\mu )}^2,$$

and so

$${\widehat{\sigma }}_{\mu }={\left(\frac{1}{n}\sum _{i=1}^n{(x_i-\mu )}^2\right)}^{1/2}.$$

Hence,

	$D^{*}(\mu )$	$=$	$-2n\log (\widehat{\sigma }/{\widehat{\sigma }}_{\mu })+{\displaystyle \frac{1}{{\widehat{\sigma }}_{\mu }^2}}{\displaystyle \sum _{i=1}^n}{(x_i-\mu )}^2-{\displaystyle \frac{1}{{\widehat{\sigma }}^2}}{\displaystyle \sum _{i=1}^n}{(x_i-\widehat{\mu })}^2$
		$=$	$-2n\log (\widehat{\sigma }/{\widehat{\sigma }}_{\mu })+{\displaystyle \frac{n{\widehat{\sigma }}_{\mu }^2}{{\widehat{\sigma }}_{\mu }^2}}-{\displaystyle \frac{n{\widehat{\sigma }}^2}{{\widehat{\sigma }}^2}}$
		$=$	$-2n\log (\widehat{\sigma }/{\widehat{\sigma }}_{\mu }).$

For a simulated dataset of size $n=25$ with $\mu =0,\sigma =1$, the profile deviance is plotted in the left panel of Figure Figure LABEL:nproflik (Link), leading to a 95% confidence interval of $(-0.12,0.48)$ for $\mu$.

The connection between profile likelihood and likelihood is shown in the right panel of the figure. In this figure contours of the 2-dimensional likelihood surface are plotted, together with the path $(\mu ,{\widehat{\sigma }}_{\mu })$ of the profile likelihood for $\mu$.

Example 5.4:  Normal data, other $\varphi$s.

$X_i\sim N(\mu ,{\sigma }^2)$. We have already obtained that

1. 1.

   If $\varphi =g(\overrightarrow{\theta })={\sigma }^2$. Then, $\widehat{\varphi }={\widehat{\sigma }}^2$ and $▽g{(\overrightarrow{\theta })}^T=(0,2\sigma )$, so

2. 2.

   If $\varphi =g(\overrightarrow{\theta })=\mu +\sigma {\mathrm{\Phi }}^{-1}(1-p)$, with $p$ known.

   Then, $\widehat{\varphi }=\widehat{\mu }+\widehat{\sigma }{\mathrm{\Phi }}^{-1}(1-p)$ and $▽g{(\overrightarrow{\theta })}^T=[1,{\mathrm{\Phi }}^{-1}(1-p)]$, so,

   	$\text{Var}(\widehat{\varphi })$	$=$
   		$=$	${\displaystyle \frac{{\sigma }^2}{2n}}(2+{[{\mathrm{\Phi }}^{-1}(1-p)]}^2).$

In each case an approximate $(1-\alpha )$ confidence interval is obtained as

$$\widehat{\varphi }\pm z_{\alpha /2}\sqrt{\text{Var}(\widehat{\varphi }}).$$

Example 5.5:  Gamma Distribution, ctd.

$X_i\sim \text{Gamma}(\alpha ,\beta )$ and $\varphi =\alpha /\beta$, the population mean. Thus, $\widehat{\varphi }=\widehat{\alpha }/\widehat{\beta }$ and $▽g{(\overrightarrow{\theta })}^T=[1/\beta ,-\alpha /{\beta }^2]$. Hence,

$$\text{Var}(\widehat{\varphi })\approx [1/\beta ,-\alpha /{\beta }^2]{\overrightarrow{I}}_E{(\overrightarrow{\theta })}^{-1}\left[\begin{array}{c}\hfill 1/\beta \hfill \\ \hfill -\alpha /{\beta }^2\hfill \end{array}\right]$$

where

and

$$\mathrm{\Delta }={\left(\frac{n}{\beta }\right)}^2\left(\alpha {\gamma }^{\prime }(\alpha )-1\right).$$

Example 5.6:  Simple Linear Regression, ctd.

$X_i\sim N(\alpha +\beta z_i,{\sigma }^2)$ with (known) explanatory variables $(z_1,\mathrm{\dots },z_n)$ and $\sigma =1$ also known. We obtained,

$$\mathrm{\ell }(\overrightarrow{\theta })=-\frac{n}{2}\log (2\pi )-n\log \sigma -\frac{1}{2{\sigma }^2}\sum _{i=1}^n{(x_i-\alpha -\beta z_i)}^2.$$

Now, suppose we wish to obtain a confidence interval for $\beta$ based on the profile-likelihood function. We have, for fixed $\beta$ and $\sigma =1$,

$$\frac{\partial \mathrm{\ell }}{\partial \alpha }=\sum (x_i-\alpha -\beta z_i)=0$$

at the maximum, and so

$${\widehat{\alpha }}_{\beta }=\overline{x}-\beta \overline{z}.$$

Recall also that

$$\widehat{\alpha }=\overline{x}-\widehat{\beta }\overline{z}.$$

Hence, we obtain

$$D^{*}(\beta )=2\left\{-\frac{1}{2}\sum {(x_i-\widehat{\alpha }-\widehat{\beta }{z}_i)}^2+\frac{1}{2}\sum {(x_i-{\widehat{\alpha }}_{\beta }-\beta z_i)}^2\right\}.$$

For the data simulated in Figure 3.7, the corresponding profile deviance for $\beta$ is plotted in Figure Figure 5.2 (Link), leading to a 95% confidence interval for $\beta$ equal to $(0.10,0.21)$.

This latter example illustrates an important use of the likelihood function for model discrimination. Suppose we are interested in assessing whether there is a ‘significant’ linear relationship between the observations $x_i$ and the explanatory variables $z_i$.

If there were no such relationship then the true value of $\beta$ would be zero.

Hence, a test of the linear regression model, versus the simpler assumption that each of the responses have a common mean, is equivalent to the test that $\beta =0$.

Since 0 falls outside of our 95% confidence interval for $\beta$, this gives reasonably strong evidence that $\beta \ne 0$; we say that the hypothesis $\beta =0$ is rejected at the 5% significance level. But could we have reached this conclusion without having to go through the entire procedure of producing the profile deviance function?

In fact, yes. All that is necessary is to evaluate the profile likelihood at $\beta =0$ (equivalent to fixing $\beta =0$ and maximizing the log-likelihood with respect to the other parameters), and compare this value with the maximum likelihood value under the complete model where $\beta$ is unconstrained. Doubling the difference gives the deviance, whose value can be compared with the ${\chi }_1^2$ distribution to check for significance.