13.4 Summary

The score function is the first derivative of the log-likelihood. The observed information is MINUS the second derivative of the log-likelihood. It will always be positive when evaluated at the MLE.

DO NOT FORGET THE MINUS SIGN!

The likelihood function adjusts appropriately when more information becomes available. Observed information does what it says. Higher observed information leads to narrower confidence intervals. This is a good thing as narrower confidence intervals mean we are more sure about where the true value lies.

For a continuous parameter of interest, $\theta$, the calculation of the MLE and its confidence interval follows the steps:

1. 1

   Write down the likelihood, $L(\theta )$.

2. 2

   Write down the log-likelihood, $l(\theta )$.

3. 3

   Work out the score function, $S(\theta )=l^{\prime }(\theta )$.

4. 4

   Solve $S(\widehat{\theta })=0$ to get a candidate for the MLE, $\widehat{\theta }$.

5. 5

   Work out $l^{\prime \prime }(\theta )$. Check it is negative at the MLE candidate to verify it is a maximum.

6. 6

   Work out the observed information, $I_O(\widehat{\theta })=-l^{\prime \prime }(\theta )$.

7. 7

   Calculate the confidence interval for ${\theta }_{\text{true}}$:

   $$(\widehat{\theta }-\frac{1.96}{\sqrt{I_O(\widehat{\theta })}},\widehat{\theta }+\frac{1.96}{\sqrt{I_O(\widehat{\theta })}}).$$

Changing the data that your inference is based on will change the amount of information, and subsequent inference (e.g. confidence intervals).

A statistic $T(𝐱)$ is said to be sufficient for a parameter $\theta$, if $𝐱$ is independent of $\theta$ when conditioning on $S(𝐱)$.

An equivalent, and easier to demonstrate condition is that the likelihood can be factorised in the form $L(\theta )=g(𝐱)\times h(T(𝐱),\theta )$, iff $T(𝐱)$ is sufficient.