Math330 Exercises Week 3

• Let $X_1,\mathrm{\dots },X_n$ be independent with $X_i\sim N(\alpha +\beta z_i,1)$ for known constants $\{z_i\}$.

  Calculate the log-likelihood $\mathrm{\ell }(\alpha ,\beta )$.

  Hence calculate the MLE, $(\widehat{\alpha },\widehat{\beta })$.

  Calculate also the Fisher’s information matrix $I_E(\alpha ,\beta )$.

  Give a condition on $\{z_i\}$ which leads to the orthogonality of $\alpha$ and $\beta$.

• Suppose $X_1\mathrm{\dots }{X}_n$ are independent identically distributed random variables from pdf:

  where $\gamma >0$.

  For this model

  $$𝔼[X_i|\alpha ,\gamma ]=\alpha +\gamma$$

  where $1\le i\le n$, and

  $$𝔼[\underset{1\le i\le n}{\min }{X}_i]=\alpha +\frac{\gamma }{n}.$$

  Suppose now we have data from this model, $x_1,\mathrm{\dots },x_n$. Show that the likelihood function can be written

  By first maximising in $\alpha$ or otherwise, find expressions for $(\widehat{\alpha },\widehat{\gamma })$ as functions of $x_1,\mathrm{\dots },x_n$.

  Is $\widehat{\alpha }$ unbiased for $\alpha$? Is $\widehat{\gamma }$ unbiased for $\gamma$? Briefly justify your answers.

• This question will be completed as the quiz on Moodle.

  Figure 1 (First Link, Second Link) shows a simulated dataset from the Gamma$(\alpha ,\beta )$ distribution, with $\alpha =5$, $\beta =0.5$. Four points are labelled ‘1’ to ‘4’ on the contour plot; corresponding density functions ‘a’ to ‘d’ are superimposed onto the histogram. This week’s Moodle quiz will ask you to match the density functions with the points ‘1’ to ‘4’.

  Figure 1: First Link, Second Link, Caption: Simulated dataset for Gamma$(\alpha ,\beta )$ distribution, with $\alpha =5$, $\beta =0.5$.

• (Invariance principle) Let $\mathrm{\Theta }\subset {ℝ}^d$ be some parameter space and $g:\mathrm{\Theta }\to {ℝ}^d$ be a one-to-one function ($g(\theta )\ne g({\theta }^{\prime })$ if $\theta \ne {\theta }^{\prime }$). Also, let $f(\theta ;X)$ be a density function. We want to use $g$ to reparameterise our model class. What is a natural choice for our new parameter space $\mathrm{\Xi }$? Also define the new likelihood function $\tilde{f}(\xi ;X)$ for all $\xi \in \mathrm{\Xi }$. Show that if $\widehat{\theta }$ achieves the highest likelihood, i.e. $f(\widehat{\theta };x)\ge f(\theta ;x)$ for all $\theta \in \mathrm{\Theta }$ and some sample $x$ then also $\tilde{f}(g(\widehat{\theta });x)\ge \tilde{f}(\xi ;x)$ for all $\xi \in \mathrm{\Xi }$. Finally, show that this also holds in the opposite direction, i.e. if $\tilde{f}(g(\widehat{\theta });x)\ge \tilde{f}(\xi ;x)$ for all $\xi \in \mathrm{\Xi }$ then also $f(\widehat{\theta };x)\ge f(\theta ;x)$ for all $\theta \in \mathrm{\Theta }$.

• In this exercise, you may use that an Exponential$(\theta )$ distribution, with $\theta >0$ has density function

  $$f(x|\theta )=\theta \exp (-\theta x)$$

  for $x>0$, and expected value $1/\theta$ and variance $1/{\theta }^2$. Let $x_1,\mathrm{\dots },x_n$ be observations assumed to be independently generated from

  $$X_i\sim \mathrm{Exponential}\left({\theta }_i\right)$$

  where $i=1,\mathrm{\dots },n$ and ${\theta }_i=\exp (\alpha +\beta z_i)$. Here $z_i$ is a known covariate for observation $i$ and $(\alpha ,\beta )$ are unknown parameters.

  • Show that expected information matrix is given by

    HINT: Recall that the $z_i$s are treated as known and the $X_i$s as random.
    (3 marks)

  • Use the expected information matrix to give a condition on the $z_i$ under which parameter orthogonality holds. Explain the relevance of parameter orthogonality for inference. (2 marks)
    

  • Find the asymptotic distribution of the maximum likelihood estimator $(\widehat{\alpha },\widehat{\beta })$, quoting the general result used. Note: You may use the following result

    (2 marks)

  • Find the standard error of $\widehat{\beta }$ in the following two cases: (1) when $\alpha$ is also unknown, and (2) when $\alpha$ is known. Under what circumstances are both standard errors identical? (Justify your answer.) (3 marks)