This section introduces the notion of information for a multivariate likelihood. We first recall some results from MATH230.
Consider a vector . Then Var is a matrix:
The diagonal of the variance matrix consists of the variance of the separate variables and the off-diagonal entries are covariances. The matrix is symmetric and positive definite.
The correlation between variables and is given by:
Let be a -dimensional random vector with and variance-covariance matrix , and let be a matrix, then
and
As in the one-parameter case, the approximation of log-likelihood or deviance surfaces in the vicinity of by quadratics — an approximation which improves with increasing sample sizes — forms the basis of an asymptotic distribution theory which can be used to obtain approximate confidence intervals.
The results are similar to those of the one-parameter case. Now, the score function is a vector function, , defined by
that is, the gradient vector for the log-likelihood. Thus at the MLE we have .
The information measures now become matrices:
and
These are, respectively, the hessian matrix and expected hessian matrix for the log-likelihood function. A general property of and (for any ) is that they are positive definite matrices, measuring respectively the observed curvature at and the expected curvature, at , of the log-likelihood surface.
For the one-parameter case, we had an asymptotic result that said the variance of the MLE was the reciprocal of the Fisher information. The corresponding result for the multi-parameter case is that the variance of the MLE vector is the matrix inverse of the Fisher Information matrix.
In the results that follow, let be a -dimensional unknown parameter and let its true value be .
Proposition: Asymptotic consistency of the MLE.
For regular estimation problems, in the
limit as , if is the true
parameter vector, then
Theorem 6: Multivariate asymptotic distribution of the MLE vector.
For regular estimation problems, in the
limit as , if is the true
parameter vector, then
Thus, the asymptotic distribution of is multivariate normal, with variance-covariance matrix given by the inverse of the expected information matrix.
For practical problems we use alternatives to which are asymptotically equivalent:
Theorem 7: Asymptotic distribution of the deviance with a -dimensional MLE vector.
For a regular estimation problem, the
deviance
in the limit as , and for .