The Likelihood Function

suppose we have some data $\overrightarrow{x}$, a realisation of some random variables $\overrightarrow{X}$ that we assume have some (joint) distribution or model $\overrightarrow{f}(\cdot |\theta )$ for the data $\overrightarrow{x}$. This is a fully generalised description: so far we are not assuming that the data are independent or identically distributed, and we may have a vector of parameters.

The fully general definition of the likelihood function is any function $L(\theta )$ such that

$$L(\theta )\propto \overrightarrow{f}(\overrightarrow{x}|\theta ),$$

viewed as a function of $\theta$. Importantly, this does not define a distribution for $\theta$, as $\theta$ is on the wrong side of the conditioning. It defines a distribution for the random variable $\overrightarrow{X}$ for each fixed value of $\theta$.

For much of this course, we will assume that $\overrightarrow{x}$ consists of $n$ independent and identically distributed (IID) realisations, i.e. $\overrightarrow{x}=(x_1,\mathrm{\dots },x_n)$ with each $x_i$ a realisation of the same random variable:

$$X_i\sim f(\cdot |\theta ),i=1,\mathrm{\dots },n.$$

In this special case, we can write the likelihood function as being proportional to the product of the densities of the observations:

$$L(\theta )\propto \overrightarrow{f}(\overrightarrow{x}|\theta )=\prod _{i=1}^{n}f(x_i|\theta ).$$

Note here $\overrightarrow{f}$ is being used to denote a joint density and $f$ a marginal density.

Recall that the proportionality in the definition allows us to discard any multiplicative constants that do not involve $\theta$. The set of possible $\theta$ values is $\mathrm{\Theta }$, with $\mathrm{\Theta }$ called the parameter space. If $\theta \notin \mathrm{\Theta }$ then $L(\theta )=0$.

It is often more useful to work with the log–likelihood function, defined by:

$$\mathrm{\ell }(\theta )=\log L(\theta )=\sum _{i=1}^n\log f(x_i|\theta ),$$

with proportionality constants of $L(\theta )$ translated into an additive constant.