3 The exponential family 3.1 Moment and cumulant generating functions 3.3 Examples of linear EF distributions

3.2 One parameter linear exponential families

To begin, we examine the very simple case of an exponential family generated by a single canonical parameter.

Definition 3.2.1.

The one dimensional random variable $Y$ has an exponential family, EF, distribution, if its probability density function is of the form

\displaystyle f(y|\theta)=q(y)\exp\{\theta T(y)-\kappa(\theta)\}.

In the continuous case, the variable $y$ takes values in the real line $\{y:\ -\infty<y<\infty\}$ and $f$ is a density function, and in the discrete case the variable takes values in the integers $\{y:\ y=0,\pm 1,\pm 2,\dots\}$ and $f$ is a probability mass function. The parameter $\theta$ is the canonical parameter, $q$ is a pdf and $\kappa$ ensures that $f$ is a pdf. The function $\kappa$ varies with $\theta$ but is constant with respect to $y$ . The function $T$ varies with $y$ but is constant with respect to $\theta$ . In most cases considered in this course $T(y)=y$ so we will use this as the default parameterisation henceforth unless otherwise specified.

3.2.1 The generating distribution $q$

Consider a random variable $Y$ with a given fixed probability density or mass function $q(y)$ . The moment generating function, $M_{q}(\theta),$ of $q$ is

\displaystyle M_{q}(\theta)=\mathbb{E}_{q}[\exp\{\theta Y\}]=\int_{-\infty}^{% \infty}\exp\{\theta y\}\ q(y)dy,

where $\theta$ is a one dimensional real valued parameter. The integration is a summation when $Y$ is a discrete random variable. The domain of $\theta$ is those values for which the moment generating function is strictly finite, so that

\displaystyle\Theta=\{\theta:\ M_{q}(\theta)<\infty\}.

We refer to this domain as the canonical parameter space. This domain is not empty because it necessarily contains the point $\theta=0$ .

The cumulant generating function, here called $\kappa(\theta)$ , is the natural logarithm of the moment generating function

\displaystyle\kappa(\theta)=\log M_{q}(\theta).

Note that $\kappa(0)=0$ . We could denote $\kappa$ by $K_{q}$ .

3.2.2 Exponential tilting

We now generate the parametric family of density functions $f(y|\theta)$ by taking the pdf to be proportional to:

\displaystyle f(y|\theta)\propto q(y)\exp\{\theta y\}.

Requiring that the integral is unity leads to defining

\displaystyle f(y|\theta)=\frac{q(y)\exp\{\theta y\}}{M_{q}(\theta)}=q(y)\ % \exp\{\theta y-\kappa(\theta)\},

just the density in the definition above. It is clearly non-negative and integrates to 1. The family is the exponential family generated by the cgf of $Y$ under $q$ .

Because $q$ is a density function with a finite cumulant generating function all these generated functions are probability density functions. This construction gives the simplest instances of exponential families.

3.2 One parameter linear exponential families

Definition 3.2.1.

3.2.1 The generating distribution q

3.2.2 Exponential tilting

3.2.1 The generating distribution $q$