Home page for accesible maths 6 Expectation (II)6.3 Decomposition of the marginal variance 6.5 Key definitions and Relationships

Style control - access keys in brackets

Font (2 3) - + Letter spacing (4 5) - + Word spacing (6 7) - + Line spacing (8 9) - +

6.4 Moment generating functions

The moment generating function or mgf of a random variable $X$ is defined through

\displaystyle M_{X}(t)=\operatorname{\mathsf{E}}\left[{e^{tX}}\right]=\left\{% \begin{array}[]{ll}\sum_{i}e^{ti}p_{X}(i)&\quad\text{if }X\text{ is discrete % rv with pmf }p_{X}(x)\\ \int_{s}e^{ts}f_{X}(s)\,\mathrm{d}s&\quad\text{if }X\text{ is continuous rv % with pdf }f_{X}(s)\end{array}\right.

for all real values of $t$ for which the expectation exists.

Moment generating functions can be manipulated in many ways to reveal properties of the underlying probability distributions. They often help in mathematical proofs of probability theorems, and will be used for this purpose in Chapter 9.

Example 6.4.1.

Find the mgf of the random variable following the exponential distribution with parameter $\beta$ ; sketch the mgf when $\beta=4$ .

Solution. $X\sim\operatorname{\mathsf{Exp}}(\lambda)\Rightarrow f_{X}(x)=\beta e^{-\beta x}$ , for $x>0$ . Hence,

	$\displaystyle M_{X}(t)$	$\displaystyle={\color[rgb]{0.76,0.01,0}\int_{0}^{\infty}e^{tx}\beta e^{-\beta x% }\,\mathrm{d}x=\lambda\int_{0}^{\infty}e^{-x(\beta-t)}\,\mathrm{d}x}$
		$\displaystyle={\color[rgb]{0.76,0.01,0}\frac{\beta}{\beta-t}}$

for ${\color[rgb]{0.76,0.01,0}\beta>t}$ . Note that $M_{X}(t)$ is only defined for $\beta>t$ , since only in that case does the integral exist. Hence, for $\beta=4$ the mgf looks like:

Unnumbered Figure: Link

Quiz: Now consider a general rv: can the mgf be negative? No; it is the expectation of a non-negative quantity.

Theorem 6.4.1.

If mgf is defined in some neighbourhood of the origin, $|t|<t_{0}$ , the following properties are satisfied:

1.

The mgf determines uniquely the distribution of the rv $X$ . That is, if two rvs have the same mgf then they have the same cdf.
2.

If $Z=a+bX$ , for $a$ real and $b$ non-zero real number, $M_{Z}(t)=e^{at}M_{X}(bt)$ .
3.

Moments about the origin can be obtained by differentiating the mgf with respect to $t$ and then evaluating the mgf at zero, i.e.

$\displaystyle M_{X}(0)=\operatorname{\mathsf{E}}\left[{X^{0}}\right]=1;M^{% \prime}(0)=\operatorname{\mathsf{E}}\left[{X}\right];M^{\prime\prime}(0)=% \operatorname{\mathsf{E}}\left[{X^{2}}\right]\ldots$

Hence the name!
4.

Let $X, Y$ be independent rvs with mgf $M_{X}(t),M_{Y}(t)$ respectively. Then,

$\displaystyle M_{X+Y}(t)=M_{X}(t)M_{Y}(t).$

Proof.

1.

Proof uses ideas from complex analysis (see Math215).
2.

If $Z=a+bX$ , then

$\displaystyle M_{Z}(t)=M_{a+bX}(t)=\operatorname{\mathsf{E}}\left[{e^{(a+bX)t}% }\right]=e^{at}\operatorname{\mathsf{E}}\left[{e^{bXt}}\right]=e^{at}M_{X}(bt).$
3.
Since $M=\operatorname{\mathsf{E}}\left[{e^{tX}}\right]$ then $M^{\prime}(t)=\operatorname{\mathsf{E}}\left[{Xe^{tX}}\right]$ , $M^{\prime\prime}(t)=\operatorname{\mathsf{E}}\left[{X^{2}e^{tX}}\right]$ etc.; but $e^{0X}=1$ so
1. $M_{X}(0)=1$ ,
2. $M_{X}^{{}^{\prime}}(0)=E(X)$ ,
3. $M_{X}^{{}^{\prime\prime}}(0)=E(X^{2})$ ,
and so on.
4.

$\displaystyle M_{X+Y}(t)$ $\displaystyle=\operatorname{\mathsf{E}}\left[{e^{(X+Y)t}}\right]$

$\displaystyle=\operatorname{\mathsf{E}}\left[{e^{Xt}e^{Yt}}\right]$

$\displaystyle={\color[rgb]{0.76,0.01,0}\operatorname{\mathsf{E}}\left[{e^{Xt}}% \right]\operatorname{\mathsf{E}}\left[{e^{Yt}}\right]}$

by independence, so $M_{X+Y}(t)=M_{X}(t)M_{Y}(t)$ . ∎

From Part 4, by induction, if $X_{1},X_{2},\dots,X_{n}$ are independent random variables:

\displaystyle M_{X_{1}+X_{2}+\dots+X_{n}}(t)=M_{X_{1}}(t)M_{X_{2}}(t)\dots M_{% X_{n}}(t).

Example 6.4.2.

Using its mgf, find the expectation and the variance of the random variable following the exponential distribution with parameter $\lambda$ .

Solution. Consider the first two derivatives of the mgf:

$M^{{}^{\prime}}(t)={\color[rgb]{0.76,0.01,0}\frac{\beta}{(\beta-t)^{2}}}$ , $M^{{}^{\prime\prime}}(t)={\color[rgb]{0.76,0.01,0}\frac{2\beta}{(\beta-t)^{3}}}$ .

Hence, $\operatorname{\mathsf{E}}\left[{X}\right]=M^{{}^{\prime}}(0)={\color[rgb]{% 0.76,0.01,0}\frac{1}{\beta}}$ , $\operatorname{\mathsf{E}}\left[{X^{2}}\right]=M^{{}^{\prime\prime}}(0)={\color% [rgb]{0.76,0.01,0}\frac{2}{\beta^{2}}}$ and

${\operatorname{\mathsf{Var}}}\left[{X}\right]=\operatorname{\mathsf{E}}\left[{% X^{2}}\right]-\operatorname{\mathsf{E}}\left[{X}\right]^{2}={\color[rgb]{% 0.76,0.01,0}\frac{1}{\beta^{2}}.}$

The mgf of a Normal random variable We first consider $Z\sim N(0,1)$ . Then

	$\displaystyle M_{Z}(t)$	$\displaystyle=\operatorname{\mathsf{E}}\left[{e^{Zt}}\right]=\int_{-\infty}^{% \infty}e^{zt}\frac{1}{\sqrt{2\pi}}e^{-\frac{z^{2}}{2}}\,\mathrm{d}z$
		$\displaystyle={\color[rgb]{0.76,0.01,0}\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{% \infty}e^{-\frac{z^{2}-2zt}{2}}\,\mathrm{d}z}$
		$\displaystyle=e^{t^{2}/2}\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}e^{-\frac% {(z-t)^{2}}{2}}\,\mathrm{d}z$

by completing the squares. Hence $M_{Z}(t)={\color[rgb]{0.76,0.01,0}e^{t^{2}/2}}$ by unit integrability of the N(t,1) density.

So if $V=\mu+\sigma Z$ then by Property 2,

\displaystyle M_{v}(t)=e^{t\mu}M_{Z}(t\sigma)=e^{\mu t+\frac{1}{2}\sigma^{2}t^% {2}}.

For instance, if $Z\sim N(0,1)$ then

$M_{Z}(t)=e^{t^{2}/2}$ ,
$M^{\prime}_{Z}(t)=te^{t^{2}/2}$ ,
$M^{\prime\prime}_{Z}(t)=t^{2}e^{t^{2}/2}+e^{t^{2}/2}$ ,
$M^{\prime\prime\prime}_{Z}(t)=t^{3}e^{t^{2}/2}+3te^{t^{2}/2}$ ,
$M^{iv}_{Z}(t)=t^{4}e^{t^{2}/2}+6t^{2}e^{t^{2}/2}+3e^{t^{2}/2}$ .

In particular $M^{\prime\prime}_{Z}(0)=1,M^{iv}_{Z}(0)=3$ so $\operatorname{\mathsf{E}}\left[{Z^{2}}\right]=1$ and $\operatorname{\mathsf{E}}\left[{Z^{4}}\right]=3$ as mentioned in Chapter 3.

Unfortunately the mgf is not defined for some rvs.

Example 6.4.3.

Let $X\sim\operatorname{\mathsf{Cauchy}}$ , then

\displaystyle M_{x}(t)=\int_{-\infty}^{\infty}\frac{e^{tx}}{\pi(1+x^{2})}\,% \mathrm{d}x,

which is not defined as, if $t>0$ the integrand $\rightarrow\infty$ as $x\rightarrow\infty$ and if $t<0$ the integrand $\rightarrow\infty$ as $x\rightarrow-\infty$ .

	$\displaystyle M_{X+Y}(t)$	$\displaystyle=\operatorname{\mathsf{E}}\left[{e^{(X+Y)t}}\right]$
		$\displaystyle=\operatorname{\mathsf{E}}\left[{e^{Xt}e^{Yt}}\right]$
		$\displaystyle={\color[rgb]{0.76,0.01,0}\operatorname{\mathsf{E}}\left[{e^{Xt}}% \right]\operatorname{\mathsf{E}}\left[{e^{Yt}}\right]}$