2.3 Empirical probability

A serious limitation of classical probability is that it only applies in situations where the sample space is finite and all outcomes are equiprobable. While this might be useful for drawing cards, rolling dice, or pulling balls from urns, it offers no method for dealing with outcomes with unequal probabilities, or where the sample space may be infinite.

The frequency or empirical approach to probability is based on the idea that the underlying probability of an event can be measured by repeated trials. Supposing that $A$ is an event for some experiment, then if you repeat the experiment a number of times, $n$, we might hope that the proportion of trials in which $A$ occurs tends to stabilise as $n\to \mathrm{\infty }$. We would like to call this Prob($A$). More precisely

$\mathrm{Prob}(A)=\underset{n\to \mathrm{\infty }}{lim}{\displaystyle \frac{n_A}{n}},$

where $n_A$ is the number of times event $A$ occurs after $n$ trials.

What can we say about:

• •

  Prob($A$) in general?
  $n_A\ge 0$, so $\mathrm{Prob}(A)\ge 0.$

• •

  Prob($\mathrm{\Omega }$)?
  $n_{\mathrm{\Omega }}=n$, so Prob$(\mathrm{\Omega })=1$.

• •

  Prob($\mathrm{\varnothing }$)?
  $n_{\mathrm{\varnothing }}=0$, so Prob$(\mathrm{\varnothing })=0$.

Furthermore, if $A$ and $B$ are exclusive events, and $C=A\cup B$, set

• •

  $n_A$ to be the number of times $A$ occurs in the first $n$ trials

• •

  $n_B$ to be the number of times $B$ occurs in the first $n$ trials

• •

  $n_C$ to be the number of times $C$ occurs in the first $n$ trials

Then $n_C=n_A+n_B$ since $A$ and $B$ are exclusive. Therefore

$$\frac{c_n}{n}=\frac{a_n}{n}+\frac{b_n}{n}.$$

Taking the limit as $n\to \mathrm{\infty }$ we see that

$$\text{Prob}(C)=\text{Prob}(A)+\text{Prob}(B).$$

But since $C=A\cup B$, we have

$$\text{Prob}(A\cup B)=\text{Prob}(A)+\text{Prob}(B)$$

for the exclusive events $A$ and $B$.

However, how can we know if this thinking is valid? It seems intuitively reasonable, but we can’t be sure. In particular, it is impossible to conduct an infinite number of trials, and it is unclear how large $n$ must be to give a good approximation. More seriously, $\frac{n_A}{n}$ may not converge at all, or even if it does, if we repeat the experiment again, we may not necessarily obtain the same limit.

The modern theory of probability works the other way round: we assume that for each event $A$ there exists a number $\mathrm{P}(A)$ called the probability of $A$, and place axioms on the function $\mathrm{P}$. We will see that these axioms imply the convergence we hope to see.