Extreme value theory: predicting the ultra rare

Extreme value theory is a really exciting — and kind of astonishing — area of statistics. This is because it can tell us about the probability of events happening that are so rare there is barely any data recorded on them.

This seems perverse. Very broadly, traditional statistics says that we may not able to make accurate predictions about what may happen on an individual level (for example, how tall one puppy may grow to be in adulthood). But, if we look at a large population (the development of large numbers of puppies) we can get an idea of the range that we expect the majority to be in.

With extreme value theory, we are not interested in the behaviour of the majority. We want to look at the likelihood of a very, very rare event happening. Such as a Dachshund puppy that grows to be bigger than a Doberman.

 

Why would we want to know that? Well, let’s say you own a VW Beetle and you want to buy a Dachshund puppy. Your family is fiercely attached to the car and will only agree to getting a puppy if it means you will not need to sell the car.

You are pretty sure this will not happen. You promise them this could never happen. But then you start to worry: could the puppy grow to be too big to fit in the car? You’ve never heard of — or seen — a Dachshund that’s too big for a Beetle. But does that mean you can be certain?

The trouble with extreme events — from a statistical point of view — is that they do not happen very often, if at all. We might want to know the probability of a once in 1000 years type event. We do not have a large body of data that can give us steer on what and when these events might occur.

So, are we stuck?

No! Thanks to extreme value theory.  

Statisticians can focus on the tails of the data — meaning they can examine the events that have a very low probability of occurring. They usually do this in one of two ways in the univariate setting.*

We can look at maxima over a certain period of time. For example, we could group Dachshunds according to the year they were born, then record the tallest in each year group. Surprisingly (to me) these tend to a distribution (the Generalised Extreme Value distribution).

This is Very Good News in statistics. It means we have mathematically backed insight into the way the population of maxima behaves.

What if we had two Dachshunds born in 2015 that grew very big? If we were looking for maxima we would only count the largest one, so we would be cutting out a potentially useful bit of data. A method that gets around this issue is to look at exceedances — data points that come above a certain threshold.

 

If we decide that any Dachshund taller than, say, 40cm is remarkable then we can look at the distribution of Dachshunds that exceed that level. This would give us data that accords to a Generalised Pareto Distribution.

One of the big academic issues here is choosing that threshold level: set it too high and you don’t get much data. Set it too low and you are out of the tails of the distribution. 

These theories have important applications — beyond prospective dog owners with families that love their car a little too much.

Flood defences are one area where governments need to know what a really, really bad flood would look like and how to protect people from it. But, because flood defences are expensive, they also don’t want to build ones that are bigger than necessary.

Finance is another area. How likely is an extreme financial or economic shock? What measures should be in place to ensure that institutions, and the financial system itself, can withstand it? Regulators would want to make sure they are not insisting on such strongly risk-averse measures that it is impossible for companies to make a profit.

*By univariate, I mean we are looking at just one variable. For example: height of dog, or observed temperatures or daily rainfall. We are not looking at several variables together (the multivariate setting).

Want to know more? There is a whole journal dedicated to Extreme Value Theory. Here’s a link to past volumes and issues.