2 Descriptive statistics for stationary time series.

2.1.1 Introduction. Many time series are evidently not IID - they may have long runs of values of the same sign, or show other patterns of association between successive values. Yet they often show a statistical similarity of appearance throughout their sample length. This may be after trends, seasonality and strong cyclical features have been removed by regression.

Differencing a time series, i.e. looking at the change from one sample point to another, may also lead to such an appearance, an example being inflation levels - monthly price increases - which are generally of more concern than the absolute price which continues its trend inexorably.

In this section we shall make simple assumptions about the structure of a single, or univariate time series so we shall now use the notation $x_t$ rather than $y_t$ which was more appropriate for the regression context.

The assumption that the statistical behaviour of a series is not changing as time progresses is called stationarity. It’s simplest description or measure uses the correlations between values in the series and leads to the following:

2.1.2 Definition. A time series $x_t$ is second order (weakly) stationary if

• •

  E$\left(x_t\right)={\mu }_x$ and  Var$\left(x_t\right)={\sigma }_x^2$ are the same for all $t$.

• •

  For each $k=0,1,2,\mathrm{\dots }$ Cov$(x_t,x_{t+k})={\gamma }_{x,k}$ are the same for all $t$.

Note that ${\gamma }_{x,0}={\sigma }_x^2$. If we also consider negative lags

$${\gamma }_{x,-k}=\text{Cov}(x_t,x_{t-k})=\text{Cov}(x_{t-k},x_{\left(t-k\right)+k})={\gamma }_{x,k}$$

The set of values ${\gamma }_{x,k}$ is called the autocovariance function (of the lag) for the series $x_t$.

Note: strict stationarity is that the joint pdf of any set of values of the series is the same as if they were all shifted in time by the same lag.

Also, a series is said to be Gaussian if this joint pdf is Normal. The assumption of second order stationarity and Gaussianity implies strict stationarity.

2.1.3 Definition. The autocorrelation function (acf) of a stationary time series $x_t$ is

$${\rho }_{x,k}=\text{corr}(x_t,x_{t+k})={\gamma }_{x,k}/{\gamma }_{x,0},k=0,1,2\mathrm{\dots }$$

Again is at times useful to think of ${\rho }_{x,k}$ being defined also for negative $k$ such that ${\rho }_{x,-k}={\rho }_{x,k}$. Obviously ${\rho }_{x,0}=1$ always.

2.1.4 Examples.
(a) If $x_t$ is white noise then by definition ${\rho }_{x,k}=0$ for $k\ne 0$

(b) Take $x_t$ to be white noise and consider the constructed series $y_t=x_t+x_{t-1}$. Then

$${\gamma }_{y,0}=\text{Var}\left(y_t\right)=\text{Var}\left(x_t\right)+\text{Var}\left(x_{t-1}\right)=2{\sigma }_x^2$$

$${\gamma }_{y,1}=\text{Cov}(x_t+x_{t-1},x_{t+1}+x_t)=$$

$$\text{Cov}(x_t,x_{t+1})+\text{Cov}(x_t,x_t)+\text{Cov}(x_{t-1},x_{t+1})+\text{Cov}(x_{t-1},x_t)=0+{\sigma }_x^2+0+0={\sigma }_x^2.$$

It is similarly found that ${\gamma }_{y,k}=0$ for $k>1$ so:

$${\rho }_{y,1}=\frac{{\sigma }_x^2}{2{\sigma }_x^2}=\frac{1}{2};{\rho }_{x,k}=0\text{ for }k>1.$$

Given a time series data set $x_1,x_2\mathrm{\dots }{x}_n$ the following estimates of the above quantities are used. Note however that they differ from the usual sample estimates used in statistics because they are not based on independent observations.

2.1.5 Time series sample estimates.

• •

  sample mean, an estimate of ${\mu }_x$: $\overline{x}=\frac{1}{n}{\sum }_{t=1}^n{x}_t.$

• •

  sample variance, an estimate of ${\sigma }_x^2$:  $s_x^2=\frac{1}{n}{\sum }_{t=1}^n{\left(x_t-\overline{x}\right)}^2.$

• •

  sample autocovariance, an estimate of ${\gamma }_{x,k}$:  $C_{x,k}=\frac{1}{n}{\sum }_{t=1}^{n-k}\left(x_t-\overline{x}\right)\left(x_{t+k}-\overline{x}\right)$  for $k=0,1\mathrm{\dots }n$ and 0 if $k\ge n$.

• •

  sample autocorrelation, an estimate of ${\rho }_{x,k}$ :  $r_{x,k}=\frac{C_{x,k}}{C_{x,0}}.$

Remark. The divisor of $n$ is used for defining $s_x^2$ and $C_{x,k}$. Some authors use the divisor $\left(n-k\right)$ when defining $C_{x,k}$.

Warning. These sample quantities can be automatically generated for any data set so the assumption of stationarity requires some check - usually a visual inspection of the series - before they are used.

In particular if the data contain trends, seasonality or cycles which appear deterministic and have not been removed by regression, then they will affect, and tend to dominate, the pattern of the sample acf, obscuring other statistical features. However, series may appear trend-like in a short sample yet stationary in the long term. The appearance depends on sample length. Figure 4 shows the sample autocorrelations of the residuals from the harmonic model fitted to the CO2 series. The figure also shows the series, the partial autocorrelations, and the sample spectral density (known as the periodogram), which we shall define later in this course. Figure 5 shows similar plots for the random series.

Consider $n$ successive time series values as a vector of random variables

where ${}^{\prime }$ indicates transpose. For stationary series, the autocorrelation function determines their $n\times n$ correlation matrix as
2.2.1

and the covariance matrix is defined by $V_n={\sigma }_x^2{R}_n$.

The matrices $R_n$ and $V_n$ have special structure: the elements are the same down any diagonal. Such matrices are called Toeplitz matrices.

Covariance matrices are useful for calculating the variance of a linear combination of variables such as $z={\alpha }_1{x}_{t+1}+\mathrm{\cdots }+{\alpha }_n{x}_{t+n}={\alpha }^{\prime }x$ where , as
2.2.2

A simple example is to evaluate Var$\left(x_1+x_2+\mathrm{\cdots }+x_n\right)$ by taking . The result is just ${\sigma }_x^2$ multiplied by the sum of the elements of $R_n$:
2.2.3

$$\text{Var}\left(x_1+x_2+\mathrm{\cdots }+x_n\right)={\sigma }_x^2\left\{n+2\left(n-1\right){\rho }_1+2\left(n-2\right){\rho }_2+\mathrm{\dots }+2{\rho }_{n-1}\right\}.$$

We can deduce

$$\text{Var}\left(\overline{x}\right)=\frac{1}{n^2}\text{Var}\left(x_1+x_2+\mathrm{\cdots }+x_n\right)=\frac{1}{n}{\sigma }_x^2\left\{1+2\left[\left(1-\frac{1}{n}\right){\rho }_1+\left(1-\frac{2}{n}\right){\rho }_2+\mathrm{\cdots }+\frac{1}{n}{\rho }_{n-1}\right]\right\}.$$

For large $n$ this may be approximated by
2.2.4

$$\text{Var}\left(\overline{x}\right)\approx \frac{1}{n}{\sigma }_x^2\left(1+2\sum _{j=1}^{\mathrm{\infty }}{\rho }_j\right).$$

A similar formula holds for the sample variance if the series is Gaussian, such that
2.2.5

$$\text{Var}\left(s_x^2\right)\approx \frac{1}{n}2{\sigma }_x^4\left(1+2\sum _{j=1}^{\mathrm{\infty }}{\rho }_j^2\right).$$

Remark. For highly autocorrelated series precise estimation of the mean and variance is very difficult because of the high magnitude of $2{\sum }_{j=1}^{\mathrm{\infty }}{\rho }_j$ and $2{\sum }_{j=1}^{\mathrm{\infty }}{\rho }_j^2$ appearing in the variances.

The autocorrelation measures the direct relationship between two values of a time series at different lags. In time series an alternative measure is the strength of association conditional upon the values in between.

2.3.1 Definition. The partial autocorrelation (pacf) at lag $k$ for a stationary time series is

$${\rho |}_1={\rho }_1\text{ and for }k\ge 2,{\rho |}_k=\text{corr}(x_t,x_{t+k}|x_{t+1},x_{t+2},\mathrm{\dots },x_{t+k-1}).$$

This may be calculated from the covariance matrix of $x_t,x_{t+1},\mathrm{\dots },x_{t+k}$ and so it depends only on ${\rho }_1,\mathrm{\dots },{\rho }_k$. The same quantities are in fact used when the Gaussian assumption is not appropriate, and we shall see later their value and interpretation in the context of model selection.

2.3.2 The sample partial autocorrelations.
Sample values ${r|}_k$ of the pacf ${\rho |}_k$ are calculated simply by using the sample values $r_k$ in place of the acf ${\rho }_k$ in the calculations.

A similar warning to that given regarding sample autocorrelations also applies here, that any strong trends and other deterministic components not removed from the series will tend to dominate the appearance of the sample pacf.