4 Defining and Estimating Treatment Effects

When performing statistical analysis of clinical trial data we need to consider (amongst other things):

• •

  the outcome (endpoint) type and comparison of interest (e.g difference in treatment group means/group proportions; ratio of treatment group means)

• •

  distributional assumptions regarding the outcome measurements/parameters of interest (e.g Normal/Bernoulli/Poisson etc.);

• •

  the structure in the data imposed by the design (e.g paired designs, cross-over designs, randomised block designs etc);

• •

  Estimating versus testing issues (clinical significance versus relevance);

• •

  The analysis set (ITT/per protocol);

• •

  Analyses specified a priori (i.e. justified by design and specified in the protocol) and those which are exploratory.

Continuous measures on trial participants are typically summarised in term of group means/medians, for example, mean diastolic blood pressure/mean height by treatment group.

Clinically the outcome of interest is the difference, $D$, between the groups: the ‘treatment effect’.

Not only are we interested in the estimated magnitude of the difference, $\widehat{D}$, we are also interested in the degree of precision of the estimate as measured by its standard error, $\text{se}(\widehat{D})$, or a confidence interval.

A p-value quantifying the play of chance under the null may be interesting but it is the size of the difference which is of interest in terms of clinical relevance.

A study may continue if $\widehat{D}$ is sufficiently large even if it is not significantly different from zero.

Direct testing approach: two-sample t-test.

Let ${\mu }_T$ denote the treatment group mean and ${\mu }_C$ the control group mean.

Research Hypothesis

Test Statistic computed under $H_{\mathrm{0}}$

$$T=\frac{\widehat{D}={\overline{X}}_T-{\overline{X}}_C}{SE(\widehat{D})}.$$

Assuming a common underlying variance the corresponding standard error is estimated by pooling the data :

$$SE(\widehat{D})=S\sqrt{\frac{1}{n_T}+\frac{1}{n_C}}$$

with

$$S^2=\frac{1}{n-2}\sum _{i\in \{T,C\}}\sum _{j=1}^{n_i}{(X_{ij}-{\overline{X}}_i)}^2$$

where $n=n_T+n_C$ is the total numer of patients recruited.

Inference

The random variable $T$ is compared to the $t$-distribution with $n_T+n_C-2$ degrees of freedom and inference upon: $p=P(|T|\ge t_{n-2,1-\alpha /2}).$

Alternatively, let $S_T$ and $S_C$ represent the sample standard deviations for the and $n_T$ and $n_C$ represent the group numbers compute the pooled variance estimate:

and as previous the standard error is given by:

Assumptions: underpinning the t-test?

Non-parametric alternative $\to$ Mann-Whitney test.

An approximate test assuming non-constant variances can be performed (Welch’s) similarly with

and an adjustment to the degrees of freedom.

The assumptions underlying the two-sample t-test are:

• •

  the data are interval scale

• •

  the data are independent

• •

  the sampling distributions are normal

• •

  a common underlying variance.

Comments:

Exploratory analyses can be used to investigate the adequacy of the underlying assumptions.

Transformation can be performed to yield approximate normality, for example, a logarithmic transform for positively skewed data.

The t-test is robust to symmetric departures from Normality. If in doubt a non-paramteric test can be used.

An approximate test assuming non-constant variances can be performed (Welch’s) similarly with

and an adjustment to the degrees of freedom.

Using R to perform the t-test is straight forward using the following code.

Standard test,

> test1=t.test(y,x, var.equal=T)

Welch’s test,

> test2=t.test(y,x, var.equal=F)

Direct testing procedures (e.g T-tests) yield p-values but do not allow for assessment of effect sizes. In practice an estimate of the difference $D$ and corresponding confidence interval is preferable.

Parameter of interest: difference, $D$, of treatment group expectations.

Estimator: observed difference of group means

$$\widehat{D}={\overline{X}}_T-{\overline{X}}_C.$$

95% confidence interval:

$$\widehat{D}\pm t_{n-2,1-\alpha /2}S\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}\mathit{\hspace{1em}}$$

with

$$S^2=\frac{1}{n-2}\sum _{i\in \{T,C\}}\sum _{j=1}^{n_i}{(X_{ij}-{\overline{X}}_i)}^2.$$

• •

  inference: Does the confidence interval for the true difference, $D={\mu }_T-{\mu }_C$, span zero?

• •

  interval interpretation: Under repeated sampling we would expect $95\%$ of such intervals to contain the true parameter value.

• •

  intervals are more informative: can assess plausible range of effect size.

• •

  statistical significance versus clinical relevance: a significant p-value does not infer clinical relevance.

Good practice: CONSORT statement.
See: Exercises on Moodle.

Patient responses will often vary according to their respective values at baseline (entry to trial).

A proper baseline measurement is a measure of the end-point of interest taken prior to randomisation.

Two common approaches are to:

• •

  base estimation on the differences from baseline $y_i=y_{i1}-y_{i0},i=1,\mathrm{\dots },n$, comparing group mean change

• •

  analysis of covariance (ANCOVA) $\to$ regression model including baseline as a covariate.

Baseline measurements can increase precision!

ANCOVA is the preferred approach: method ‘adjusts’ the estimate for baseline imbalance and regression to the mean

• •

  example: Blood pressure after 12 weeks of treatment or difference from baseline

• •

  For a detailed discussion of the relative merits see Senn S, 1999

• •

  See Vickers, A. and Altman, D., Statitics notes: Analysing controlled trials with baseline and follow-up measurements. BMJ, 2001, 1123-112.

Significance testing of baselines is not appropriate.

For paired designs (for example, a patients left eye and right eye) a one sample t-test is performed by

1. 1.

   averaging the within-pair differences:

   $$\overline{D}=\frac{{\sum }_i^n{d}_i}{n}$$

   where $d_i=y_{i1}-y_{i2},i=,1\mathrm{\dots },n.$

2. 2.

   Computing the standard deviation of the differences $S_d$.

3. 3.

   Calculating the standard error of the mean of the differences $se(\overline{D})=\frac{S_d}{\sqrt{n}}$.

4. 4.

   Computing the test statistic:

   $$T=\frac{\overline{D}}{se(\overline{D})}.$$

   under $H_0:\overline{D}=0$ versus $H_1:\overline{D}=\mathrm{\Delta }\ne 0.$

5. 5.

   Comparing to a t-distribution on $n-1$ degrees of freedom.

Using R is again straight forward using:

> test3=t.test(y,mu=0)

where $y$ is the observed vector of within-pair differences $d_i$.

Assumptions for the paired design?

Note for later: a paired t-test is used to perfom a ‘simple analysis’ of cross-over trial data.

Responses may be dichotomous, for example, cancer free at five years/cancer recurred, died/survived, diseased/not disease.

Clinically interest then lies in comparing the treatment group proportions, $p_1,p_2$, say.

Consider the following general tabular representation of results

Let $p_1$ and $p_2$ denote the respective risks (success probabilities) for the treatment and control groups.

Risk Difference

the risk difference, $p_1-p_2$, is estimated:

$$RD=\frac{a}{a+c}-\frac{b}{b+d}.$$

Relative Risk

the relative risk, $p_1/p_2$, is estimated:

$$RR=\frac{a/(a+c)}{b/(b+d)}.$$

Odds Ratio

The disease odds, $p/(1-p)$, provides the ratio of success to failure. The odds ratio comparing groups is estimated:

$$OR=\frac{p_1}{1-p_1}÷\frac{p_2}{1-p_2}=\frac{ad}{bc}.$$

Aim: association between hay fever and eczema in 11 year old children.

Look at the table the other way around:

• •

  risk difference: $RD={\displaystyle \frac{141}{561}}-{\displaystyle \frac{928}{14453}}=0.189$

• •

  relative risk: $RR={\displaystyle \frac{141/561}{928/14453}}=3.91$

• •

  odds ratio: $OR={\displaystyle \frac{141\cdot 13525}{928\cdot 420}}=4.89$

$OR$ the same whichever way round one looks at table!

Intervals based upon the Normal approximation $1-\alpha$ CI: $\widehat{\theta }\pm z_{1-\alpha /2}SE(\widehat{\theta }).$

Risk Difference

$SE(RD)=\sqrt{{\displaystyle \frac{p_1(1-p_1)}{n_1}}+{\displaystyle \frac{p_2(1-p_2)}{n_2}}}$ with $p_1={\displaystyle \frac{a}{a+c}}$, $p_2={\displaystyle \frac{b}{b+d}}$, $n_1=a+c$ and $n_2=b+d.$

Relative Risk

$SE(\log RR)=\sqrt{{\displaystyle \frac{1}{a}}-{\displaystyle \frac{1}{a+c}}+{\displaystyle \frac{1}{b}}-{\displaystyle \frac{1}{b+d}}}$.

Odds Ratio

$SE(\log OR)=\sqrt{{\displaystyle \frac{1}{a}}+{\displaystyle \frac{1}{b}}+{\displaystyle \frac{1}{c}}+{\displaystyle \frac{1}{d}}}$.

Aim is to compare the odds of eczema amongst hayfever sufferers to non-sufferers.

Compute the odds ratio:

Compute the standard error of the log of the odds ratio and then compute a $95\%$ confidence interval for the log odds ratio and then back-transform:

95% CI: $\log (4.89)\pm 1.96\cdot 0.103$ $\to$ antilog: $[4.0;5.99]$ (asymmetric).

Further examples on exercise sheet.

Case-control studies: $RR$ should not be used! (subj. selection depends on outcome).

Logistic regression: adjustment for covariates.

Odds ratios do not contain information as to absolute risk difference.

If outcome rare, then $OR\approx RR$ ($a/c\approx a/(a+c)$).

Further reading: Bland JM, Altman DG (2000) BMJ 320: 1468.