10 Reviewing the evidence

Why?

Growing interest in the development of measures to ensure health care (policy and delivery) is better informed by the results of relevant and reliable research (Evidence Based Medicine).

EBM

’the integration of individual expertise with the best available external evidence from systematic research’.

Cochrane

failure to combine evidence to address research questions: possibly imprecise; prone to selection bias.

How? Systematic Review/Overview

first step in the chain by which research evidence can inform policy and practice.

Quantitative methods - meta analytic

formal quantitative process for combining the evidence about a treatment effect.

Cochran collaboration

Planning: vital stage of the review process and the rules of the procedure should be formalised a priori in a study protocol (objectives, research questions, methods. General guidelines (Armitage et al. 2002)

1. 1.

   Data from the different trials should be kept separate so that treatment contrasts derived from individual trials are pooled rather than the original data. Pooling the original data reduces precision and may cause bias.

2. 2.

   Inclusion criteria should be clear - differences inevitable; treatments, patient characteristics, outcomes etc. should be ’similar’.

3. 3.

   Trials should be randomised trials with adequately ’blind’ assessment. Protocol departures should be handled in same way preferably ITT.

4. 4.

   All relevant data should be included: unpublished studies!

Different approaches $\mathrm{\to }$ different results.

Point estimates from different studies will almost certainly differ.

Homogeneous

results vary due to sampling error (random variation) as opposed to systematic differences in estimates: underlying true effect the same in each study.

Random variation can be handled using so called fixed effects methodology.

Heterogeneous

variability exceeds that expected due to sampling differences: ’real’ differences in estimated effects.

Note heterogeneity can occur when effects are in the same direction or different directions.

Forest plot

Examine heterogeneity and present results.

Plot of effect size $T_i,i=1,\mathrm{\dots },k$ and corresponding confidence interval for each study on a single axis.

The pooled estimate $\overline{T}$ and confidence interval is also displayed.

Size of the plotting symbol is often proportional to the reciprocal of the variance. More precise estimates $\to$ more influence.

Galbraith diagram

plot $z=T_i/\sqrt{v_i}$ (z-score) versus reciprocal of the standard error: ${(\sqrt{v_i})}^{-1}$.

Unnumbered Figure: Link

$H_0$: treatment effects the same in all $k$ primary studies: (${\theta }_1={\theta }_2=\mathrm{\cdots }={\theta }_k$) versus $H_1$ not all effect sizes are equal.

Test statistic:

$$Q=\sum _{i=1}^k{w}_i{(T_i-\overline{T})}^2$$

where $T_i$ is the estimated treatment effect in study $i$.

$$\overline{T}=\frac{{\sum }_i{w}_i{T}_i}{{\sum }_i{w}_i}$$

is the weighted estimator of the treatment effect and $w_i=\frac{1}{v_i}$ is the weight for study $i$ with $v_i$ denoting the variance of the estimate from study $i$.

Distribution of Q: approximately ${\chi }^2$ on $(k-1)$ degrees of freedom under $H_0$
Reject $H_0$ if Q is significantly large.
Limitations: lacks power, large sample sizes can result in rejecting null when difference small.

Weight

Each study estimate is given a weight inversely proportional to its variance: $w_i=1/v_i$.

Effect

For each of the $k$ studies let $T_i,i=1,\mathrm{\dots },k,$ denote the estimate of the treatment effect and $v_i$ the variance of the estimate (e.g. log odds ratio/difference in treatment group means.

Hypothesis

$H_0$ all effects equal versus $H_1$ not all equal.

Pooled estimate

$$\overline{T}=\frac{{\sum }_{i=1}^k{w}_i{T}_i}{{\sum }_{i=1}^k{w}_i}$$

where $w_i=1/v_i$.

NOTE Formula for $v_i$ depends on measure of effect (in notes previously)!

Variance of pooled estimate

$$\text{var}(\overline{T})=1/\sum _{i=1}^k{w}_i.$$

Approximate $\mathrm{(}\mathrm{1}\mathrm{-}\alpha \mathrm{)}$ confidence interval

$$\overline{T}\pm z_{\alpha /2}\sqrt{\text{var}(\overline{T})}$$

Example: Treatment of common cold; binary outcome variable.

Example: Effectiveness of Antibiotics for the Common Cold (Sutton et al, 2000, Sec. 4.3.1).

Endpoint: cure or general improvement within first 7 days?
Binary outcome: yes/no.
Data from five randomised clinical trials: antibiotics vs. placebo

Study	No. of patients		No. cure
	antibiotics	control	antibiotics	control
1	154	155	67	69
2	146	142	46	48
3	174	87	166	83
4	13	16	9	10
5	129	59	117	56

Parameter of interest is the log-odds-ratio.

Can represent study specific data in standard tabular form:

Maximum likelihood estimation:

$$T_i=\text{log}\left(\frac{a_i.d_i}{b_i.c_i}\right)$$

$$w_i=\frac{1}{\text{var}(T_i)}={\left(\frac{1}{a}+\frac{1}{b}+\frac{1}{c}+\frac{1}{d}\right)}^{-1}$$

Combined estimate of the log-odds-ratio:

$$\overline{T}=\frac{\sum T_i{w}_i}{\sum w_i}=\frac{-3.430}{41.257}=-0.083$$

Variance of pooled estimate:

$$\text{var}(\overline{T})=1/\sum _{i=1}^k{w}_i=1/41.257=0.024$$

Approximate $\mathrm{(}\mathrm{1}\mathrm{-}\alpha \mathrm{)}$ confidence interval for the log-odds ratio:

$$\overline{T}\pm z_{\alpha /2}\sqrt{\text{var}(\overline{T})}=(-0.387,0.222)$$

Antilog $\to$ (0.679, 1.248)

Mantel-Haenszel Method for Combining Odds Ratios (Sutton et al, 2000, Sec. 4.3.1).
History: method first described for use in stratified case-control studies.

tables

$2\times 2$ tables from $k$ studies; table from study $i$ with $n_i$ patients:

Treatment	Failure	Success
Experimental	$a_i$	$b_i$
Control	$c_i$	$d_i$

pooled estimate

$${\overline{T}}_{MH(OR)}=\frac{{\sum }_{i=1}^k{a}_i{d}_i/n_i}{{\sum }_{i=1}^k{b}_i{c}_i/n_i}$$

Example: Effectiveness of Antibiotics for the Common Cold (Sutton et al, 2000, Sec. 4.3.1).

Endpoint: cure or general improvement within first 7 days.
Data from five trials: antibiotics vs. placebo

pooled estimate of the OR: ${\overline{T}}_{MH(OR)}=0.95$

95% confidence interval (normal approximation): [0.70; 1.29]

Conclusion?

To avoid bias we need to ensure that relevant primary studies included.

Extensive literature searching (including grey matter) may not produce unbiased sample.

Funnel plot

Plot of sample size (or reciprocal of standard error) versus treatment effect.

Small studies

small true effect - small effect size - not significant - may not be published.

Large studies

small true effect - small effect size - significant- likely to be published.

Small studies

large effect size, more likely to be significant - more likely to be published.

Result: lack of ‘small effect size’ small studies in funnel plot $\to$ skewed. larger effects in smaller studies; smaller effects larger studies.

Unnumbered Figure: Link

Comments?

• •

  Altman DG (1991) Practical statistics for medical research. Chapman & Hall, London.

• •

  Bock J (1998) Bestimmung des Stichprobenumfangs. Oldenbourg Verlag, Muenchen.

• •

  Jones B, Kenward MG (1990) Design and analysis of cross-over trials. Chapman & Hall, London.

• •

  Piantadosi S (1997) Clinical trials: A methodologic perspective. Wiley, New York.

• •

  Senn S (1993) Cross-over trials in clinical research. Wiley, Chichester.

• •

  Senn S (1997) Statistical issues in drug development. Wiley, Chichester.

• •

  Sutton AJ et al (2000) Methods for meta-analysis in medical research. Wiley, Chichester.