Evaluating new surgical procedures

Needs collaboration between surgeons and trialists

Surgery has been slow to take up the challenge of British epidemiologist Archie Cochrane: to prevent the introduction of new therapeutic procedures until randomised trials have shown them to be more effective than existing treatments.1 For example, laparoscopic cholecystectomy was first performed in 1987 and became the standard treatment for symptomatic gall stones within about seven years. During this period no more than 10 trials comparing laparoscopic with conventional forms of cholecystectomy were published worldwide. Of three peer reviewed randomised trials comparing laparoscopic and minilaparotomy cholecystectomy published in Britain since 1992,2 3 4 only one randomised more than 100 patients,5 justified this with a calculation of sample size, and analysed the results by intention to treat.6

Many potential problems have been cited to explain the shortage of rigorous surgical trials.7 Some are practical--for example, recruiting patients may be difficult. This problem can be resolved by undertaking the trial in many centres. Another potential problem--differences in skills between these centres--can be overcome by stratified sampling.7 A third practical difficulty is that measuring appropriate outcomes may require years of follow up. This can be overcome if the centre coordinating the trial has long term funding,8 something that may be easier to achieve in Britain when the new arrangements for funding NHS research facilities are established.9

An often cited methodological problem is that patients, when asked for their consent, may refuse to be randomised because they have a definite preference for one procedure rather than another.7 There are at least three possible solutions to this problem. Firstly, the new procedure could be made available only in a randomised trial.10 Secondly, it could be part of a patient preference trial, in which patients with a preference for one type of treatment receive their preferred procedure and patients with no preference are randomised.11 Thirdly, the new procedure could be part of a randomised consent trial, in which only patients randomised to the new procedure are asked for consent.12

The fact that surgical trials cannot be double blind or placebo controlled is often seen as a major methodological problem. It is helpful to distinguish here between “explanatory” or “fastidious” trials (which seek to draw conclusions about defined scientific hypotheses) and “pragmatic” trials (which are designed to select the better of two procedures in clinical practice).13 Fastidious trials define both the alternative procedures and the eligible patients by a rigid protocol that equalises the placebo effect of each procedure; participating surgeons must adhere to the protocol, and patients who fail to comply are excluded from the analysis.14 In contrast, pragmatic trials define both procedures and patients by a flexible protocol that optimises the placebo effect of each procedure; participating surgeons may exercise clinical judgment, and patients who withdraw from experimental or control groups remain in that group for analysis by intention to treat.6

This distinction implies that, while fastidious trials may be helpful in developing a new surgical procedure, pragmatic trials are essential in evaluating its effectiveness in clinical practice. In particular, blinding both the surgeon and the patient to which procedure has been undertaken is not only impractical but also irrelevant since it is not part of the normal practice about which evidence is needed. Instead the real challenge is to ensure that the assessment of outcome is blind or at least unbiased. Use of a placebo or sham operation is equally impractical and irrelevant. Instead the real challenge is to predict the procedures between which commissioners and providers will need to choose.

Probably the most intractable of the methodological problems cited is the need to compare new surgical procedures with established ones.7 There are two potential solutions to this problem, which are best used in combination. Firstly, analysis should take account of how experienced each surgeon is in performing the new operation. Secondly, and more difficult to achieve, randomisation should begin as soon as is feasible; this enables the researchers to monitor the learning curve and thus evaluate the short term costs, both clinical and economic, of establishing the new procedure. Thereafter the trial should be continued well beyond the end of the learning curve; this enables the researchers to evaluate whether that procedure also brings long term benefits.

In short, the randomised trial is the design of choice for evaluating new surgical procedures. Some of the methodological problems can be overcome by using pragmatic trials, but will need close collaboration between surgeons and trialists.