Results

The 144 trials included comprised a total of 16 659 patients. The median sample size was 60 patients; it increased over time (p=0.01) and was 110 in the most recent time period, 1988–1997.

Table 1 shows the usefulness of the reported outcomes for meta-analysis. Global evaluation was recorded in 127 trials. It was optimally reported in 52 trials (41%) and usefully reported in 88 trials (69%). There was a non-significant trend towards poorer reporting in the most recent years (p=0.32 for optimal reporting and p=0.15 for useful reporting).

Pain was recorded in 98 trials. It was optimally reported in 27 trials (28%) and usefully reported in 47 trials (48%). There was a significant trend towards better reporting over time (p=0.003 and p=0.09 respectively).

Number of tender joints, or closely similar measures such as Ritchie's joint count index, was recorded in 123 trials. It was optimally reported in 41 trials (33%) and usefully reported in 49 trials (40%). There was a significant trend towards better reporting over time (p=0.001 and p=0.007 respectively).

Grip strength was recorded in 124 trials. It was optimally reported in 34 trials (27%) and usefully reported in 41 trials (33%). There was a significant trend towards better reporting over time (p=0.03 and p=0.07 respectively).

Table 2 shows the mode of reporting in detail. The most common problem was the lack of standard deviations of the measurements. Confidence intervals, standard errors, exact p or t values, or other relevant test statistics that would have allowed the calculation of standard deviations, were also missing. Twenty out of 127 reports (16%) provided no data at all on global evaluation; the same applied for 21 out of 123 (18%) reports on pain.

Discussion

The reporting quality improved over time, but even in the most recent time period, and even if rather broad criteria were adopted, only about half of the data were reported in a way that made them potentially useful for a meta-analysis. This deficit in reporting was similar in small and large trials. For pain, for example, 48% of the trial reports were useful, and if the unit of analysis was patients, this proportion was very similar, 49%.

It was surprising that not a single confidence interval was presented in any of the 34 most recent trial reports, as all the reviewed trials aimed to compare the effect and tolerability of two similar drugs. Such trials are pragmatic, as the questions they address are directly relevant for clinical decision making. The 95% confidence interval allows the reader to judge whether a clinically relevant difference can be ruled out with reasonable certainty. Without confidence intervals, the non-statistically minded clinician may be misled into believing curious claims such as the new drug is “at least as effective”5 as the standard drug. As such claims are often accompanied by doubtful or biased statements about less toxicity,5 these trials seem primarily to have served marketing purposes.

Most of the trials were considerably underpowered, even if the sample size estimation is based on the patient's global evaluation of the effect, which is the most sensitive outcome.6 If, in accordance with the examples above, one wishes not to overlook the possibility that a new drug may be only half as effective as the standard drug, the necessary sample size is 290 patients for a power of 80%.6 ,7 As NSAIDs are effective drugs,6 a decrease in effect of 25% could also be clinically relevant to detect, in which case 1160 patients would be needed.

When sample sizes are too small, it is particularly important that the results are reported in a way that facilitates meta-analysis. Even though small trials tend to exaggerate the treatment effect,8 they can sometimes be useful for meta-analysis if the aggregated result is interpreted cautiously and the likely magnitude and direction of bias is taken into account. For example, it has been shown in meta-analyses of small trials with ordinal and interval scale outcomes that low dose prednisolone is highly effective in rheumatoid arthritis9 and, conversely, that chemical and physical measures to decrease exposure to house dust mites are ineffective in patients with asthma.10

For a symptomatic treatment such as an NSAID, the global evaluation of the effect is an important outcome because it allows the patient to express the various benefits and adverse effects of the drug in a simple measure. As this outcome is almost always recorded on a four or five point ranking scale, it was surprising that the data were reported in the original ordered categories in only 52 (41%) of the trials (table 2). In 26 trials (20%), the data were reduced to a scale with only two or three categories before reporting, which creates a risk of bias, as the decision to choose new categories and cut offs between them may not be independent of which drug will be favoured. It is also problematic for the validity of a meta-analysis that no data were shown for global evaluation in as many as 16% of the trials, as one would not expect such omissions to occur when the result favours the new drug over the standard.5

These problems could be minimised if research ethics committees ensured that raw trial data became publicly available after a period of time that allows the investigators to publish their results. By making unpublished data available, publication bias could also be avoided. As long as this has not happened, however, it remains an important challenge for meta-analysts to determine how to take stock of all the imperfectly reported results. For example, trials could be weighted by their sample size rather than by their inverse variance,11and missing standard deviations could be estimated from similar trials. It is not the purpose of this article, however, to explore these possibilities.

Previous surveys of statistical reporting of randomised trials seem not to have addressed the objective of this survey—that is, whether the reporting allows the data to be used for meta-analysis. It is a limitation of the survey that only one drug category was included. On the other hand, it is likely that the results can be generalised. Most clinical trials are drug trials supported by industry and many of the largest companies have developed and marketed NSAIDs. A similar reporting pattern would therefore be expected for many other trials sponsored by drug companies. Furthermore, the types of outcomes I have reviewed are very common; in fact, most trials in health care have a primary outcome that has been measured on an ordinal or an interval scale.4

It can therefore be assumed that a large body of trials of healthcare interventions have been reported inadequately in relation to meta-analysis. This deficit needs urgent improvement. It would also be helpful if investigators always specified a priori what constituted an important treatment effect and reported numbers of patients who improved. This would lead to a minor decrease in statistical power which seems acceptable, in particular as there is an important gain in clinical relevance. It is difficult to know whether an improvement of 1.5 cm on a 10 cm pain scale or some points on a composite score matters, whereas it is obviously relevant if a severe pain or depression has gone.