Background Evidence synthesis is becoming a standard approach for establishing the comparative effectiveness of rheumatoid arthritis (RA) treatments in health technology assessments. Numerous indirect and mixed treatment comparisons (MTCs) of biologic therapies for RA have been published. These have used varying methodological approaches to study selection as well as model specification.^1,2

Objectives To explore the impact of study heterogeneity on MTC estimates via the performance of a sensitivity analysis of the results of an investigation of anti–tumor necrosis factor agents (anti-TNFs) for treating RA in methotrexate (MTX) nonresponders.³

Methods The authors’ published WinBugs code was modified to allow for additional analyses of the relative risk (RR) of ACR20 and ACR50 response as well as mean percent change in Health Assessment Questionnaire Disability Index (HAQ-DI) score. Analyses were performed in the original data set and in alternate data sets that excluded Asian studies, an outlying study based on observed HAQ-DI scores, or treatment arms using unlicensed doses. In each data set, models were fit using the authors’ specifications with or without adjustment for year of study publication, early escape in case of treatment nonresponse, or MTX dose. Summary statistics were generated for posterior parameter estimates, including medians, means, and 95% credible intervals (CrIs). History plots were examined to assess estimator convergence.

Results In our base case analyses, we were able to replicate the authors’ point estimates. The authors had reported 80% CrIs for their point estimates. The estimation of conventional 95% CrIs reduced the frequency with which anti-TNFs were deemed to differ significantly from one another. The exclusion of studies had a limited effect on RR estimates for ACR response. However, in analyses of the percent change in HAQ-DI score, excluding unlicensed doses and the outlier study impacted estimates for etanercept relative to placebo and other anti-TNFs. The convergence criteria for models that included a covariate were generally unsatisfactory. However, the magnitude of parameter estimates varied with covariate adjustment, which suggested that the base case results could have been biased due to confounding.

Conclusions Our results indicate that MTC estimates can be sensitive to study selection. We were unable to estimate reliably the effects of some key study differences due to collinearity between these features and the treatments investigated. The inability to control for potential sources of heterogeneity reflects inherent limitations in the evidence base and suggests that a cautious interpretation of the authors’ base case results is warranted. While including a broad body of evidence may be desirable for informed decision making, the resulting heterogeneity may reduce the validity of conclusions. Investigating and attempting to control for such heterogeneity is a necessary step for producing evidence syntheses that are both useful and reliable.

1. Nixon RM, et al. Stat Med. 2007;26:1237–54.

2. Bergman GJD, et al. Semin Arthritis Rheum. 2010;39:425–41.

3. Schmitz S, et al. Ann Rheum Dis. 2012;71:225–30.

Disclosure of Interest A. Benedict Employee of: United BioSource, under contract with Abbott, D. Vanness Employee of: United BioSource, under contract with Abbott, J. Shaw Shareholder of: Abbott, Employee of: Abbott, M. Cifaldi Shareholder of: Abbott, Employee of: Abbott