• outcome and process assessment, health care
• epidemiology
• prevalence
• incidence

Kim et al recently published a ‘population-based nationwide cohort study’, a currently popular study type.1 It concludes that treatment with high-dose non-steroidal anti-inflammatory drugs (NSAIDs) is associated with a higher risk of cardiovascular disease (CVD) in patients with ankylosing spondylitis (AS), an interesting observation on first sight. Such a study uses large databases, often from insurance claims, as sources to investigate exposure-outcome relationships. Their trendiness is understandable given their extremely large sample sizes allowing the analysis of rare (safety) outcomes, their wide coverage of the target population and the current availability of computationally powerful but accessible statistical software.

However, data collected in claims databases lack the quality of those collected in rigorously designed hypothesis-driven observational studies. Both the selection of the population of interest and the determination of outcomes is done by applying International Classification of Disease 10th revision (ICD-10) codes recorded in the database. Exposure (here: NSAID usage and dosage) is ascertained through records of prescription billed to the insurance company. All codes and records are tied to financial incentives, namely reimbursement of costs associated with medical care. In other words, disease, exposure and outcome will only be picked up if their presence has resulted in an insurance claim.

The ICD-10 classification system is not up-to-date. It may, for instance, allow different terms for one disease and place different diseases under one umbrella. Defining the study population and the study outcome based on such diagnostic coding is sensitive to misclassification, since proper source check is impossible beyond piloting in samples. For instance, the study by Kim et al addresses patients with AS registered between 2010 and 2018. But in this same period the field gradually moved from AS to the broader concept of axial spondyloarthritis (axSpA). ICD-10 only includes AS (under different names) and does not recognise axSpA as a classifiable disease.

Exposure is assessed by the reasonable assumption that drugs prescribed are also taken, but we know this relationship is far from perfect. In claims databases, this uncertainty is compounded by the lack of direct information from prescribing physicians or their patients. Consequently, when the exposure of interest (here: high-dose vs low-dose NSAIDs) is inherently imprecise, and ICD-10 coding is used to select the population (patients with AS), and the outcome (CVD occurring over time), problems may increase exponentially and the study results are susceptible to insurmountable biases.

The most important bias of any observational study on the effects of drug treatment on outcomes is confounding by indication (CBI).2 In the study population, the choice for low-dose or high-dose NSAIDs was intentional, and not determined by randomisation or a chance occurrence: it was a joint decision of the prescribing physician and the patient with AS, weighing the substantial benefits of high-dose NSAIDs in active AS against their risks. Thus, it is highly unlikely that patients starting high-dose NSAIDs carried the same risk for CVD as patients starting a low dose. Clearly, risk factors for CVD determine the incidence of CVD and the choice for low-dose versus high-dose NSAIDs, confounding the assessment of the association between NSAID dosage and incident CVD. Similarly, AS inflammatory disease activity is known to be a risk factor for CVD,3 and is also a determinant for high-dose versus low-dose NSAIDs.

Often, researchers try to limit the effects of CBI by adjusting for known confounders through multivariable or propensity score techniques. Both aim to create artificial subgroups of patients that are prognostically similar, differing only in treatment (or another exposure of interest). Here, applying insurance databases outside of their original purpose (addressing insurance claims) may fall short. The truthfulness of the study result relies entirely on a proper selection of, and information on, the relevant confounding factors, but in an insurance database most of such information is missing, intangible, still unknown or simply unmeasurable. In the study by Kim et al, the unadjusted and adjusted risks were almost exactly similar. Since it is unlikely that the real background risk on CVD is totally similar in both NSAID dose groups, we must assume that the authors were unable to adjust for relevant confounding factors. In that case, residual, unmeasured confounding is the most likely explanation for the findings.

What were the findings? Kim et al reported a 10% excess cumulative risk of CVD for a ‘normal’ (recommended) daily NSAID dose (vs no NSAID use) over an average period of 4.7 years, with 95% CIs between 8% and 13%. The precision of the effect estimate is a feature of many big-data analyses with their large sample sizes. Assuming it is true, the real impact of such a risk is relatively minor. Based on the sparse data, we estimated a number needed to harm of >40 for one whole year of daily NSAID treatment, a reassuring figure in this context of relatively young patients who may experience large benefits from an optimal NSAID dose when they have active AS.

In sum, the title of a Shakespearean play comes to mind: ‘Much Ado About Nothing’.4 Certainly, modern big-data analyses may have a signalling function, for example, for rare or unknown side effects, but we should be mindful that finding a significant association is not sufficient to declare causality. There are examples of superb big-data analyses in the literature (see Fu et al5 for a recent example). Such studies start with the premise that their main result is subject to several epidemiological biases, and explain step-by-step and in a traceable manner how the authors have tried to circumvent confounding, for instance, by providing ample propensity matching diagnostics and justifying the choice of potential confounders beyond convenience. The best big-data studies interpret their main results with a sufficiently large degree of caution.