Article Text
PDF
Abstract
Background/objectives The European League Against Rheumatism (EULAR)/American College of Rheumatology (ACR) 2019 classification criteria for systemic lupus erythematosus system showed high specificity, while attaining also high sensitivity. We hereby analysed the performance of the individual criteria items and their contribution to the overall performance of the criteria.
Methods We combined the EULAR/ACR derivation and validation cohorts for a total of 1197 systemic lupus erythematosus (SLE) and n=1074 non-SLE patients with a variety of conditions mimicking SLE, such as other autoimmune diseases, and calculated the sensitivity and specificity for antinuclear antibodies (ANA) and the 23 specific criteria items. We also tested performance omitting the EULAR/ACR criteria attribution rule, which defines that items are only counted if not more likely explained by a cause other than SLE.
Results Positive ANA, the new entry criterion, was 99.5% sensitive, but only 19.4% specific, against a non-SLE population that included other inflammatory rheumatic, infectious, malignant and metabolic diseases. The specific criteria items were highly variable in sensitivity (from 0.42% for delirium and 1.84% for psychosis to 75.6% for antibodies to double-stranded DNA), but their specificity was uniformly high, with low C3 or C4 (83.0%) and leucopenia <4.000/mm³ (83.8%) at the lowest end. Unexplained fever was 95.3% specific in this cohort. Applying the attribution rule improved specificity, particularly for joint involvement.
Conclusions Changing the position of the highly sensitive, non-specific ANA to an entry criterion and the attribution rule resulted in a specificity of >80% for all items, explaining the higher overall specificity of the criteria set.
- lupus erythematosus
- systemic
- antibodies
- antiphospholipid
- autoantibodies
- synovitis
Data availability statement
All data relevant to this study are included in this manuscript (including the supplementary information) or the previous two manuscripts on the EULAR/ACR classification criteria.
Statistics from Altmetric.com
Key messages
What is already known about this subject?
The European League Against Rheumatism (EULAR)/American College of Rheumatology (ACR) 2019 classification criteria for systemic lupus erythematosus (SLE) have higher sensitivity than the ACR criteria, but maintained high specificity.
Key messages
What does this study add?
In the combined EULAR/ACR derivation and validation cohorts, which contained large groups of patients with other connective tissue diseases in the non-SLE population, antinuclear antibodies (ANA) had high sensitivity, but only 19.4% specificity.
The most important factors for the increase in specificity of the EULAR/ACR criteria were the shifting of ANA to the position of an entry criterion, and the attribution rule, by which items count towards SLE only if there is no more likely alternative explanation.
Fever, the entirely new item in the EULAR/ACR criteria, had a sensitivity of 14.8% and a specificity of 95.3% for SLE.
How might this impact on clinical practice or future developments?
While the EULAR/ACR classification criteria should not be used as diagnostic criteria, the findings are applicable to the diagnostic process: ANA are a useful screening test, but not specific for SLE. Other symptoms should only be attributed to SLE if there is no more likely other diagnosis found. This of course applies to fever, which, if otherwise unexplained, is an argument for SLE.
The most relevant performance characteristics of any classification criteria are sensitivity and specificity of the whole set of criteria in a relevant validation cohort. Accordingly, for the European League Against Rheumatism (EULAR)/American College of Rheumatology (ACR) 2019 classification criteria for systemic lupus erythematosus (SLE), these data were reported in the joint publication1 2 together with comparative sensitivity and specificity values of the ACR 1997 and the Systemic Lupus International Collaborating Clinics (SLICC) 2012 criteria systems.
The EULAR/ACR classification criteria project comprised four phases.1–3 Phase I evaluated ANA as an entry criterion4 and generated criteria items in an expert Delphi exercise,5 an international early SLE cohort6 and an SLE patient survey.7 Phase II reduced the number of items in a nominal group technique exercise,8 evaluated the interdependency of items in a two-step association study9 and precisely defined the remaining items.10 Phase III grouped the items and derived an individual weight for each item, based on a multicriteria decision analysis.11 Phase IV relied on a large international cohort collected worldwide, and split into a derivation and a validation cohort.1 2 This cohort contains data from a total of 1197 SLE patients and 1074 non-SLE patients.
The EULAR/ACR 2019 criteria require a positive antinuclear antibody (ANA) test with a titre of ≥1:80 by HEp-2 immunofluorescence or equivalent at any time as an obligatory entry criterion. For ANA positive patients, criteria ordered in seven clinical and three immunological domains are applied according to specific weights. A total sum score of 10 or more allows for classification as SLE. Criteria items count only if there is no more likely explanation than SLE. As in previous criteria systems, at least one clinical item is required, and historically positive items count. With these features, the EULAR/ACR 2019 criteria achieved about as high a specificity as the 1997 updated ACR criteria12 and still as high a sensitivity as the 2012 SLICC criteria,13 with an improved combined specificity and sensitivity as compared with the previous criteria.1 2 These characteristics are maintained across sexes and ethnicities and in those with early disease.14
While the performance of the individual items was not reported, the information on their sensitivity and specificity may be highly instructive. Indeed, individual criterion performance characteristics of the ACR 198215 and SLICC 201213 criteria were considered in the EULAR/ACR criteria approach, particularly in its early phases.10 For example, the strategic decision to evaluate repositioning ANA as an entry criterion3 was influenced by these data. Here, we present the sensitivity and specificity analysis of the EULAR/ACR criteria individual items.
Patients and methods
Subjects
Data on the 1982 ACR patient database15 and 2012 SLICC derivation cohort13 were taken from the respective publications. For the analysis of the EULAR/ACR international patient cohort, the derivation and validation cohorts were combined. To this cohort, every centre contributed up to 100 SLE cases and 100 non-SLE cases with conditions mimicking SLE.1 2 SLE diagnosis was used as a gold standard. Diagnoses of non-SLE control subjects included other connective tissue diseases (62.6%), other rheumatic and musculoskeletal diseases (28.7%), cancer, membranous nephritis and chronic infections (online supplemental table 1). A standardised form containing all items of the EULAR/ACR, SLICC and ACR criteria sets was used to collect data on all cases and controls. All laboratory tests, including ANA and SLE-specific autoantibodies, were performed locally, according to the submitting centre’s laboratory standards. The diagnoses of SLE versus non-SLE were triple-confirmed by independent investigators. Where needed, this procedure involved several steps, including direct queries. Cases were only included if the submitting centre and all three external investigators agreed.1 2 Sixty-eight cases (2.9%) were excluded due to lack of consensus on the diagnosis. The cohorts were comprised the 1197 triple-confirmed SLE cases and 1074 triple-confirmed non-SLE cases.1 2
Supplemental material
Analyses
The sensitivity and specificity of the individual items were evaluated using the EULAR/ACR criteria attribution rule. In an additional analysis, sensitivities and specificities were evaluated without the EULAR/ACR criteria attribution rule. The attribution rule states that items shall only be counted if there were no more likely explanation than SLE. This rule would for example preclude counting arthritis in a patient with rheumatoid arthritis (RA) for SLE. Sensitivity was calculated as the proportion of SLE patients (truly) positive for a criterion in the whole cohort, specificity as the proportion of all non-SLE patients negative for a criterion. CIs were calculated using the bias-corrected and accelerated bootstrap method (BCa method) with B=2000 bootstrap samples.1 2
Results
Specificity of individual items
With the anticipated exception of the entry criterion of a positive ANA, which had a very high sensitivity of 99.5%, but a low specificity of only 19.4%, all criteria items had a specificity of at least 80% (table 1). In fact, only three items had a specificity lower than 90%, namely low complement C3 or C4 (83.0%), leucopenia (83.8%) and positive antiphospholipid antibodies (87.7%). The limited specificity appeared almost exclusively due to their prevalence in other conditions, such as primary antiphospholipid syndrome and primary Sjögren’s syndrome, which were included in the non-SLE cohorts (see online supplemental table 1). Of relevance, these items were also assigned relatively low weights of 2 or 3 (table 1). The 95% CIs are shown in online supplemental table 2.
Supplemental material
Relative item weight, number of SLE and non-SLE patients fulfilling and not fulfilling the individual EULAR/ACR criteria items and the resulting sensitivities and specificities
Sensitivity of individual items
As expected, given that the goal of the new criteria set was to maximise the specificity for SLE, the differences in sensitivity were greater than those for specificity. Even when not taking the entry criterion of positive ANA into account, sensitivity ranged from as high as 75.6% for antibodies to double-stranded DNA (dsDNA), 72.0% for joint involvement and 71.7% for low C3 or C4 to as low as 1.84% for psychosis and 0.42% for delirium. An acute cutaneous LE (ACLE) rash was seen in 42.8%, 10.9% had a subacute cutaneous LE (SCLE) and 8.4% a discoid rash, while 35.5% had non-scarring alopecia and 27.0% oral ulcers. One-third of the SLE patients had proteinuria, with class III or IV nephritis more than twice as common as class II or V nephritis (table 1). Pleural and/or pericardial effusions were found in 16.0%, and 14.8% had non-infectious fever.
Sensitivity plus specificity
Based on these data, despite the low sensitivity of some items, every single item has a combined sensitivity plus specificity of >1.0, ranging from 1.004 for delirium to 1.693 for specific antibodies to dsDNA.
Comparison with the ACR 1982 and SLICC 2012 data sets
We compared the sensitivity and specificity data with those of the ACR 1982 criteria15 and the SLICC 2012 criteria,13 in as much as these data have been reported. Fever, acute pericarditis, the separate categories of class III/IV and class II/V lupus nephritis and low C3 and C4 were items newly defined within the EULAR/ACR classification criteria project10 and could therefore not be compared. For some of the other variables (eg, delirium), data are reported only for the whole (neurological) criterion in the SLICC data set, and therefore likewise not available for comparison. As shown in table 2, most of the available values were in a similar range. However, there was a difference in the specificity of arthritis, which was reported as 37% for the ACR and 43.6% for the SLICC criteria systems, but 90.9% in the EULAR/ACR 2019 system, using the SLICC criteria definition. When the same data were analysed without the attribution rule, specificity fell to 57.6%, much closer to the range of the previous sets (table 2, right column).
Sensitivities and specificities of individual items in comparison with the data reported in the ACR 1982 classification criteria and SLICC 2012 classification criteria manuscripts
Role of correct attribution
We analysed the role of the attribution rule for all EULAR/ACR classification criteria items. As shown in table 3, which is ordered by the size of the change, arthritis was most affected by correctly attributing this symptom to the more likely explanation, which was most commonly RA. Other items that were less affected were fever, proteinuria, oral ulcers (eg, Behçet’s disease) and acute pericarditis. The remaining items were not affected. Had the attribution rule not been applied, the numbers of SLE patients for whom items would have been counted, that is, the sensitivity of the items, would only have relevantly increased for fever, and marginally for joint involvement (table 3). Neuropsychiatric symptoms, including delirium, seizures and psychosis, may also be due to an underlying disease process other than SLE.
Influence of correctly applying the EULAR/ACR criteria attribution rule on individual items, ordered by magnitude of the effect
Changes to low-specificity items in the EULAR/ACR classification criteria
In addition to the impact of the attribution rule for increasing the specificity of the joint involvement criterion and the strategic shift in position of ANA positivity to an entry criterion, effectively eliminating an impact of its low specificity (19.4%), the exclusion of lymphopenia by the vote of the external experts in the nominal group technique exercise also influenced specificity (table 4). Likewise, photosensitivity was not included, increasing the specificity of ACLE. Finally, for antiphospholipid antibodies, specificity remained at below 90%, but this item has received a low weight,11 carrying only two points.1 2 In contrast to the previous classification criteria sets, all items now reached >80% specificity and most items had a specificity of 90% or more (table 1).
Changes to items of (relatively) low specificity made in the course of the EULAR/ACR SLE classification criteria project
Discussion
Our evaluation of individual items comprising the EULAR/ACR 2019 classification criteria and previous iterations of SLE classification criteria provides important insights. First, individual classification criteria items in the EULAR/ACR 2019 criteria system have relative homogeneity in specificity that was unprecedented in previous sets. Limiting the criteria set to relatively high specificity items improves overall specificity. Second, these data shed light on the differences in performance between the different criteria sets. Third, the current data highlight the prevalence of various organ manifestations in three different classification criteria cohorts over four decades.
These data on the individual items revealed that the EULAR/ACR 2019 classification criteria approach has led to relatively uniform specificities, ranging between 83% and 100%. In fact, all but three items had a specificity of 90% or higher. This is an important insight as classification criteria are only as strong as their weakest component.16 Compared with the ACR15 and SLICC13 criteria systems, the most pronounced impact on specificity was found for ANA and joint involvement or arthritis. A positive ANA test result ranged in specificity between 19% and 49%. The particularly low specificity of positive ANA in the EULAR/ACR cohort is explained by the fact that almost two-thirds of the non-SLE patients with conditions mimicking SLE had other connective tissue diseases. Positive ANA was strategically repositioned to constitute an obligatory entry criterion at a titre of ≥1:80, after the high sensitivity of this test was reconfirmed.4
For joint involvement, the attribution rule of the EULAR/ACR criteria increased specificity. This attribution rule took the place of statements for each individual criterion regarding conditions that should not be scored. The ACR15 and SLICC criteria,13 for example, defined thrombocytopenia ‘in the absence of offending drugs’ and ‘in the absence of other known causes such as drugs, portal hypertension, and thrombotic thrombocytopenic purpura’, respectively, with such exclusions defined for various items. The EULAR/ACR 2019 attribution rule defines for all items that they should only be counted for SLE if there is no more likely alternative explanation.10 More likely alternative explanations also include other systemic autoimmune diseases. RA, usually anticyclic citrullinated peptide positive, rheumatoid factor positive and erosive, was one of the examples used for illustrating this rule during the criteria process. RA was also one of the diseases specifically addressed, for allowing the classification of RA/SLE overlap (Rhupus) situations.17 18 However, this attribution rule now increased the specificity from 58% to 91%.
SLE joint disease differs from RA in that frank palpable synovitis needs not be present. Indeed, in contrast to RA, ultrasonography provides a clear increase in sensitivity,19 and the SLICC criteria definition was superior to palpable synovitis.1 2 13 The new criteria set therefore employs the same definition as in the 2012 SLICC criteria system, and, from the EULAR/ACR review on, termed articular manifestations joint involvement instead of arthritis. Joint involvement was the most prevalent clinical item in all of the three sets. If anything, the percentage of patients with joint involvement declined slightly over time, from 86% in the ACR via 79% in the SLICC criteria to 72% in the EULAR/ACR patient cohort. With most SLE patients having joint involvement, SLE needs to be in the differential diagnosis of symmetrical polyarthritis. Still, RA, for one example, is at least 10 times more common, reiterating the point of attribution made above.
The attribution rule also influenced the specificity of other criteria items, namely fever, proteinuria, oral ulcers and acute pericarditis. Fever is one example, which must not be attributed to SLE when there is another, more likely explanation, such as infection. This will also affect sensitivity. Of 226 SLE patients with fever, only 177 (78%) had clear non-infectious fever best explained by SLE. In the non-SLE group, of 89 patients with fever, only 51 (57%) had no more likely other explanation, be it infections or other clearly febrile conditions, such as adult onset Still’s disease or familial Mediterranean fever (FMF). It is important to stress the cohort included a small number of patients with infection, which is not representative of the situation when patients are admitted with fever. Oral ulcers in 7 of 82 non-SLE patients were expectedly attributable to other diseases, such as Behçet’s; and serositis was more likely attributed to causes like FMF or tuberculosis in 3 of 14 non-SLE patients with acute pericarditis.
Finally, the highest weighted items also have very high specificity, while items with a specificity of less than 90% carry a maximum of 3 points (table 1). The introduction of low complement levels by the SLICC group13 was an important step. However, having either C3 or C4 low only showed a limited specificity of 83%, which should also be taken into consideration for diagnostic purposes. In contrast, low values for both C3 and C4 increased the specificity to 96%.
There are some limitations to consider when analysing changes in item sensitivity and specificity from this data set. These include different definitions, such as the definition of joint involvement, different cut-off values for haematology, and slightly different definitions for serositis and neuropsychiatric items. Temporal changes in pattern may still be of interest. The 1982 ACR criteria cohort15 included almost 50% of patients with lupus nephritis (by proteinuria); this percentage was lower and amounted to approximately one-third for both the SLICC13 and EULAR/ACR1 2 data sets. Oral ulcers stayed remarkably constant at 27% sensitivity in the ACR and EULAR/ACR criteria. While ACLE lesions were found in approximately half of the patients throughout the cohorts, discoid LE was found in less than 10% of the EULAR/ACR cohort patients, as compared with almost 20% in the ACR and SLICC cohorts. In the EULAR/ACR cohort, SCLE was slightly more prevalent than discoid LE. Some of these changes may represent earlier disease and earlier interventions over time,20 or differences in case selection between the cohorts. In the haematological domain, leucopenia stayed and thrombocytopenia remained at percentages of just below 50% and around 20, while autoimmune haemolytic anaemia had decreased to around 5% in the more recent cohorts. Serosal manifestations also decreased from half in the ACR 1982 to one in eight in the EULAR/ACR 2019 criteria cohort, and neuropsychiatric manifestations fell to 5%. Since items were counted if present at any time, differences in disease duration could also influence sensitivity.
Limitations to inferences made about the specificity of criteria items relate to the case-mix and frequency of control diagnoses. This cohort included a large number of systemic autoimmune rheumatic diseases including Sjögren’s syndrome, systemic sclerosis, RA and primary antiphospholipid antibody syndrome. This cohort also included controls with membranous nephritis, thyroiditis, autoimmune hepatitis, viral infections, tuberculosis and sarcoidosis. However, the numbers of these control subjects were smaller. The relatively high proportions of patients with connective tissue diseases and patients with RA have implications for the performance of several of the items. Also, autoantibody testing was performed locally, resulting in variability in test performance. While we believe that the array of conditions is reflective of the typical differential diagnoses of SLE at our institutions, in other clinical settings RA and fibromyalgia may be the two most common considerations. This could influence item specificity and thus limit the generalisability of the data. In general, however, attribution to SLE only if there is no more likely alternative explanation should help specificity under any conditions. The same should hold true for limiting the criteria to higher specificity tests. With regard to undifferentiated connective tissue disease, also a usual consideration, two studies reassuringly show that a subset of these patients not only fulfil EULAR/ACR classification criteria, but then also apparently have true SLE.21 22
In conclusion, data on the individual items comprising the new EULAR/ACR classification criteria for SLE demonstrate a wide range in sensitivity and a relatively uniform high level of specificity for all weighted items. Use of ANA as an obligatory entry criterion and the attribution rule contribute to the improved specificity.
Data availability statement
All data relevant to this study are included in this manuscript (including the supplementary information) or the previous two manuscripts on the EULAR/ACR classification criteria.
Ethics statements
Ethics approval
Ethics committee approval and subject consent were obtained as per local requirements.
Acknowledgments
The classification criteria project has been jointly funded by EULAR and the ACR. This research was supported in part by the Intramural Research Programme of the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health.
References
Footnotes
Handling editor David S Pisetsky
Twitter @none, @edvital
Deceased Ray Naden deceased on 16 August 2020
Contributors All authors were involved in the EULAR/ACR classification criteria exercises, contributed to the manuscript and critically read and approved the final version. This final version could unfortunately not be read and approved any more by RN, who died in early autumn of 2020, as referred to on the title page. He was intimately involved in the study and had been up to date with the manuscript until a few days before his death.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement statement SLE patient representatives were involved at every stage of the EULAR/ACR classification criteria project.
Provenance and peer review Not commissioned; externally peer reviewed.
Author note We mourn Prof Ray Naden, who passed away on August 16th 2020. The authors wish to dedicate this article to his memory as a friend and colleague, and to highlight his efforts for these and other criteria.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.