Objective To develop and validate ClinESSDAI (Clinical European League Against Rheumatism Sjögren's Syndrome Disease Activity Index), ie, ESSDAI without the biological domain.
Patients and methods The 702 fictive vignettes derived from 96 real cases of primary Sjögren's syndrome of the ESSDAI development study were used. As for ESSDAI development, the physician assessment of disease activity (0–10 scale) was used as the ‘gold standard’ in a multivariate model for weighting domains, after removing the biological domain. The reliability, assessed by intraclass correlation coefficient (ICC) between ClinESSDAI and ESSDAI, explored if ClinESSDAI was equivalent to ESSDAI. Its psychometric (ie, measurement) properties were compared with that of ESSDAI in an independent cohort. Also, its use was evaluated on data of two clinical trials.
Results In multivariate modelling, all 11 domains remained significantly associated with disease activity, with slight modifications of some domain weights. Reliability between clinESSDAI and ESSDAI was excellent (ICC=0.98 and 0.99). Psychometric properties of clinESSDAI, disease activity levels and minimal clinically important improvement thresholds and its ability to detect change over time in clinical trials were very close to that of ESSDAI.
Conclusions ClinESSDAI appears valid and very close to the original ESSDAI. This score provides an accurate evaluation of disease activity independent of B-cell biomarkers. It could be used in various circumstances: (i) in biological/clinical studies to avoid data collinearity, (ii) in clinical trials, as secondary endpoint, to detect change independent of biological effect of the drug, (iii) in clinical practice to assess disease activity for visits where immunological tests have not been done.
- Disease Activity
- Sjøgren's Syndrome
- Outcomes research
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Primary Sjögren's syndrome (pSS) is a systemic disorder primarily characterised by lymphocytic infiltration of exocrine glands, resulting in functional impairment of salivary and lachrymal glands. The inflammatory process however extends beyond the exocrine glands and can potentially affect any organ. As a consequence, the disease spectrum extends from sicca syndrome alone to very diverse systemic extraglandular manifestations.
In the last few years, the European League Against Rheumatism (EULAR) has promoted an international collaborative project that has developed consensus disease activity indexes: the EULAR Sjögren's Syndrome Disease Activity Index (ESSDAI)1 to assess these extraglandular systemic manifestations, and the EULAR Sjögren's Syndrome Patient Reported Index (ESSPRI)2 for patients’ symptoms.
The ESSDAI includes 12 domains: 11 referring to organ involvements and a biological domain (see online supplementary material). The ESSDAI provides thus a standardised instrument to evaluate homogeneously systemic involvement in clinical trials and daily practice. The ESSDAI has been validated and is reliable and sensitive to change.3 ,4 Also, disease activity levels have been defined.5
The biological domain of ESSDAI includes B-cell biomarkers, that is, elevated gamma-globulin or immunoglobulin G serum levels, low complement levels, the presence of cryoglobulinaemia and/or of a monoclonal gammopathy. This domain is the more frequently involved ESSDAI domain affecting one-third to two-thirds of the patients in recent cohorts.3 ,6 ,7 In the context of biological studies focusing on identification of new biomarkers, association with disease activity is a crucial question to be answered, since the purpose of these markers is to know if they are implicated in the disease process. This association is often assessed using the ESSDAI as reference disease activity measure.2 Nevertheless, these biomarkers, frequently implicated in B-cell activity, usually correlated with the biological domain of ESSDAI. The biological domain may therefore induce collinearity of data and might falsely induce or increase association between the biomarker and activity measure. Removing these B-cell biomarkers might allow assessment of the association between a new biomarker and the true clinical activity. The objective of the present study is to derive, from the original ESSDAI, a score without the biological domain, the ClinESSDAI, and to compare it with the original ESSDAI score.
Patients and methods
This work is a part of the EULAR Sjögren’s task force international project (project code CLI 010) conducted to develop and validate outcome measures for primary SS and supported by the EULAR. The present work relies on data of the original study conducted to develop the ESSDAI1 and of the EULAR cohort set-up to validate EULAR disease activity indexes.
Patients and vignettes
For the development of the ESSDAI score, 720 fictive clinical vignettes were generated from 96 real patient profiles of patients fulfilling American–European Consensus Group (AECG) criteria, as previously described.1 Vignettes were randomly assigned to the 40 experts, of whom 39 completed the survey. Thus, data of 702 vignettes were obtained. Each expert evaluated disease activity on a 0–10 numerical scale (Physician Global Assessment (PhGA)) of 5 real patient profiles and 20 clinical vignettes, through an internet-secure relational database. As for the ESSDAI, the development of the ClinESSDAI relied on the use of 702 clinical vignettes of patients with pSS, rated at least by one expert.1
Real patients’ profiles
As for the development of ESSDAI, the clinical vignettes of 96 real patients’ profiles of the development study were used as preliminary validation set.
EULAR validation cohort
Between May 2009 and July 2011 395 pSS real patients were included in the validation study of EULAR disease activity indexes.3 This international observational study was conducted in 14 countries with the approval of the institutional review board of GHU Paris Nord (n° IRB0006477). To be included, patients had to fulfil the AECG criteria.8 Patients were prospectively followed with a second visit at 6 months. No therapeutic intervention was planned.
Clinical trial data
The data of the 28 patients with pSS who participated in a prospective open-label trial with rituximab conducted at the University Medical Center of Groningen (trial registration number: METc2008.179) were reanalysed. All patients were aged 18 years and fulfilled the AECG criteria for pSS. Patients were enrolled as part of a long-term follow-up study of (re)treatment with rituximab and were treated with rituximab (1000 mg) infusions at days 1 and 15. Patients were evaluated at baseline and 16, 24, 36 and 60 weeks after rituximab treatment. (Data presented here are those of the first course of rituximab, ie, week w0 to w24.)9
The 30 patients with pSS included in the prospective BELISS open-label trial conducted in two European Centres, Paris, France and Udine, Italy (clinical trial registration numbers: NCT01160666 and NCT01008982), were also reanalysed.10 The patients fulfilled AECG criteria,8 were positive for anti-Sjögren's syndrome A (SSA) or SSB antibodies, and had at the time of inclusion at least one of the three following characteristics: systemic complications or persistent salivary gland enlargement, early disease (≤5 years from the first symptoms) and presence of at least one biomarker of B-cell activation (increase in IgG level or free light chains or β2-microglobulinemia, decrease in complement component 4 (C4) level, presence of cryoglobulinaemia or monoclonal component). The patients received belimumab, 10 mg/kg, at w0, w2 and w4 and then every 4 weeks to w24. Last evaluation of this first phase was performed 4 weeks after the last infusion (ie, at w28).
The ESSDAI includes 12 domains (ie, organ systems: cutaneous, respiratory, renal, articular, muscular, peripheral nervous system, central nervous system, haematological, glandular, constitutional, lymphadenopathy and biological). Each domain is divided in 3–4 levels depending on their degree of activity (table 1). The score of each domain is obtained by multiplying the level of activity by the domain weight. The final score, the sum of all domain scores, falls between 0 (no disease activity) and, theoretically, 123.
Development of the ClinESSDAI
As for the development of the original ESSDAI, the PhGA of disease activity was used as the ‘gold standard’. To derive the ClinESSDAI, the biological domain was removed and analyses were rerun to obtain the adjusted weights of the remaining domains. Thus, the clinESSDAI includes all ESSDAI domains except the biological domain. The weighting of domains used a robust regression model with the least-median-of-squares method with an MM estimator.11 ,12 The PhGA was used as a dependant variable and the explanatory variables included all ESSDAI domains except the biological domain. The weights assigned to each domain were derived from the regression coefficients of the multivariate model and were rounded to form simplified index.
Validation of the ClinESSDAI
First step: concurrent validation with ESSDAI
Since ClinESSDAI is dedicated to be used as a ‘surrogate’ of ESSDAI for studies that evaluate biomarkers, its validation relies on its similarity with ESSDAI. Therefore, our primary objective was to demonstrate that ClinESSDAI was ‘equivalent’ to ESSDAI. For that purpose, the ClinESSDAI was compared with the original ESSDAI in terms of reliability using the intraclass correlation coefficient (ICC)13 and Bland and Altman graphical analysis.13–15 ICC values vary from 0 (totally unreliable) to 1 (perfectly reproducible); an ICC ≥0.75 is considered excellent.14 ICC CIs were estimated with bootstrapping methods, with 1000 replications.16 These analyses were performed on real profiles and clinical vignettes from the EULAR development cohort and on real patients from the EULAR validation cohort.
Second step: psychometric properties of ClinESSDAI
The psychometric (ie, measurement) properties were determined on data from the EULAR validation cohort as follows:
The construct validity was assessed by measuring Spearman’s correlation coefficient between the ClinESSDAI, PhGA and ESSDAI.
The inter-rater reliability was assessed with the ICC13 and their 95% CIs, on a subsample of patients assessed independently on the same day by two physicians.
The sensitivity to change was assessed between the baseline and the 6-month visits, in each subgroup of patients considered as (i) improved, (ii) stable or (iii) worsened. Evaluation of change at 6 months by the physician rated on a 5-point Likert scale (much worse, worse, the same, better, and much better) was used as external anchor. Sensitivity to change was assessed with the standardised response mean (SRM), which is the mean change in score between two visits divided by the SD of the change in score.17 If indexes correctly detected changes, sensitivity-to-change scores should be (i) <0 for patients with improved condition, (ii) around 0 for patients with stable condition and (iii) >0 for patients with worsened condition. In case of improved or worsened disease activity, the larger the SRM, the greater the sensitivity to change of the instrument is. SRM values can be considered large (>0.8), moderate (0.5–0.8) or small (<0.5).18–20 An SRM closer to zero, when disease activity is unchanged, indicates that the assessment of stability is more accurate.
Third step: reanalysis of clinical trial data
To assess the impact of deleting the biological domain on evaluation of treatment effect, particularly if the effect of treatment was mainly or partly driven by its biological effect or by its clinical effect, we reanalysed the data of two trials of B-cell-targeted therapies, rituximab and belimumab, that are known to be associated with an effect on B-cell biomarkers.9 ,10 The change in scores and the sensitivity to change of ESSDAI and clinESSDAI were assessed in these trials, using the SRM.
Definition of activity levels and minimal clinically important improvement (MCII) of the ClinESSDAI
As for ESSDAI, theses analyses were performed in the population of the EULAR validation cohort.5
Definition of disease activity levels
This step involved the use of two distinct statistical methods:
A receiver operating characteristic (ROC) curve analysis. In this analysis physician’s evaluation of disease activity with a 4-point Likert scale (inactive, low, moderate or high) was used as an external standard to determine the activity levels. For each cut-off, we computed a separate ROC curve and calculated area under the curve and the sensitivity and specificity of each cut-off. For each cut-off, we selected the clinESSDAI value having the better Youden index.21 When two values have similar discriminating properties, the one being closer to the corresponding ESSDAI cut-off was chosen.
An anchoring method based on the minimal disease activity (MDA) definition (ie, definition identical to that used for rheumatoid arthritis).22 ,23 For this analysis, two groups were considered: those in MDA (MDA group) and those not (non-MDA group). For each group, the 75th percentile of the clinESSDAI values determined the threshold between low and moderate activity in the MDA group and the threshold between moderate and high activity in the non-MDA group.24
Definition of MCII
The MCII was estimated using an anchoring method based on the physician's assessment of evaluation of change in disease activity. MCII estimates were defined as the median value (95% CI) of the absolute change in clinESSDAI score in the population judged as ‘better’.25 ,26
All statistical analyses involved use of SAS release V.9.3 and R release V.3.1.1 statistical software packages.
Development of the ClinESSDAI
In the multivariate modelling, all 11 domains remained significantly associated with disease activity after exclusion of the biological domain. The weights of the ClinESSDAI domains, derived from the regression coefficients, were slightly different from those of the original ESSDAI domains (table 1 and see online supplementary table).
Validation of the ClinESSDAI
First step: Concurrent validation of ClinESSDAI with ESSDAI
When testing the reliability of ClinESSDAI, taking ESSDAI as reference, it was excellent. The ICC values (95% CI) between ESSDAI and ClinESSDAI were 0.98 (0.97 to 0.98) for clinical vignettes, 0.99 (0.98 to 0.99) in real profiles and 0.90 (0.87 to 0.92) in the EULAR cohort, respectively.
Bland and Altman graphical analyses revealed no systematic bias in disease activity evaluation (bias=0.60 (0.53; 0.67) in clinical vignettes and 0.51 95% CI (0.27 to 0.74) in real profiles) and narrow limits of agreement intervals (see online supplementary figure).
Second step: psychometric properties of ClinESSDAI
Correlation between ClinESSDAI and PhGA was comparable with that of ESSDAI (r=0.59). As expected, original ESSDAI and ClinESSDAI were also highly correlated (r=0.91). Regarding reliability and sensitivity to change, ClinESSDAI performed similar to ESSDAI (table 2).
Third step: reanalysis of clinical trial data
Both ESSDAI and clinESSDAI showed significant improvement at weeks 12 and 28 with belimumab (figure 1A) and at weeks 16 and 24 after rituximab (figure 1B). At each time point, sensitivity to change of ClinESSDAI was also large but tended to be numerically slightly lower than that of ESSDAI (table 3).
Definition of activity levels and MCII of the ClinESSDAI
Using both anchoring method and ROC curve analysis (table 4), the estimates of low disease activity were similar (clinESSDAI<5). As for ESSDAI, this threshold was higher at the baseline visit, particularly with the MDA method, due to the inclusion of patients with more active disease. For the estimates of high disease activity two thresholds had 11 and 14. The value of 14 was chosen being the most sensitive and being similar to that of ESSDAI (clinESSDAI≥14). Finally, these thresholds were identical to that of ESSDAI.
MCII estimates were obtained for the whole cohort (−3 (95% CI −5 to −2)) and in the population of patients having at least moderate activity (ESSDAI>5) at inclusion (−4 (95% CI −7 to −2)). The two thresholds were similar to that obtained for ESSDAI (table 4). Therefore, like for ESSDAI, an improvement of at least three points of clinESSDAI was considered the MCII.
In this study we developed the clinESSDAI, a disease activity index derived from the ESSDAI by exclusion of the biological domain. We demonstrated that the clinESSDAI is a comparable tool to the original ESSDAI for measuring systemic activity in patients with primary SS. Also, its psychometric properties were good, the score being highly reproducible and having a large sensitivity to change. In two clinical trials that evaluated B-cell-targeted therapies, clinESSDAI was compared with the original ESSDAI, and displayed a good sensitivity to change. Nevertheless, the sensitivity to change of the clinESSDAI was slightly lower than ESSDAI, confirming the biological effect (on B-cell biomarkers) of these drugs. Finally, we also determined disease activity levels definition and MCII using clinESSDAI and found thresholds similar to that found with the original ESSDAI.
With this new tool, in the context of studies focusing on identification of new biomarkers, the measure of their association with disease activity will now be correctly addressed. Compared with previous study that logically used the ESSDAI as reference disease activity measure,6 ,27–29 the forthcoming studies will avoid the pitfall of circular reasoning induced by collinearity of data, since the clinESSDAI will allow a measure of activity independent of B-cell biomarkers.
In systemic lupus erythematosus (SLE) the most frequent way to remove the biological effect is to use the systemic lupus erythematosus disease activity index (SLEDAI) without anti-DNA and complement items.30 However, this method is methodologically inaccurate. In contrast, we, here, developed a new index, not simply by removing the biological domain, but by rerunning initial analyses. The modifications, even small, of some domain weights confirmed that removing one domain might impact the weight of others and justify the process we used to develop this new tool.
When evaluating the effect of a drug, physicians are usually more interested in its clinical effect than in its biological effect. Thus, when assessing the response to treatment using disease activity scores that include biomarkers, one can wonder to what extent the observed decrease in disease activity is linked to the effect of the drug on biological markers rather than its true clinical effect. In SLE, the SLEDAI includes immunological parameters (low complement level and increased anti-DNA levels) that usually improved with treatment.31–33 Only improvement of these biological parameters might induce a 4-point decrease of the score, this 4-point decrease also being the MCII of SLEDAI25 and the threshold used for defining response to treatment in the recently developed SLE responder index (SRI).30 Therefore, in some cases the observed SLEDAI decrease might only be linked to improvement of biological data. For these reasons, the deletion of immunological data has sometimes been used for evaluation of the clinical effect of the drug, for example in belimumab trials in lupus with the use of the ‘modified’ SRI.30 ,34 In pSS, improvement of the biological domain of ESSDAI might induce only a maximum of two points decrease, which is below but close to its MCII (three points). Therefore, ESSDAI is a robust index that can be used as a primary end-point in clinical trials. Nevertheless, we defined here the MCII of the clinESSDAI and showed that the clinESSDAI was able to detect change, with a sensitivity to change close to that of ESSDAI. Thus, clinESSDAI could represent a complementary end-point in a randomised clinical trial (RCT).
Last, in the context of clinical practice, the frequent reassessment of immunological tests included in the biological domain is not always available. Nevertheless, being able to monitor disease activity when immunological tests have not been performed remains an important issue, and the clinESSDAI might be useful for this purpose. Likewise, in rheumatoid arthritis, DAS28 with the three clinical values excluding erythrocyte sedimentation rate or C-reactive protein, or clinical disease activity index, is frequently used for measuring disease activity in the setting of clinical practice, and for therapies that impact acute-phase reactants. Another point was the discussion of removing or not the haematological domain. It was decided by consensus to keep it since this domain did not reflect only biological activity but also a true systemic complication, which is lymphopenia or autoimmune cytopenia. Furthermore, the included parameters (blood cell counts) are usually routinely performed during each follow-up visit.
Declination of the clinESSDAI, that is, ESSDAI without biological domain, appears valid and very close to the original score. In the clinESSDAI, the weight of some domains slightly changed compared with ESSDAI. The psychometric properties of the ClinESSDAI, including reliability and sensitivity to change, were similar to those of ESSDAI. Disease activity levels and MCII thresholds of clinESSDAI were also similar to those of ESSDAI. The clinESSDAI provides an accurate evaluation of disease activity independent of B-cell activation biomarkers, allowing its use in biological studies avoiding data collinearity. Also, even if ESSDAI remains the gold standard, ClinESSDAI could be also considered as a valuable end-point in RCT to assess the true clinical effect of the treatment, particularly if the latter is known to have a biological effect. Finally, clinESSDAI may be useful in clinical practice to assess disease activity for visits where immunological tests have not been carried out.
This project was supported by a grant from the European League Against Rheumatism (EULAR). We thank Maxime Dougados, Alan Tyndall, Iain McInnes and Daniel Aletaha for their guidance and support. We thank the EULAR house in Zurich for their hospitality and outstanding organisation (Ernst Isler, Anja Schönbächler and their associates). We also thank physicians who helped for patient recruitment (John Hamburger and Andrea Richards, Birmingham Dental Hospital & School, Saaeha Rauz, Academic Unit of Ophthalmology, University of Birmingham, Birmingham UK; Emmanuel Chatelus, Strasbourg University Hospital, Strasbourg, France, Frederic Desmoulins, Bicêtre University Hospital, Le Kremlin Bicêtre, France, Petra Meiners, University Medical Center Groningen, Groningen, The Netherlands, Roberto Gerli, Perugia University Hospital, Perugia, Italy), who participated in the BELISS study (Sara Salvin and Miriam Isola, Udine University, Udine, Italy) and all members of the EULAR Sjögren's Task Force.
Handling editor Tore K Kvien
Collaborators Members of the EULAR Sjögren's Task Force who participated in this study: Elena Bartoloni and Roberto Gerli, Rheumatology Unit, Department of Clinical & Experimental Medicine, University of Perugia, Italy; Stefano Bombardieri, Rheumatology Unit, Department of Internal Medicine, University of Pisa, Italy; Hendrika Bootsma, Cees Kallenberg and Petra Meiners, Department of Rheumatology and Clinical Immunology, University Medical Center Groningen (UMCG), Groningen, Netherlands Simon J Bowman, Rheumatology Department, University Hospital Birmingham, Birmingham, UK, Johan G Brun, Department of Rheumatology, Haukeland University Hospital, Bergen, Norway; Roberto Caporali, Department of Rheumatology, University of Pavia, IRCCS S. Matteo Foundation, Pavia, Italy; Salvatore De Vita, Clinic of Rheumatology, University Hospital of Udine, University of Udine, Udine, Italy; Valerie Devauchelle and Alain Saraux, Rheumatology Department, la Cavale Blanche Teaching Hospital, Brest, France; Thomas Dörner, Rheumatology Department, Charité, University Hospital Berlin, Berlin, Germany; Anne-Laure Fauchais, Department of Rheumatology, University Hospital, Limoges, France; Jacques Eric Gottenberg, Department of Rheumatology, Strasbourg University Hospital, Strasbourg, France; Eric Hachulla, Department of Internal Medicine, Claude Huriez Hospital, Lille, France; Aike A Kruize, Departments of Rheumatology & Clinical Immunology, University Medical Center, Utrecht, The Netherlands; Thomas Mandl and Elke Theander, Department of Rheumatology, Malmö University Hospital, Lund University, Sweden; Xavier Mariette, Frederic Demoulins and Raphaèle Seror, Department of Rheumatology, Bicetre Hospital, Le Kremlin Bicêtre, France; Carlomaurizio Montecucco, Department of Rheumatology, University of Pavia, Pavia, Italy; Wan-Fai Ng, Musculoskeletal research group, University of Newcastle, Newcastle, UK; Sonja Praprotnik and Matija Tomsic, Department of Rheumatology, University Medical Centre, Ljubljana, Slovenia; Manel Ramos Casals, Laboratory of Autoimmune Diseases ‘Josep Font’, Hospital Clinic, Barcelona, Spain; Philippe Ravaud, Center of Clinical Epidemiology, Hopital Hotel Dieu, Paris, France; Hal Scofield and Kathy L. Sivils, Arthritis and clinical immunology, Oklahoma Medical research foundation, Oklahoma City, USA; Roser Solans Laqué, Department of Autoimmune systemic Diseases, Valld'Hebron University Hospital, Barcelona, Spain; Takayuki Sumida, Department of Internal Medicine, University of Tsukuba, Japan; Sumusu Nishiyama, Rheumatic Disease Center, Kurashiki Medical Center, Kurashiki, Japan; Athanasios Tzioufas, Department of Pathophysiology, School of Medicine, University of Athens, Greece, Guido Valesini, Rheumatology Unit, Department of Clinical & Experimental Medicine, Sapienza University of Rome, Rome, Italy; Valeria Valim, Division of Rheumatology, Department of Medicine, Federal University of Espírito Santo, Brazil; Claudio Vitali, Department of Internal Medicine and section of Rheumatology, ‘Villamarina’ Hospital, Piombino, Italy, Cristina Vollenweider, Department of Rheumatology, German Hospital, Buenos-Aires, Argentina.
Contributors Conception and design: RS, PR, SJB, XM, HB. Acquisition of data: all authors. Analysis and interpretation of data: RS, GB, PM, SJB, XM, PR. Drafting the article or revising it critically for important intellectual content: all authors. Final approval of the version published: all authors.
Funding This work is a part of a project conducted by the EULAR Sjögren's task force. This project has been endorsed and supported by EULAR (project code CLI 010). The Beliss study was sponsored by AP-HP (Assistance Publique-Hopitaux de Paris) and by the Azienda Ospedaliero Universitaria ‘Santa Maria della Misericordia’, Udine, Italy. It received an unrestricted research grant from Human Genome Science/GlaxoSmithKline. The Rituximab trial from Netherlands is an investigator-driven study and was financially supported by Roche, Woerden, The Netherlands, which supplied the study medication. There was no involvement of this funding source in study design, patient recruitment, data collection, analysis and interpretation and writing of the report.
Competing interests None declared.
Patient consent Obtained.
Ethics approval The Institutional Review Board of GHU Paris Nord (n° IRB0006477).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data are available on request.