Article Text
PDF
Abstract
Objective To develop a disease activity index for patients with primary Sjögren's syndrome (SS): the European League Against Rheumatism (EULAR) Sjögren's syndrome disease activity index (ESSDAI).
Methods Thirty-nine SS experts participated in an international collaboration, promoted by EULAR, to develop the ESSDAI. Experts identified 12 organ-specific ‘domains’ contributing to disease activity. For each domain, features of disease activity were classified in three or four levels according to their severity. Data abstracted from 96 patients with systemic complications of primary SS were used to generate 702 realistic vignettes for which all possible systemic complications were represented. Using the 0–10 physician global assessment (PhGA) scale, each expert scored the disease activity of five patient profiles and 20 realistic vignettes. Multiple regression modelling, with PhGA used as the dependent variable, was used to estimate the weight of each domain.
Results All 12 domains were significantly associated with disease activity in the multivariate model, domain weights ranged from 1 to 6. The ESSDAI scores varied from 2 to 47 and were significantly correlated with PhGA for both real patient profiles and realistic vignettes (r=0.61 and r=0.58, respectively, p<0.001). Compared with 57 (59.4%) of the real patient profiles, 468 (66.7%) of the realistic vignettes were considered likely or very likely to be true.
Conclusions The ESSDAI is a clinical index designed to measure disease activity in patients with primary SS. Once validated, such a standardised evaluation of primary SS should facilitate clinical research and be helpful as an outcome measure in clinical trials.
Statistics from Altmetric.com
Primary Sjögren's syndrome (SS) is a systemic disorder characterised by lymphocytic infiltration and progressive destruction of exocrine glands. The inflammatory process can, however, affect any organ. As a result, clinical features can be divided into two facets: (1) benign but disabling manifestations such as dryness, pain and fatigue, affecting almost all patients; and (2) severe systemic manifestations that affect 20–40% of patients.
Evidence-based therapy for SS is largely limited to treatments that improve sicca features.1 Clinical trials of disease-modifying therapies have used a variety of ad hoc outcome measures mainly based on glandular features or patient symptoms, but not systemic features.2,–,6 Valid activity indices are needed7,–,9 to assess the effectiveness of new therapies, such as B-cell targeted therapies that have shown promising results for both severe systemic10 11 and glandular features.12,–,15 Two disease activity indices have recently been proposed: the SS disease activity index (SSDAI)16 and the Sjögren's systemic clinical activity index (SCAI).17 The development of these indices was based on exploratory studies conducted in single countries, but they serve as the basis of the present collaborative project. The European League Against Rheumatism (EULAR) has thus promoted an international collaboration to develop consensus disease activity indices. Two indices are currently in development: (1) a patient-administered questionnaire to assess patient symptoms, the EULAR Sjögren's syndrome patient reported index (ESSPRI); and (2) a systemic activity index to assess systemic complications, the EULAR Sjögren's syndrome disease activity index (ESSDAI).
We now describe the development and initial validation of the ESSDAI. This index was developed with the help of a worldwide panel of primary SS experts using physician global assessment (PhGA) scale of disease activity as an external criterion. The aim is for the ESSDAI to be used as outcome criteria to evaluate primary SS in a standardised way in both clinical trials and daily practice.
Methods
This paper results from a collaboration of experts identified through their involvement in the primary SS field, headed by a steering committee of seven physician experts in SS (HB, SJB, JEG, XM, ET, AT, CV), a clinical epidemiologist (PR) and a rheumatologist, fellow in clinical epidemiology (RS). The research protocol was endorsed by EULAR (project code CLI 010).
The steps of the development of the ESSDAI are summarised below; the entire methodology is available in appendix 1, available online only.
Selection of relevant domains and definition of items
Domains of organ-specific involvement relevant to assess disease activity were selected in these steps. For each domain, the different clinical manifestations were ranked by level of activity (ie, items). For selection of domains relevant to disease activity and a definition of items for each domain, steering committee members prepared a preliminary proposal on the basis of their clinical experience, literature review and previous work.16 17 The preliminary selection of domains and items were successively submitted to the expert panel. Experts had to rate the importance of each domain or suggest any additional domains or changes to proposed items. Intention-to-treat was used as a help for experts to define the different activity levels that ranged from no activity (requiring no treatment) to high activity (requiring high dose steroids or immunosuppressant). The experts' proposals were analysed, then discussed and voted on during a meeting.
Elaboration of clinical vignettes
In this step, realistic clinical vignettes were generated from real patient profiles.
Abstraction and standardisation of real patient profiles
Five members of the steering committee supplied 96 profiles of their patients with systemic complications of primary SS. Each profile had to contain sections on ‘history’ (demographic data and past medical history), ‘today’ (clinical symptoms and results of imaging examination) and ‘laboratory’ (biological features). Patient profiles included data from the baseline and two follow-up visits (3 and 6 months).
Abstraction of descriptions of items from real patient profiles
From patient profiles, 96 histories and 364 items, included in the ‘today’ and ‘laboratory’ sections, were extracted and standardised by the same investigator (RS). Descriptions of all ESSDAI items were obtained and entered in a database with their corresponding scoring (domain and activity level). Each item had a median of 8.5 (interquartile range 4–15) descriptions.
Generation of realistic clinical vignettes
Determination of construction rules
Data from primary SS patient cohorts of five members of the steering committee (SJB, XM, ET, AT, CV)16,–,20 were used to construct a sample of vignettes with characteristics similar to European patient cohorts.
Generation of clinical vignettes
In total, 720 clinical vignettes were generated by a combination of ‘history’ and items from the ‘today’ and ‘laboratory’ sections, with respect to the domain and item distribution defined previously. However, because items in the database referred to only systemic features, descriptors of symptoms such as dryness, pain and fatigue were generated and assigned to 30% of the patient vignettes.
Assessment
The 96 real patient profiles and the 720 clinical vignettes were randomly assigned to the 40 experts. Each expert had to rate five real patient profiles (rated by two raters each) and 20 clinical vignettes (18 were ‘unique’ and two were ‘common’ to two raters). For the survey, an internet-secure relational database was constructed. Patient data were presented chronologically, and the responses could not be changed. For all visits of each profile or vignette, experts had to assess disease activity by use of the PhGA on a 0–10 numerical scale and a five-point scale (inactive, low, moderate, high, very high activity). For the first visit of each profile or vignette, they also had to evaluate the plausibility of each patient case with the use of a five-point scale (very unlikely, unlikely, possible, likely, very likely) by answering the following question: ‘Please indicate, according to your clinical experience and knowledge of the disease, the likelihood that this patient scenario is a real case.’
Statistical methods
Determination of domain weights and construction of the ESSDAI
Realistic clinical vignettes were used to determine domain weights. Disease activity assessed by the PhGA was used as an external criterion. Bivariate analysis involved Pearson's correlation between PhGA and each domain separately; for each domain, scores ranged from 0 ‘no activity’ to 3 ‘high activity’. All domains were entered into multivariate models; the PhGA was used as a dependant variable and each domain was an explanatory variable. Two models were evaluated: a multiple linear regression model and a robust regression model with the least-median-of-squares method with a modified maximum likelihood estimator.21 22 The weights assigned to each domain were derived from the regression coefficients of the multivariate model and rounded to form simplified indices. The weight of each item was obtained by multiplying the weight of the domain by the level of activity.
Preliminary validation
The ESSDAI was then calculated for all real patient profiles and realistic clinical vignettes. Construct validity was assessed by the strength of correlation between the ESSDAI score and the PhGA.
Sensitivity analyses
To evaluate the stability/robustness of the domain weight estimation, other models were tested: a logistic regression model with the five-point scale used as an external criterion and different multiple linear regression models after pooling items that clustered.
Patient profile plausibility
Evaluation of patient profile plausibility of realistic clinical vignettes was compared with that of real patient profiles by a Cochran–Armitage trend test.
Reliability of disease activity scoring
The evaluation of clinical vignettes common to two raters was used to assess interrater reliability:
For the 0–10 PhGA: intraclass correlation coefficient (ICC) and Bland and Altman graphical analysis23 24
For the five-point scale: global agreement and kappa statistics25 26
The evaluation of real patient profiles was used to assess intrarater reliability by the ICC, if at the first follow-up visit, the physician considered the disease activity unchanged. ICC CI were estimated with bootstrapping methods, with 1000 replications.27
For all statistical analyses, a p value less than 0.05 was considered statistically significant. All statistical analyses involved the use of SAS release 9.1 and R release 2.2.1 statistical software packages.
Results
Characteristics of expert panel
Of 40 invited primary SS experts, 39 took part in the study (35 Europeans from 13 countries and four North Americans). The median age of experts was 49 years (interquartile range 46–58); 35 were rheumatologists, three were internists and one was an oral medicine practitioner. All but two (94.9%) had 10 or more years of experience in managing primary SS. All were involved in clinical research, and 23 (59.0%) were also involved in basic science research into primary SS.
Selection of domains and definition of items
All 10 domains (constitutional and lymphadenopathy, glandular, articular, cutaneous, pulmonary, renal, muscular, peripheral nervous system, central nervous system, haematological) proposed by the steering committee were included. Experts decided to divide the ‘constitutional and lymphadenopathy’ domain into two domains and to add a biological domain but not add a hepatic domain (considered to result from damage). The definition of the different activity levels (items) of each domain was obtained by consensus after discussion during meetings of the steering committee and experts.
Characteristics of real patient profiles and realistic vignettes
Thirty-nine of the 40 experts completed the rating of the 96 real patient profiles and 702 of the 720 clinical vignettes (table 1). Real patient profiles, selected for the extent of systemic involvement, had a significantly higher number of involved organs than did realistic clinical vignettes (2.83±1.46 vs 2.14±1.08; p<0.001).
Demographic characteristics for the 96 real patient profiles and 702 realistic clinical vignettes
Determination of domain weights and derivation of the ESSDAI
All domains, except haematological, glandular, articular and biological domains, showed a significant positive correlation with the PhGA score (table 2). All domains were entered in two multivariate regression models. Multiple linear and least-median-of-squares regression models provided similar results (R2=0.29 and R2=0.30, respectively). In both models, all domains were significantly associated with disease activity (PhGA), and the weight estimation was similar. The weights derived from the regression coefficients were rounded to obtain a simple index (tables 2 and 3).
Correlation between domains and disease activity, as assessed by the PhGA scale, and regression coefficients and domain weights obtained with the least-median-of-squares robust regression model with an MM estimator
The EULAR Sjögren's syndrome disease activity index (ESSDAI): domain and item definitions and weights
Preliminary validation of the ESSDAI in real patient profiles and realistic vignettes
The mean ESSDAI scores were 15.48±9.16 and 9.04±6.43 for real patient profiles and realistic vignettes, respectively. ESSDAI scores were significantly correlated with the PhGA score (r=0.58 for realistic vignettes and r=0.61 for real patient profiles, p<0.001; figure 1). The maximum theoretical ESSDAI score is 123; only 25% of realistic vignettes and the real patient profiles had a score of 13 or more and 21 or more, respectively. The highest score was 42 and 47 for the realistic vignettes and real profiles, respectively (figure 1).
Distribution of EULAR Sjögren's syndrome disease activity index (ESSDAI) scores and correlation with disease activity in real patient profiles and realistic vignettes. (A), (B) and (C) refer to the 702 realistic clinical vignettes, and (D), (E) and (F) refer to the 96 real patient profiles. Distribution of ESSDAI scores in (A) realistic vignettes and (D) real patient profiles; (B and E) ESSDAI score for each level of global activity as defined by physicians on the five-point scale; and (C and F) correlation between ESSDAI scores and physicians' ratings of disease activity by the physician global assessment (PhGA) scale (0–10 scale). For box plots of ESSDAI scores, the boxes represent the 25th and 75th percentiles; the lines within the box represent the median; the dot inside the box, linked by a line, represents the mean; and the whiskers extend to the most extreme data point, which is no more than 1.5 times the interquartile range (difference between the 75th and 25th percentiles) from the box. Values that are more extreme were considered outliers and are plotted individually (dots). EULAR, European League Against Rheumatism.
Sensitivity analyses
Other models for testing sensitivity analyses led to similar domain weights and similar correlation with the PhGA score.
Patient profile plausibility
Overall, experts considered 468 (66.7%) of the 702 vignettes likely or very likely to be true, compared with 57 (59.4%) of the 96 real patient profiles (p=0.09).
Reliability of disease activity scoring
Interrater reliability assessed by the ICC on 76 common vignettes was 0.41 (0.18–0.60) for the PhGA. Bland and Altman graphical analysis revealed no systematic errors (mean difference −0.16) but a variability of rating among experts (95% agreement interval; −4.92 to +4.61). The ratings of the same vignette by two different experts differed by 1 or less points for 37 vignettes (48.7%), by 2 to 3 points for 30 (39.5%) and by over 3 points for nine (11.8%). The weighted kappa statistic for disease activity rating by the five-point scale was 0.32 (0.18–0.47). When grouping the highest activity scores (high and very high activity) and the lowest scores (inactive, low and moderate activity), the observed agreement was 72.4% and the kappa coefficient was 0.42 (0.21–0.63).
Intrarater reliability of the PhGA for 20 real patient profiles with unchanged activity at the first follow-up visit as assessed with the ICC was 0.86 (0.68–0.94).
Discussion
The ESSDAI is a consensus clinical index designed to measure disease activity in patients with systemic complications of primary SS. This index is modelled on physicians' judgement of disease activity. It results from a large collaboration of European and North American experts in primary SS. Compared with the PhGA, the ESSDAI performed satisfactorily for an evaluation of disease activity in primary SS.
In the absence of an available ‘gold standard’ or true understanding of the disease process, the most accurate and meaningful method of disease activity assessment is to attempt to model the physicians' judgement. Any scale quantifying physicians' judgement of disease activity is a simplification of a complex mental process. For that purpose, two main gold standards have been used in the development of disease activity indices: (1) the PhGA28,–,30 and (2) the intention-to-treat approach.31 32 The PhGA was used for the development of the systemic lupus disease activity index (SLEDAI) in systemic lupus erythematosus (SLE),28,–,30 whereas the intention-to-treat approach was used for the development of the British Isles lupus assessment group (BILAG) for SLE31 and the DAS for rheumatoid arthritis.32 However, unlike rheumatoid arthritis that quasi-exclusively affects the articular system and in which therapeutic decisions are reproducible, the multisystemic nature of primary SS makes therapeutic decisions more variable. In addition, the evidence-based therapeutic management of SLE is currently more advanced than in primary SS. Moreover, in BILAG, this approach was used to define, in each domain, the different classes (A, B, C, D, E) and not as a gold standard to determine domain weights. Therefore, in the absence of effective treatment or consensual therapeutic management and because of the variability of physician habits, the intention-to-treat approach might be more difficult to apply as a gold standard for primary SS at this time.
In addition, the extent to which each organ involvement or patient symptoms of fatigue and pain can influence the physicians' evaluation of disease activity, in such a polymorphous disease, is extremely variable, as demonstrated by the limited reliability of the PhGA. These discrepancies among physicians' views, even among disease experts, justify the necessity for a more objective and standardised scoring system to homogenise the assessment of disease activity in different settings, by different physicians, experts but also less experienced physicians. Similar to the correlation of SCAI scores with the PhGA,17 that of ESSDAI scores with the PhGA was approximately 0.60. These correlations were lower than those from other studies evaluating DAS for various systemic disorders.16 29 33 However, in most of those studies, the experts involved were trained to the rating of the PhGA and the different activity tools to improve reliability and homogeneity of this scoring. In the present study, to be closer to usual practice, we decided not to perform a training exercise.
The ESSDAI was developed by a large panel of primary SS experts and attempted to reflect their thought process. This may have ensured the content validity of the ESSDAI, including all relevant determinants of disease activity. The validity of the ESSDAI was further confirmed by the significant association of all domains with disease activity in our model. Previous primary SS activity indices have been developed with the use of cohorts in which approximately half of the patients had inactive or weakly active disease.16 17 Our strategy was to use data from selected patients with systemic features to generate realistic clinical vignettes. This methodology enabled us to obtain a large number of vignettes (more than would have been possible with real patients) representing all possible systemic disease involvement (ie, items). We then evaluated the extent to which each item influenced the evaluation of disease activity, which had not been possible in previous studies that did not include all organ-specific features.16 17 As in all global scoring systems,16 29 similar ESSDAI scores (same disease activity) may reflect different domains involved.As a further component of this project we will also be evaluating the most common patient-reported symptoms, such as dryness, pain and fatigue in a patient-completed questionnaire, the ESSPRI.
A major challenge in designing a systemic index is distinguishing between damage and disease activity. The most frequent approach, to avoid scoring damage, is to consider manifestations as active only if ‘new’ or ‘worsening’. Under these scoring systems, when patients are evaluated at two time points, a persistent manifestation will not be rated at the second time point, which may cause an erroneous interpretation of improvement even though the patient's condition has not changed. To avoid this, all ESSDAI items were defined without reference to a previous assessment, but with an advice not to rate as active stable long-lasting features related to damage.
The ESSDAI is a systemic disease activity index developed to allow a standardised evaluation of disease activity in primary SS patients. Further studies are needed to assess the reliability and sensitivity to change of the ESSDAI. Once validated, if uniformly applied, the ESSDAI might enable comparison between studies and facilitate clinical research into primary SS. After the development of the patient-completed questionnaire (ESSPRI), the use of both the ESSDAI and ESSPRI for outcome assessment in randomised controlled trials should allow for assessing all facets of the disease.
EULAR Sjögren's Task Force Karsten Asmussen, Soren Jacobsen, Department of Rheumatology, University Hospital, Copenhagen, Denmark; Johannes WJ Bijlsma, Aike A Kruize, Department of Rheumatology and Clinical Immunology, University Medical Center, Utrecht, The Netherlands; Stefano Bombardieri, Rheumatology Unit, Department of Internal Medicine, University of Pisa, Pisa, Italy; Arthur Bookman, Division of Rheumatology, University of Toronto, Ontario, Canada; Hendrika Bootsma, Cees Kallenberg, Department of Rheumatology and Clinical Immunology, University Medical Center, Groningen, The Netherlands; Simon J Bowman, Rheumatology Department, University Hospital, Birmingham, UK; Johan G Brun, Roland Jonsson, Department of Rheumatology, Haukeland University Hospital, Bergen, Norway; Steven Carsons, Division of Rheumatology, Allergy and Immunology, Winthrop University Hospital, Mineola, USA; Salvatore De Vita, Clinic of Rheumatology, University Hospital, Udine, Italy; Nicoletta Del Papa, Department of Rheumatology, G Pini Hospital, Milano, Italy; Valerie Devauchelle, Alain Saraux, Rheumatology Department, la Cavale Blanche Teaching Hospital, Brest, France; Thomas Dörner, Rheumatology Department, Charité, University Hospital, Berlin, Germany; Roberto Gerli, Rheumatology Unit, Department of Clinical and Experimental Medicine, University of Perugia, Perugia, Italy; Jacques Eric Gottenberg, Jean Sibilia, Department of Rheumatology, University Hospital, Strasbourg, France; Eric Hachulla, Department of Internal Medicine, Claude Huriez Hospital, Lille, France; Gabor Illei, Sjögren's Syndrome Clinic, National Institute of Dental and Craniofacial Research, Bethesda, USA; David Isenberg, Centre for Rheumatology, University College, London, UK; Adrian Jones, Rheumatology Unit, City Hospital, Nottingham, UK; Menelaos Manoussakis, Athanasios Tzioufas, Department of Pathophysiology, School of Medicine, University of Athens, Athens, Greece; Xavier Mariette, Department of Rheumatology, Bicêtre Hospital, Le Kremlin Bicêtre, France; Carlomaurizio Montecucco, Department of Rheumatology, University of Pavia, Pavia, Italy; Roald Omdal, Department of Internal Medicine, University Hospital, Stavanger, Norway; Ann Parke, Clinical Immunology Unit, Division of Rheumatology, Saint Francis Hospital and Medical Center, Hartford, USA; Sonja Praprotnik, Matjia Tomsic, Department of Rheumatology, University Medical Centre, Ljubljana, Slovenia; Elizabeth Price, Department of Rheumatology, Great Western Hospital, Swindon, UK; Manel Ramos Casals, Laboratory of Autoimmune Diseases ‘Josep Font’, Hospital Clinic, Barcelona, Spain; Philippe Ravaud, Raphaèle Seror, Department of Epidemiology, Biostatistics and Clinical Research, Bichat Hospital, Paris, France; Josef Smolen, Division of Rheumatology, Department of Internal Medicine III, Medical University, Vienna, Austria; Serge Steinfeld, Department of Rheumatology, Erasme University Hospital, Brussels, Belgium; Nurhan Sutcliffe, Department of Rheumatology, Barts and The Royal London Hospital, London, UK; Elke Theander, Lennart Jacobsson, Department of Rheumatology, Malmö University Hospital, Lund University, Lund, Sweden; Guido Valesini, Rheumatology Unit, Department of Clinical and Experimental Medicine, Sapienza University, Rome, Italy; Claudio Vitali, Department of Internal Medicine and Section of Rheumatology, ‘Villamarina’ Hospital, Piombino, Italy; Frederick B Vivino, Department of Rheumatology Penn Presbyterian Medical Center, University of Pennsylvania, Philadelphia, USA.
Acknowledgments
The authors would like to thank Maxime Dougados and Alan Tyndall for their guidance and support. They thank the EULAR house in Zurich for their hospitality and outstanding organisation (Ernst Isler, Anja Schönbächler and their associates). The authors also thank the expert panel for their active and fruitful participation.
References
Supplementary materials
Web Only Data ard.2009.110619
Files in this Data Supplement:
Footnotes
-
Funding This project was supported by a grant from the European League Against Rheumatism (EULAR).
-
Competing interests None.
-
Ethics approval This study was conducted with the approval of the institutional review board of GHU Paris Nord (no IRB0006477).
-
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Miscellaneous