Objective Systemic sclerosis (SSc) is associated with a significant reduction in life expectancy. A simple prognostic model to predict 5-year survival in SSc was developed in 1999 in 280 patients, but it has not been validated in other patients. The predictions of a prognostic model are usually less accurate in other patients, especially from other centres or countries. A study was undertaken to validate the prognostic model to predict 5-year survival in SSc in other centres throughout Europe.
Methods A European multicentre cohort of patients with SSc diagnosed before 2002 was established. Patients with SSc according to the preliminary American College of Rheumatology classification criteria were eligible for the study when they were followed for at least 5 years or shorter if they died. The primary outcome was 5-year survival after diagnosis of SSc. The predefined prognostic model uses the following baseline variables: age, gender, presence of urine protein, erythrocyte sedimentation rate (ESR) and carbon monoxide diffusing capacity (DLCO).
Results Data were available for 1049 patients, 119 (11%) of whom died within 5 years after diagnosis. Of the patients, 85% were female, the mean (SD) age at diagnosis was 50 (14) years and 30% were classified as having diffuse cutaneous SSc. The prognostic model with age (OR 1.03), male gender (OR 1.93), urine protein (OR 2.29), elevated ESR (1.89) and low DLCO (OR 1.94) had an area under the receiver operating characteristic curve of 0.78. Death occurred in 12 (2.2%) of 509 patients with no risk factors, 45 (13%) of 349 patients with one risk factor, 55 (33%) of 168 patients with two risk factors and 7 (30%) of 23 patients with three risk factors.
Conclusion A simple prognostic model using three disease factors to predict 5-year survival at diagnosis in SSc showed reasonable performance upon validation in a European multicentre study.
Statistics from Altmetric.com
Systemic sclerosis (SSc) is associated with a significant reduction in life expectancy.1,–,4 The life-threatening complications of SSc are lung, heart and renal complications such as pulmonary fibrosis, pulmonary hypertension, myocardial dysfunction or arrhythmias and renal crisis.5 When patients with a high risk of SSc-related death can be identified already at diagnosis, this opens the opportunity for preventive measures including early and intensive treatment of SSc and early diagnosis of lung, heart or renal complications. For an individual patient, a useful prognostic model will produce the correct probability of death in the years after diagnosis, after entering patient and disease characteristics in the model. To be clinically useful, a prognostic model preferably includes measures that are routinely collected in daily practice or are otherwise easy to obtain. A simple prognostic model to predict 5-year survival has already been developed by Bryan et al in a British tertiary referral centre in 1999.6 After correction for age and sex, the model contained only three variables that were independently predictive of the subsequent 5-year mortality: a positive level of urine protein (more than a trace), an erythrocyte sedimentation rate (ESR) ≥25 mm/h and a lung carbon monoxide diffusing capacity (DLCO) <70%.6 Testing of prediction models in external data (external validation) is important because usually the predictions of a prognostic model are less accurate when the model is applied to other patients, especially from other centers or countries. The objective of this study was therefore to validate the prognostic model to predict 5-year survival in SSc developed by Bryan et al in other SSc centres throughout Europe.
For this prognostic study using clinical practice data, a European multicentre cohort of patients with SSc diagnosed before 2002 was established using clinic-owned databases. Patient data were included upon diagnosis and included observation time up to 5 years. Participating centres were invited through the network of the European League against Rheumatism – Scleroderma Trials and Research group (EUSTAR) and this study was sponsored and endorsed by EUSTAR.
Patients diagnosed with SSc according to the preliminary American College of Rheumatology (ACR) classification criteria were eligible for the study when they had been followed for at least 5 years or shorter if they died.7 To allow for a complete 5-year follow-up, all patients should have been diagnosed before 2002. Patient observation time started at the moment when the ACR criteria were fulfilled (baseline) and ended at 5 years of follow-up or at the date of death. Data collected at the baseline visit of this 5-year period were used for the prognostic modelling. Patients who dropped out before 5 years were not included by design because their 5-year survival status was unknown.
Outcome and predictors
The primary outcome of interest in the study was whether patients were still alive or dead 5 years after the diagnosis of SSc. The predictor variables of the model by Bryan et al were age, gender and three disease-specific variables (presence of urine protein, elevated ESR and low DLCO). The presence of urine protein was tested in a single urine sample using a labstick or quantitative determination according to local laboratory standards, elevated ESR was defined as ESR ≥25 mm/h and low DLCO was defined as <70% of predicted. Also, anti-centromere and anti-topoisomerase antibodies were tested according to local laboratory standards. SSc disease subsets were defined as limited cutaneous SSc (skin thickening present on the trunk in addition to the face, proximal and distal extremities) or diffuse cutaneous SSc (skin thickening restricted to sites distal to the elbow and knee but also involving the face and neck). The data for the predictor variables came from patient inquiry and clinical tests performed at the visit in which the diagnosis of SSc was made. The outcome data came from chart review and medical correspondence. The data were electronically transferred to the research centre using a common code book and data that were made anonymous.
Baseline variables were grouped according to 5-year survival status and analysed using the two-sample t test, the two-sample Wilcoxon test or the χ2 test, as appropriate. Missing values of baseline variables were replaced by single imputation using multiple regression modelling including a random component on the assumption that data were missing at random.8 Next, the five baseline variables used by the prognostic model (age, gender, presence of urine protein, elevated ESR and low DLCO) were analysed using univariate and multivariate logistic regression with survival status (alive=0, dead=1) as the dependent variable. Discrimination of the model was tested using the area under the receiver operating characteristic (ROC) curve. The presence of urine protein, elevated ESR and low DLCO are all equally weighted in the prognostic model of Bryan et al, while age and gender were not counted in the risk score. In an attempt to improve discrimination, the model was recalibrated by using the regression coefficients of the multivariate logistic regression model with all five baseline variables. To study whether discrimination could be improved further, a predictor describing disease subset at diagnosis was added to update the model. The area under the ROC curve of the original model, the recalibrated model and the updated model were compared graphically and by 95% CIs. Calibration of the original model was analysed using a plot of observed and expected frequencies and a frequency table comparing the observed and expected frequencies of survival.
Data were available for 1049 patients; 119 (11%) died within 5 years after diagnosis. There were 891 (85%) women, mean (SD) age at diagnosis was 50 (14) years and 301 (29%) were classified as having diffuse cutaneous SSc. Positivity for anti-centromere antibodies was present in 374 patients (36%) and positivity for anti-topoisomerase 1 was present in 304 patients (29%). The data in table 1 show that patients who died were on average older, more frequently were male, more often had diffuse cutaneous SSc and had a higher modified Rodnan skin score (mRSS). Urine protein, elevated ESR or low DLCO were more frequently present in patients who died. The causes of death were pulmonary disease in 42/119 (35%) (primary arterial hypertension, respiratory failure, lung cancer, interstitial lung disease, fibrosis, aspiration), cardiac disease in 35/119 (29%) (myocardial infarction, heart failure) and renal failure in 3/119 (3%); 20 patients (17%) died from cancer, sepsis or other diseases, in 18 (15%) the cause of death was unknown and one patient committed suicide.
Age, gender and disease subset were missing for 31 of the 1049 patients (3%), ESR values were missing for 147 patients (14%), DLCO values for 294 (28%) and the presence of urine protein was missing for 189 patients (18%). After imputation there were no changes in the distributions of these variables between patients who survived and those who died.
Univariate and multivariate logistic regression
The results of the univariate and multivariate logistic regression are shown in table 2. Univariately, all five baseline predictors were statistically significant and predictive for 5-year survival. An OR of >1 indicates a higher probability of dying in the 5-year period after SSc diagnosis. In the multivariate regression including all five variables, the size of the ORs was lower and the p values increased. Notably, the ORs for the three disease-related factors (presence of urine protein, elevated ESR and low DLCO) were of similar size, as was the case in the original model of Bryan et al.
Discrimination and calibration
The discriminative performance of the original prognostic model (solid line) is shown in figure 1A. In addition, we tried to improve the model in two ways. By estimating new regression coefficients, new weights were attached to the clinical predictors (recalibration) and, by the addition of a new predictor (limited cutaneous or diffuse cutaneous SSc), the model was updated. Figure 1A also shows the ROC curves of the same model with the newly estimated regression coefficients (recalibrated model: broken line) and the ROC curve if the SSc subset was added (updated model: dotted line). The area under the ROC curve of the original prediction model was 0.78 (95% CI 0.74 to 0.82). The discrimination of the original model could only be slightly improved, with both recalibrated and updated models having an identical area under the ROC curve of 0.81 (95% CI 0.78 to 0.85).
In table 3 the results of the observed and expected 5-year survival are shown. According to the original model of Bryan et al, the 5-year mortality of patients without any of the three risk factors at baseline is 7.1%, increasing to 100% for patients with all three risk factors. This is the mortality that is expected. The mortality that was observed in the current data was 2.2% for patients with no risk factors, increasing to about 31% for patients with two or three risk factors. This overestimation of the original model is also depicted in the calibration plot in figure 1B. Here the expected and observed probabilities are plotted analogous to table 3, each group of patients with 0, 1, 2 and 3 risk factors being represented by a dot.
SSc is associated with a significant reduction in life expectancy. A simple clinical useful prognostic model to predict 5-year survival in patients with newly diagnosed SSc was developed by Bryan et al in 1999 on 280 patients in a well-designed study.6 However, the prediction model was not validated in other samples. In the current study we tested the validity of the prognostic model using a large European multicentre cohort of patients with SSc.
According to the results of this study, the simple prognostic model showed reasonable discrimination but it was also considerably overoptimistic, meaning that it overestimated the risk of dying within 5 years. We also found that recalibration of the model by adapting the regression coefficients and updating the model by adding SSc subset as a sixth variable did not significantly improve discrimination. Accordingly, 5-year survival in patients with SSc newly classified according to the ACR criteria can be predicted with a simple clinical prediction formula using the presence of urine protein, elevated ESR and low DLCO.6 However, to take overestimation into account, we recommend that the probability of dying in 5 years should be reduced in accordance with the observations of this study (table 3).
The predictor variables we used in this study (age, sex, urine protein, ESR and DLCO) were prespecified by the existing prediction model.6 In the univariate analyses these variables were highly significant (all p<0.0008), but in the multivariate analysis the p values ranged between 0.002 and 0.063. Gender and the presence of urine protein had a p value >0.05; however, this does not mean that these variables should be rejected from the prediction model because the focus of the current analyses was on the quality (discrimination and calibration) of the predictions from the logistic regression model rather than on determining the best predictors. Perhaps surprisingly, using p values <0.05 in variable selection for prediction models is generally not indicated because it may make the prediction models worse rather than better.9 We tried to improve the prediction model by applying new weights for the prediction formula acquired by logistic regression (recalibration) and further by adding SSc subset as a new prediction variable (updating). However, neither of these approaches substantially improved the performance of the predictions.
For this study and for the original study by Bryan et al, logistic regression was used rather than analysis of survival times.6 This was because, for clinical prediction models, it is advantageous to have a fixed time frame (eg, 30-day mortality after surgery, 5-year survival after SSc diagnosis, 10-year survival for death from cardiovascular disease) and to calculate the corresponding probabilities. When analysing survival times, however, the resulting hazards inform about the strength of the predictors but do not readily inform about the clinical useful probabilities (eg, the probability of dying between now and 5 years). The time frame of 5 years was primarily chosen because the original model of Bryan et al had a time horizon of 5 years which seems clinically useful.
The need for external validation of a prediction model, in agreement with its intended clinical use, cannot be overemphasised. The performance of prediction models is usually better on the data on which the model has been developed than the performance of the same model in different patients.10 As a consequence, prediction models often produce overoptimistic predictions for new patients and therefore external validation of prediction models in external data is recommended.11 In our study the prediction model of 5-year survival developed by Bryan et al was indeed overoptimistic, especially if all three risk factors were present. This may be caused by the relatively small dataset of 280 patients and 76 deaths (27%) at 5 years in which the prediction formula was developed.6 12 If predictors are predefined, such as in the current study, this may lead to less overoptimism.12 Another reason for external validation is that the model of Bryan et al was developed >10 years ago. Survival in SSc has improved over the past several decades, mainly due to better treatment of renal crisis as death from renal crisis has reduced from 42% to 6%.13 Currently, the major cause of death in patients with SSc is not renal crisis but interstitial lung disease followed by pulmonary hypertension.13 14 This may have changed the performance of the prediction model and also gives rise to the question whether a new model with new predictors should be developed. Sex, age at diagnosis, disease subset and organ involvement are generally accepted determinants of survival.15,–,17 Recently, a similar prognostic model for 5-year survival in SSc was developed and validated.18 The model also contains five variables (age, gender, renal involvement, DLCO and forced vital capacity (FVC)), which is very similar to the variables included in the earlier model of Bryan et al (with ESR and no FVC). The objective of the current study was not to develop a new model or to select new predictors. For development and validation of new predictors and a new prediction model, the design of the study would be different from the current approach and would include a sample for development and one for external validation, a variable selection procedure for small samples and shrinkage using a bootstrap method.12
This study has limitations. The sample size may be regarded as small (1049 patients and 119 deaths) but is sufficiently large for the purpose of this study. With five pre-specified variables (age, gender, urine protein, elevated ESR and low DLCO) and 119 patients (11%) who died, the number of events per variable exceeds 20, which indicates sufficient power.9 Death certificates were not obtained for practical reasons. However, as the cause of death was not the outcome in this study, we do not think this will have led to major bias. Missing values occurred for the variables in the prediction model (ESR in 14%, DLCO in 28% and the presence of urine protein in 18%). Notably, deleting patients with missing values does not only lead to a loss of power but also may induce bias.8 After imputation of the missing values with regression techniques, there were no changes in the distributions of these variables for patients who survived and those who died. Furthermore, we included in the study only patients who fulfilled the ACR classification criteria for SSc, while 15–39% of patients with SSc fail to satisfy the ACR classification criteria.7 19 Therefore, the model may not fit all patients encountered in daily practice.
The model of Bryan et al contains dichotomised variables that facilitate clinical use; however, this leads to a loss of information. For future research a new model for prognosis of survival in SSc should be developed including other variables and tested against the performance of the existing model of Bryan et al. A new model could include variables associated with disease subset such as autoantibodies and skin score, and variables predictive of SSc-related causes of death such as primary arterial hypertension. Moreover, a prediction model could be developed in a large cohort with cause-specific death (eg, renal, lung or heart) as outcome. When data are available, these models should also be tested in patients with ‘early’ SSc.20 Patients are already being diagnosed earlier in the disease process and consequently survival after diagnosis will be longer.
In conclusion, a simple prognostic model using three disease factors to predict 5-year survival at diagnosis in SSc can be used in daily clinical practice. Upon validation, the model showed reasonable discriminatory performance but considerable overoptimism. We therefore recommend that the probability of dying in 5 years should be reduced in accordance with the observations of this study.
Funding This study was endorsed and has been supported by an epidemiological grant from the European League against Rheumatism - Scleroderma Trials and Research group (EUSTAR).
Competing interests None.
Ethics approval Because this is a multicentre study, each institution obtained ethical approval from its own ethics committee.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.