Materials and methods

Results

A total of 290 patients with SSc (244 female, 46 male; age 8–87 years, mean age 53) were enrolled in the study. The epidemiological and clinical features of this series are reported in part I of the study. Here, we briefly summarise the main findings.

As a result of the selection process, all the 290 patients satisfied the preliminary ACR criteria for the classification of SSc7; specifically, 165 satisfied the major criterion (scleroderma proximal to metacarpophalangeal joint), while 125 met at least two of the three minor criteria (pitting scars, sclerodactyly, or bibasilar lung fibrosis). One hundred and seventy three had the limited form (149 female, 24 male, age 21–87 years, mean age 55) and 117 the diffuse form (95 female, 22 male; age 8–86 years, mean age 49) of the disease, according to the criteria of Le Roy et al. 9

Table 3 shows the results of univariate analysis of the chart items that correlated with our gold standard for disease activity (the consensus disease activity scores). We ruled out from the analysis the items with a significant number of missing values (for example, pulmonary hypertension) and those that had been investigated by different methods (for example, lung interstitial involvement byx ray or high resolution computed tomography (HRCT)). Through multiple linear regression analysis we were then able to define three different sets of items that correlated with disease activity in all patients with SSc, in patients with dSSc, and in patients with lSSc, respectively (table 4). A satisfactory regression coefficient was obtained between each set of items and the gold standard in each patient group.

Our findings (summarised in table 5) allowed us to construct three weighted disease activity indices: one for the entire SSc population, one for the dSSc subset, and one for the lSSc subset. The weight of a given item was assigned on the basis of the regression coefficient (b) and adjusted so that the maximum value for each index was 10. The disease activity for a given patient could therefore be calculated by summing the weights of the criteria fulfilled.

We validated our criteria sets using the jackknife statistical approach. Thus we calculated in multiple linear regression analysis the regression coefficient between the consensus disease activity score and the disease activity items and that by the calculated index and the same items by leaving out one patient at a time. As the whole series is concerned, the regression coefficient between the score and the items (n=152) ranged from 0.831 to 0.846 (mean 0.837; SE 0.0002; confidence interval (CI) 0.836 to 0.837) and that between the index and the items ranged from 1.0 to 1.0. For lSSc, the regression coefficient between the score and the items (n=91) ranged from 0.763 to 0.787 (mean 0.778; SE 0.0004; CI 0.777 to 0.779) and that between the index and the items ranged from 1.0 to 1.0. Finally, for dSSc, the regression coefficient between the score and the items (n=74) ranged from 0.751 to 0.789 (mean 0.768; SE 0.0006; CI 0.766 to 0.769) and that between the index and the items from 1.0 to 1.0. In conclusions, all the three indexes were validated.

Predictably, a significant correlation was found between the activity index score calculated for each patient (by summing the weights of the criteria fulfilled) and the consensus disease activity score (previously assigned by the protocol management team) using Pearson's correlation coefficient (r=0.763, p<0.0001 for dSSc; r=0.763, p<0.0001 for lSSc; andr=0.830, p<0.0001 for the whole series). An activity score, however, constitutes a comparative ranking rather than an absolute number. It is therefore more appropriate to analyse the validity of this score on an ordinal scale and to test its efficacy by Spearman's rank correlation coefficient. This analysis also showed significant correlations between the index score and the consensus score (r _s=0.760, p<0.0001 for dSSc; r _s=0.787, p<0.0001 for lSSc; r _s=0.835, p=0.0001 for the whole series); that is, the higher the index score, the higher the consensus score. These correlations were also validated by the jackknife technique. For the whole series, the correlation coefficient (r _s) between the score and the index ranged from 0.832 to 0.844 (mean 0.835; SE 0.0002; CI 0.835 to 0.835). For lSSc, r _s ranged from 0.752 to 0.804 (mean 0.787; SE 0.0006; CI 0.786 to 0.788). Finally, for dSSc, r _s ranged from 0.749 to 0.791 (mean 0.760; SE 0.0008; CI 0.758 to 0.761).

Finally, we compared the disease activity index scores with the qualitative assessments of disease activity (“inactive to moderately active” or “active to very active”) made by the two protocol management team members. Figure 1 (A–C) shows the ROC curves for the entire SSc series, and for the dSSc and lSSc subgroups, respectively. For all three conditions an index of three showed a quite high specificity and a fairly good sensitivity.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Receiver operating characteristic curves showing the relation between the value of the calculated activity index and the presence of active or very active disease in (A) SSc whole series; (B) dSSc; (C) lSSc. AUC = area under the curve.

Discussion

We have constructed three preliminary sets of criteria to calculate disease activity in patients with SSc as a whole, in patients with dSSc, and in patients with lSSc, respectively. Defining disease activity in SSc is much more difficult than it is for other rheumatic and connective tissue diseases, such as systemic lupus erythematosus and rheumatoid arthritis, in which inflammation has a key role and flares and quiescent phases can be easily recognised.28-31 Patients with SSc, particularly those with the limited form, do not present such a clear picture. Nevertheless, from a pathophysiological point of view, two distinct stages can be defined in SSc: firstly, a potentially reversible stage in which activated cells directly or indirectly activate or damage endothelial cells and stimulate fibroblasts to overexpress genes encoding the extracellular matrix components; and secondly, a definitely irreversible stage in which vascular occlusion and interstitial fibrosis occur.32-34 The first stage reflects activity and the second, damage. The first stage is not marked by clear episodes, however, and the two stages are not mutually exclusive, whether the disease affects a single organ or different organs in the same patient. Finally, the two stages are difficult to distinguish, especially in those patients with longlasting, indolent SSc in its limited form.

The European Scleroderma Study Group did its best to overcome the problem of the unclear symptomatology of SSc in its study design. The standardised clinical chart drawn up by the group contained all of the symptoms, laboratory and other diagnostic parameters most widely used by clinicians treating this disease (for a total of 88 items), carefully defined according to the most authoritative sources available. A gold standard was then set by assessing disease activity in the patient series both semiquantitatively and qualitatively. Both these measures of disease activity were analysed and found to be reliable. Indeed, the disease activity scores calculated by the three protocol management team members were found to be significantly correlated with one another (ICC=0.684; p<0.0001). A good correlation was also found between the qualitative evaluations assigned to the patients by the two protocol management members. It should be noted that the level of agreement for the qualitative evaluation could actually be considered greater than that demonstrated by the k value, because we discovered a systematic bias (one member consistently gave a higher activity evaluation than the other) (table 2).26

The activity indexes developed by us have been derived from the charts of 290 patients with SSc recruited by 11 European centres. Because the prevalence of some disease aspects in the series from various centres showed a high variation (see part I of the study), it might be suggested that the variability can be explained by observer error and, consequently, that these aspects are not reliable. However, it should be emphasised that the observer error, if any, would probably have been random, causing significant misclassification and making it difficult to detect any association. The fact that the measures are useful in predicting the consensus activity score is therefore unlikely if there had been observer error. In addition, we must emphasise that all the participants in the study were experienced clinical investigators who had been provided with clear cut guidelines. Therefore, we believe that the variability in the prevalence of some items among various series must be ascribed to a different pattern of attendance at each centre as shown by differences in age, sex, and subset distribution.

It should be noted that all the items in the three indexes were searched for in a high percentage of the 290 patients investigated. The percentage of values missing ranged from 0 (for total skin score, scleredema, digital necrosis, and arthritis) to 6.2% (for Δ-vasc) and exceeded 15% only for hypocomplementaemia (19.7%). Because each of the indexes is made up of at least four items, we do not believe that the presence of one item with a slightly higher than acceptable percentage of missing values would seriously affect the results.

Of the items found to be related to disease activity, hypocomplementaemia is not commonly thought of as a laboratory parameter characteristic of SSc. However, both Seiboldet al 35 and Benbassatet al 36 found hypocomplementaemia in 12% and 22.5% respectively—that is, in percentages not different from that detected by us (14%).

For the Δ-factors, we chose to rely on patients' self reporting at enrolment and on both patients' and doctors' global assessment at six and 12 months' evaluation. Of the three sets of criteria, that for dSSc is based only on Δ-factors, those for the whole series and lSSc both contain three Δ-factors. Such an approach might be questioned because of lack of standardisation. However, it has been accepted as valid and important in assessing disease activity in rheumatoid arthritis.31 ,37

We have shown that the criteria identified in our study are correlated with disease activity. Significant correlations were found using both the Pearson and Spearman correlation coefficients between the disease activity indexes and the consensus activity score. Moreover, the construct validity of our indices was tested by jackknife statistical analysis29 but remains to be confirmed on separate groups of patients. ROC curves, constructed by plotting the value of the index score against the qualitative assessment of disease activity, were quite satisfactory. An index of three was found to identify, with a quite high specificity and fairly good sensitivity, those patients with active to very active disease in both disease subgroups and in the group of patients with SSc as a whole. Actually, an index of three would define active to very active disease with sensitivity ranging from 62 to 81% and specificity ranging from 86 to 93%. It will be important, therefore, to identify other parameters correlated with disease activity to improve the somewhat low sensitivity of the present, preliminary indices. At the beginning of the study we asked all participants to store aliquots of serum and plasma for each patient. In the next phase of this study we will analyse various laboratory parameters (such as circulating activation markers) in greater detail in order to improve the sensitivity of our indices.

It should also be noted that our disease activity indices may not be entirely comprehensive because no patients with SSc with renal crisis were enrolled. Defining activity criteria in such patients would appear to be straightforward, but still remains to be done on the basis of prospective studies. In addition, discriminant validity (that is, sensivity to change) still needs to be tested.

At its current stage our study has certain limitations. Our activity indices were constructed on the basis of the correlations of a series of diagnostic parameters with the disease activity scores arrived at by three protocol team members (the “gold standard”). The activity scores subsequently calculated using the indices were then compared with the gold standard disease activity scores. Therefore our study design has an inherent element of circularity. In addition, although this study was based on the analysis of data from a large number of actual patients with SSc, we cannot claim to have confirmed and validated an already existing consensus about the criteria for disease activity. Our preliminary indices represent a tool which may be used for further research in order to reach a consensus. In this sense, our study has analogies with the continuing study by Furstet al using the Delphi technique.

Activity criteria for a given disease must be valid, reliable, and easily measurable in a typical clinical setting. The indices presented in this paper were developed using two reliable measures of disease activity as the gold standard, and all the criteria included are reasonably easy for any clinician specialising in connective tissue diseases to determine during a routine evaluation of their patients. Moreover, because our activity criteria are based on data from real patients, they reflect everyday clinical practice and may be universally applied. Their use would also facilitate the gathering of comparable data in studies conducted by different groups. The fact that some examinations which specifically measure the extent of internal organ involvement (HRCT, echocardiography, upper gastrointestinal series) were not included among the criteria might slightly lessen the accuracy of the indexes, but in compensation make it possible for the indexes to be used in any clinical setting.

In conclusion, we have developed three preliminary sets of disease activity criteria, one for patients with SSc as a whole, one for those with limited SSc, and one for those with diffuse SSc. These indices appear to be simple and reliable; we are currently carrying out further analyses to confirm their construct validity and assess their discriminant validity (that is, sensitivity to change).