Article Text

PDF

European Scleroderma Study Group to define disease activity criteria for systemic sclerosis. III. Assessment of the construct validity of the preliminary activity criteria
  1. G Valentini1,
  2. W Bencivelli2,
  3. S Bombardieri2,
  4. S D’Angelo1,
  5. A Della Rossa2,
  6. A J Silman3,
  7. C M Black4,
  8. L Czirjak5,
  9. H Nielsen6,
  10. P G Vlachoyiannopoulos7
  1. 1Second University of Naples, Italy
  2. 2Clinical Immunology/Rheumatology Units, Department of Internal Medicine, University of Pisa, Italy
  3. 3University of Manchester, UK
  4. 4Royal Free Hospital, London, UK
  5. 5University of Pés, Hungary
  6. 6Rheumatology Unit, Herlev Hospital, Denmark
  7. 7National University, Athens, Greece
  1. Correspondence to:
    Professor G Valentini, U.O di Reumatologia, Edificio 3, Via Pansini, 5 Naples, Naples, Italy;
    gabriele.valentini{at}unina2.it

Abstract

Objective: To further assess the construct validity of the three European Scleroderma Study Group (EScSG) preliminary activity indices for systemic sclerosis (SSc): for SSc as a whole, for diffuse SSc (dcSSc), and for limited SSc (lcSSc).

Methods: 30/290 SSc clinical charts collected for the EScSG study used to develop activity criteria for SSc were selected and sent to four clinical experts in SSc. The experts ranked the charts from 1 to 30 (1=lowest activity, 30=highest activity). The relationships among the ranks given by each investigator and each of the three scores, and between any two of the ranks were investigated.

Results: A consistently significant correlation (rs=0.530–0.712) was found between the ranks given by each of the four investigators and the index for the entire patient group. A similar level of agreement was detected between each couple of the four experts (rs=0.428–0.720). Moreover, the ranks given in patients with an index >3 were significantly higher than those given for patients with an index ⩽3. This cut off point had previously been shown to best discriminate patients with active disease.

Conclusions: Of the originally developed activity indexes, the whole series index has been externally validated. The index comprises the first preliminary, but necessary, groundwork to improve the concept of disease activity in SSc, which is still ill defined. It can be used as a preliminary activity index in clinical investigational studies.

  • systemic sclerosis
  • disease activity criteria
  • construct validity
  • EScSG, European Scleroderma Study Group
  • dcSSc, diffuse cutaneous systemic sclerosis
  • lcSSc, limited cutaneous systemic sclerosis
  • SSc, systemic sclerosis

Statistics from Altmetric.com

The European Scleroderma Study Group (EScSG) recently proposed three activity 10 point indices for systemic sclerosis (SSc): one for SSc as a whole (whole series index), one for diffuse cutaneous SSc (dcSSc), and one for limited cutaneous SSc (lcSSc).1,2 The indices were created after analysing the clinical charts of 290 patients with SSc from 19 European medical centres and identifying the set of items that was most significantly correlated with a subjectively defined disease activity score (on a scale from 0–10). The latter had been assigned to the patients under blinded conditions by three members of the protocol management committee and resulted in a reliable measure.2

In that study we assessed the construct validity of the indices by the jackknife technique3—that is, a statistical method in which one patient at a time is excluded and the significance of the data is evaluated by its dispersion. This approach, while statistically correct, is characterised by an intrinsic circularity: clinical investigators assign a score which is used to establish an index, which in turn is validated by comparing it with the assigned score.

We therefore decided to further test the validity of our indices by assessing the correlation of each of them with the activity ranks given by a different set of observers.

MATERIAL AND METHODS

Table 1 shows the three preliminary activity indices established in our previous study.2 For the external validation of the indices, the protocol management members (GV, SB, SD’A, ADR) stratified the dataset in three subgroups with increasing activity (inactive, moderately active, and active/very active). Thirty charts (10 out of each group) were then randomly selected and sent to four experts (CMB, LC, HN, PGV) identified as representative of different countries and geographical areas and available to give their evaluation at the time the study was carried out. The four experts were requested to rank the charts under blinded conditions from 1 to 30 (least active disease to most active disease).

Table 1

Proposed EScSG disease activity indices for SSc

We then investigated the correlation between the ranks given by each expert and the respective activity indices for each patient using a Spearman rank correlation. In our previous work we had shown that a score of over 3 was the best single point discriminator between active and inactive disease. We then compared the median rank, for each expert, between those with and without active disease using the Mann-Whitney U test.

RESULTS

Table 2 shows the correlations between the disease activity rankings given by each of the four experts and the three EScSG indices. A significant correlation was found between the ranks given by each of the four external investigators and the derived index for the whole series. However, the correlations were not uniformly significant for the dcSSc index and very rarely significant for the lcSSc index. Therefore, we decided only to consider the whole series index for further evaluation.

Table 2

Correlations between the rankings given by four investigators and the calculated index in the whole series, in patients with dcSSc, and in patients with lcSSc

Table 3 shows the correlations between the ranks given by the four experts for the 30 charts. This table clearly shows that even among experts in the field there is not a very high level of agreement in the assessment of disease activity.

Table 3

Correlations between the rankings given by four investigators in 30 patients with SSc considered as a whole

Figure 1 shows the distribution of the ranks given by each expert in patients with a whole series index ⩽3 and >3 (that is, the value that had been found, in our previous paper, to discriminate between patients with low and high activity). The ranks in patients with high activity disease were in each case significantly higher than those given in patients with low activity.

Figure 1

Distributions of the ranks given by each investigator in patients with active and inactive disease according to the whole series index.

DISCUSSION

We identified and validated three sets of criteria devoted to defining disease activity in SSc as whole, in dcSSc, and in lcSSc. Our first validation, however, was carried out by the jackknife approach,3 which is affected by an intrinsic circularity. In this report we provide the results of an external validation process. Our data show that the index derived for use in all patients is the only one which was validated by all the external investigators. We are inclined to think that the unsatisfactory performance of the dcSSc index may be due to the small number of patients investigated, and the unsatisfactory performance of the lcSSc to the difficulties in identifying activity in patients whose disease has an indolent course. Therefore, further evaluation of these subgroup indices should be undertaken.

To define disease activity criteria for SSc, we used the same methodology previously used by two of us (SB and WB) to define both classification criteria for Sjögren’s syndrome5 and activity criteria for systemic lupus erythematosus, which has been validated both internally6 and externally.7 Therefore we relied on real patients. Such methodology had the substantial advantage of being based on parameters which are currently evaluated in each centre, enhancing the feasibility of the developed index.

In the present study, as well as in our first study, the correlation between the activity items and the “gold standard” (that is, the activity score, from 1 to 10, given in blinded conditions by protocol management members2) was lower than that detected in other diseases such as systemic lupus erythematosus.6 This might suggest that the developed index has an intrinsic weakness. However, when a correlation between the ranks given by different experts was carried out (table 3), a similar level of agreement was detected. We therefore suggest that the low level of agreement found reflects the low level of concordance on the degree of disease activity in SSc. Defining activity in SSc is different from defining activity in other diseases such as rheumatoid arthritis and systemic lupus,6,8–10 because of the lack of clear flares and quiescent phases and its indolent course.

The presence of three Δ factors based on patient self assessment may cause some concern because of the lack of standardisation. However, such an approach has been accepted as valid and reliable in assessing disease activity in rheumatoid arthritis.10,11

In conclusion, we have developed a validated index to assess disease activity in SSc. This index must still be considered preliminary. However, it should be viewed as the first necessary step towards developing a better agreement among different investigators in the assessment of disease activity in SSc. It can be used as a preliminary activity index in clinical investigational studies. Nevertheless, the search for other measures of disease activity in SSc must continue.

REFERENCES

View Abstract

Footnotes

  • See also “European Scleroderma Study Group to define disease activity criteria for systemic sclerosis. IV. Assessment of skin thickening by modified Rodnan skin score” in this issue of the journal p 904–5.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles