OBJECTIVES

Numerous ways of assessing RA radiologically have been developed and tested for degree of reliability and other characteristics. The objectives of this work are to review the principal joint damage scoring systems and compare their measurement properties, advantages, and limitations.

LITERATURE SEARCH

Medline was searched for articles reporting the use of scored radiographs in RA. Key words (Mesh, title and abstract) included “RA”, “radiographs”, “radiographic”, “radiologic”, “x ray”, “scoring method”, “score”, “comparison”, “progression”, “reproducibility”, “reliability”, and “sensitivity to change”. A selection of articles pertaining to the description of scoring methods and to the analysis of their measurement properties was obtained by the search and by examining relevant reference lists and other review articles. Particular care was taken to identify papers that compared two or more x ray reading and scoring methods.

ABNORMALITIES IDENTIFIED (TABLE 1)

The numerous abnormalities that can be seen on radiographs of patients with RA (such as soft tissue swelling, osteoporosis, erosions, JSN, subluxation and malalignment, and ankylosis)1 reflect disease severity and progression. No existing scoring method includes them all, but erosions and, to a lesser extent, JSN seem to be most widely accepted as important.8 They are highly specific to RA, can be reliably assessed, and provide independent information.4 ,12 ,19 The relative weight given to erosion versus JSN varies, and no consensus has yet been established.10 The potential ratio of weighted erosion/JSN is always one or higher: 1.2 for Sharp, 1.7 for van der Heijde/Sharp, 1 for SENS. The Larsen methods combine erosion and loss of cartilage scores. Because osteoporosis is difficult to quantify and highly dependent on radiographic technique, it is generally excluded.7 ,8 ,14 ,29 Substantial destruction related to cyst formation is not captured by Larsen's or Sharp's (1985) systems. Cyst formation can be counted as erosion. Subluxation and luxation are responsible for another difficulty in scoring films: very severe luxation makes it impossible to measure the degree of erosion or even to detect erosive changes. Healing phenomena are also a problem. The Sharp and Larsen methods do not consider healing as manifested by improvement in bone damage.30 Most authors do not report whether they considered this problem in developing their scoring strategies. Those that did, did not allow for decreases in score.31 ,32

METRIC SCALING PROPERTY

By relating radiological abnormality scores to clinical features that reflect the severity of RA, it is possible to make a semiquantitative assessment of the value of radiographic change. The use of graphical statistical techniques allows the course of the disease to be described visually, and numerical assessment of the progression of radiological abnormalities has prognostic value.7 However, semiquantitative scoring techniques in the evaluation of RA are limited by the association between the extent of a finding on the radiograph and the score assigned to it. An ideal scoring method supposes a linear relation between the quantity of change and the scale itself. This is not the case for the Larsen and the Sharp methods.26 ,30 The SES method developed by Wolfe targets this goal.28 Other methods use a count of abnormalities on a continuous scale.33

HAND OR FOOT RADIOGRAPHS, OR BOTH?

With the exception of the Sharp method, all scoring techniques are based on the evaluation of hand and foot joints. In most studies comparing the Sharp method with other techniques, only hand radiographs were used, and the authors did not speculate on whether omission of the data for feet might have influenced their results.29 ,34-37 Radiographic progression in RA has been shown to occur early, and the first erosions are more often found in the feet than in the hands. Van der Heijde underlined this fact in a study based on the Sharp/van der Heijde modification.17Plant et al justified extending the Sharp method to the joints of the foot38 by pointing to the fact that erosions of MTP in the first year were almost as good predictors of outcome as the overall rate of radiological progression, whereas wrist and MCP erosions became better predictors beyond the first two years. This is consistent with MTP damage occurring at an earlier stage, and other joints being affected later. Paimelaet al extended the Sharp method by including joints of the foot39; van der Heijde also successfully used this modification. These authors emphasised the importance of evaluating both hand and foot radiographs when assessing therapeutic intervention in early RA.

JOINT SITES (TABLES 2 AND 3)

Most scoring techniques assess the same joint areas. Early systems included more joints. It has been shown that eliminating areas that are technically difficult to read (hamate and capitate bones, radioulnar and lunate-triquetrum joints) and areas that are not commonly affected (DIP and MCB, for example) leaves an appropriate sample of the joints in the hands and wrists that still accurately represents the radiological abnormalities of patients with RA.8 ,14 In the feet, MTP are the sites where erosions first develop and thus must be considered in any scoring method. Finally, the following joints are generally counted: PIP, MCP, MTP, IP, and joints of the wrist. Considerable information is gained from assessment of the PIP, MCP, and wrist joints, in particular.1

RADIOGRAPHIC TECHNIQUES

Any study based on radiographic assessment is highly dependent on the technical quality of the radiographs, particularly with regard to the reproducibility of the results. It is important to consider several technical variables. Firstly, accurate joint positioning is essential. Most investigators have opted for the posteroanterior view for hand and foot radiographs. However, other views such as the Norgaard view (a 45° supine view with straight finger) and the Brewerton view (a tangential view with the MCP joints flexed at 65° and with a 15° volar beam) have been used. When these three different approaches were compared in a prospective study,40 the authors found no significant advantage in the Norgaard or the Brewerton view compared with the standard posteroanterior view used in most investigations.12 ,17 ,38 Secondly, the exposure of the film is extremely important, as evidence of erosions is lost on both under- and overpenetrated films. Thirdly, high resolution films are essential in detecting early erosive disease. Single screen/film combinations give both an acceptable system speed and an acceptable exposure time.3 Although excellent baseline film may be obtained, it is difficult to maintain high quality data because of technical variability. Even minuscule changes in the rotation of an imaged joint will cause modification of the film.

Thus accurate interpretation of radiographs depends on appropriate positioning, film exposure, screen film combination used, and reproducibility of radiographic production. Most technical problems are due to the fact that plain radiographs are two dimensional pictures of a three dimensional object.

READING STRATEGIES

Radiographs can be scored in a random order (that is, single order; radiographs are randomly ordered with regard to patient and sequence), paired without knowledge of the sequence (that is, paired order; two radiographs of the same patient without information about their chronological sequence), or ordered with known sequence (that is, chronological order; two radiographs of the same patient and of known chronological sequence). There are advantages and disadvantages for all these methods. Scoring in a chronological order probably gives the reader most information, but it introduces a bias in that he or she may expect progression of damage over time. Paired radiographs have the advantage that the positioning can be compared. Random order has the disadvantage that the quality of radiographs can vary greatly, but seems to give the best results and to be a good way of validating a scoring method. Most studies reviewed here used a known sequence—that is, chronological order, for reading radiographs.

Salaffi et al compared these three different reading procedures using the Sharp method, with two hand radiographs for each patient (at baseline and after 18 months).41 They concluded that paired order is preferable. In a randomised, controlled trial with multiple readers, Fries et alconcluded that paired reading (reading the films pairs simultaneously) was preferable to separated reading (reading the films at separate times).10 Using several statistical methods, Ferraraet al produced results suggesting that paired reading was the most suitable for evaluating the progression of joint damage.42 Recently, van der Heijdeet al devised two studies using the Sharp/van der Heijde method to score radiographs of hands and feet32: one compared random, chronological, and paired order; one evaluated random, chronological, and so called “single-paired” order (the films are grouped by anatomical region from a particular patient at a single point in time). It was concluded that chronological order was more sensitive to change than the other approaches, and that the difference was particularly pronounced with longer follow up.

NUMBER OF RATERS

The number of raters reading radiographs in reviewed studies ranged from one to more than 10. The optimum number was defined by the highest efficiency. Using the results of more than one reader reduced measurement error and gave more precise results (averaging scores increases the reliability of progression scores) but required more time for rater training and therefore cost more. The number of patients needed depends primarily on the amount of change within the subject group, and may be reduced as the number of raters increases. Finally, the number of raters varies according to the purpose of the study. A randomised, controlled trial conducted by Fries et al showed that two readers was the best compromise,10 and that reader training was essential: of eight experienced readers, the four who were trained (that is, who had extensive experience in scoring radiographs in clinical trials) exhibited consistently higher agreement. Other comparisons between experienced and inexperienced readers for scoring methods showed no significant difference,14 ,26 ,40 though agreement was slightly higher among experienced raters. The variability in the scores of experienced and inexperienced readers remained within a tolerable range. Most studies based on radiographic scoring systems have used trained readers.12 ,19 ,36 ,39 ,43 The discipline of the reader (rheumatologist or radiologist) had no effect on the quality of the readings.3 ,10

TIME CONSUMPTION

Several authors have calculated the time needed to score radiographs with different methods. Wassenberg et al found that the time to score seven radiographs of hands and feet was 3.9 minutes for Larsen, 19 minutes for Sharp, 25 minutes for the Sharp/van der Heijde method, and 9 minutes for the Ratingen method.43 Other studies gave similar results for the Ratingen score method and the Sharp/van der Heijde method.19 ,26 The time needed to score seven radiographs of hands and feet was 7 minutes for SENS.19 The time needed to score 12 radiographs of hands and feet with the Sharp/van der Heijde method ranged from 11.1 minutes to 20.5 minutes.32The time needed is one drawback of both the Sharp method and the Sharp/van der Heijde method; it is related to their higher degree of detail as compared with the Larsen and SENS methods.

RELIABILITY (TABLE 5)

The value of any scoring method used to measure a clinical variable depends on its reliability as shown by intra- and interreader reproducibility.1 This can be assessed at a given point in time using absolute scores, or between two time points using values related to progression. Spearman and Pearson correlation coefficients, intraclass correlation coefficients (ICC), and κ statistics are used to assess reliability. However, the most appropriate methods are ICC and κ statistics because Spearman and Pearson correlation coefficients measure only association and not agreement.44

The most detailed methods, such as Sharp, are most reliable. The level of intrarater reliability is higher than those of interrater reliability and reliability for progression. Most studies in this area compared the Sharp and the Larsen methods. Plant et al compared the Sharp, Larsen, and C:MC methods.38Expressed as percentages of the maximum score, Sharp and Larsen scores showed substantial changes in the first two years (negligible for C:MC). The Spearman correlation coefficients for the three methods were similar: intrarater and interrater reliability ranged from 0.90 to 0.99; reproducibility of changes in scores between successive films was better with Sharp than with Larsen. The κ values were higher for JSN than erosions or Larsen and better for intrarater agreement. In conclusion, the Sharp and Larsen methods performed similarly in early RA, but the Sharp system reproducibility was better for change in score. Similar results were found in an other study.45

This was confirmed when Guth et al compared the same three methods using ICC and Bland and Altman graphics (see below).36 The results for intrarater reliability were similar to those in the earlier study. Paimela et al compared the Sharp, Larsen, and Larsen's modified (especially for long term studies) methods.39 Reliability assessed by ICC exceeded 0.88 and 0.94 for interrater and intrarater agreement, respectively. When van der Heijde scored radiographs of 20 patients,19 she found ICC (as defined by Streiner and Norman) of 0.99 for Sharp/van der Heijde and 0.98 for SENS. This indicated high intrarater reliability for both methods during the first five years. Expressed as percentages of maximum scores, the means of SENS were higher than the means of Sharp/van der Heijde. Wassenberget al compared the Larsen, Sharp, Sharp/van der Heijde, and the Ratingen score methods.43 He determined intrarater reliability using the quotient of intrapatient SD (standard deviation) to intrarater SD. Higher values indicated better results. As expected, the most detailed methods—that is, Sharp and Sharp/van der Heijde, were more reliable, followed by the Ratingen score and, finally, the Larsen method.

Rau et al studied the reliability of their method26 and found it to range from 0.70 to 0.90, with higher values for intrarater reliability. The ratio of radiographic change SD to intra- or interrater SD defined the reliability over time. If this is equal to 1, detected radiographic change is due to measurement error; if it is over 1, the method can detect true change from one point to the next, with higher values corresponding to higher sensitivity. The ratio ranged from 1.7 to 3.2, indicating that the method could detect real radiographic change over time.

A graphical method described by Bland and Altman is used to assess reliability between two methods46 by plotting the difference in progression between the two observers for each method. The y axis represents the difference in progression as assessed by the two observers, and the x axis the mean of progression as assessed by the two observers. The ideal situation would be for all points to be situated on or close to y=0. A confidence interval is calculated using a normal approximation. This method determines whether reliability is sufficient and whether one reader scores higher than the other, thus providing an absolute and metric estimate of random measurement error. This additional method was used in some studies.31 ,32 ,36

SENSITIVITY TO CHANGE (TABLE 6)

Sensitivity to change is measured to determine whether a radiographic scoring method can detect a real change over time. Several authors have compared the sensitivity to change of various methods. Some used the standardised response mean (SRM), whereas others used the minimal detectable change (MDC) (also called the smallest detectable difference (SDD)), or the G coefficient. SRM is defined as a unitless expression of change and is calculated by dividing the mean of the difference between scores at two times by the SD of change score. A value above 0.80 is considered to reflect high potential to detect changes. The G coefficient is also defined as a unitless expression of change, but corresponds to the ratio of the signal SD to the noise SD derived from a mixed effect analysis of variance model.32It can range from 0 to 1, and values above 0.80 are considered to be good.

Rau et al determined that MDC = 3.3% of the maximum score (intrarater) and MDC = 4.6% of the maximum score (interrater) for their method. Seven points were scored in 20 patients: 14 patients selected randomly from 128 subjects in a prospective trial were followed up over five years, and six patients with severe progressive disease were followed up over 10 years in an outpatient department. MDC was defined by 1.96 × 2 × intra- (or inter-)rater SD with 95% confidence if the variances were from a normal distribution. Below the value of the MDC, the difference between two readings of the same set will lie with a probability of 97.5%. A difference above this value confirms true change in the radiographs. They found that the MDC was substantially better with the new method than published data gathered using the Larsen method.26

The sensitivity to change of the Sharp/van der Heijde method using the G coefficient was shown by van der Heijde to range from 0.22 to 0.3932 (10 patients, one year follow up). In another study the same author determined the SRM of SENS (1.15, 1.63, and 1.60 for sequential, paired, and random order, respectively).47Paimela et al calculated SRM by dividing the mean change between the baseline and year 1 scores by the SD of change score in 83 patients.39 Their result was 0.80 for the Larsen method, 0.88 for the variant Larsen method (1995), and 0.72 for the Sharp method. The variant Larsen method was the most responsive system, but the differences between the three methods were small. These findings indicated that all three methods were sensitive to change during the first year of RA.

When Wassenberg used the same formula as Rau to calculate MDC43 the results were as follows: 2.3% for the Sharp and Sharp/van der Heijde methods, 3.2% for Larsen, and 3.3% for Ratingen (20 patients). All results were expressed as a percentage of maximum scores. Smaller values indicate better precision. In a comparison between SENS and Sharp/van der Heijde, van der Heijde used the Norman's quasi-classical ICC formula and SDD to calculate the sensitivity to change.19 She found G coefficient = 0.88 for SENS and 0.84 for Sharp/van der Heijde modification. From derived results of SDD, it was concluded that SENS was approximately as good as Sharp/van der Heijde in detecting progression.

More recently, Guillemin et al compared scoring methods (Sharp, Sharp/van der Heijde, Larsen, Rau/Larsen, and SENS methods) using hands radiographs of 20 patients at two different times by applying adjusted SRM.48 As elsewhere, adjusted SRM was computed as the mean of the difference between scores at two times over its standard deviation. The difference between this value and “classical” SRM lies in the adjustment, based on a mixed model, for fixed rater and country effects, and the variance was adjusted according to interpatient variability. When the Larsen method was used as reference, the ratios between SRMs showed that all other methods (excluding SENS, but restricted to the hands) were more sensitive to change than the reference (van der Heijde/Sharp over Sharp over Rau/Larsen).

Drossaers-Bakker et al compared three scoring systems for the long term assessment of RA.49Sensitivity to change was calculated using SRM for the van der Heijde/Sharp, Kellgren and the “Sharp max” scores. To look for a possible ceiling effect, the Sharp max score consists of the extension of the erosive score for each hand joint to 10 (rather than 5), as for the erosive score for the feet. The SRM invariably exceeded 0.80. SRM values over 0.20, 0.50, and 0.80 are considered to reflect low, moderate, and high potential to detect changes, respectively. The Kellgren and Sharp scores appeared to be equally sensitive to change over time in early RA, whereas the Kellgren score proved to be more sensitive late in the disease.

Wolfe et al compared the ability of the SES score and the Larsen score to distinguish change over time.28 Two radiographs were randomly selected from each of 857 patients who had at least two radiographs over an average three years' follow up. They calculated for the two methods an effect size (that is, standardised change score) defined by the ratio of the difference between the two films to the pooled standard deviation. They found 0.38 and 0.34 for SES and Larsen, respectively. The SES is as sensitive at detecting change as the Larsen scale.