Objective Investigation of the reliability of the qualitative and semiquantitative scoring of nailfold videocapillaroscopy (NVC) assessment between two raters in a systemic sclerosis (SSc) cohort.
Methods Two raters from different centres blindly assessed the NVC images of 71 consecutive patients with SSc qualitatively as belonging to the scleroderma spectrum (SDS) category (‘early’, ‘active’, ‘late’ scleroderma pattern or ‘scleroderma-like’ pattern) or to the ‘normal’ category and semiquantitatively by calculating the mean score for capillary loss, giant capillaries, microhaemorrhages and capillary ramifications. Inter-rater/intrarater agreement was assessed by calculation of the proportion of agreement and by κ coefficients. Rater agreement of mean score values of hallmark parameters was assessed by intraclass correlation coefficients.
Results The inter-rater/intrarater proportion of agreement to qualitatively assess an image as belonging to the SDS category or not was 90% and 96%, whereas the agreement to distinguish between only ‘early’, ‘active’ and ‘late’ scleroderma NVC patterns was 62% and 81%. The agreement of the semiquantitative scoring, as assessed by intraclass correlation coefficient, was 0.96 and 0.95 for capillary loss, 0.84 and 0.95 for giant capillaries, 0.90 and 0.95 for microhaemorrhages and 0.64 and 0.95 for capillary ramifications.
Conclusions This is the first study to demonstrate reliability of the qualitative and semiquantitative NVC assessment in an SSc cohort between raters at different centres. Reliability of NVC assessment is essential for use of this tool in multicentre SSc trials.
Statistics from Altmetric.com
Systemic sclerosis (SSc) is a heterogeneous autoimmune disorder characterised by vasculopathy as well as progressive fibrosis of the skin and internal organs. When patients meet the American College of Rheumatology (ACR) criteria, they already have ‘clinically recognisable’ scleroderma (skin involvement) and can be classified into limited cutaneous SSc (LcSSc) and diffuse cutaneous SSc (DcSSc) subsets.1 2 To date, no treatment has been proved in a randomised controlled trial to halt the natural progression of the ‘clinically recognisable’ disease. Consequently, efforts are being made to study the disease ‘early’, before ‘clinically recognisable’ disease and irreversible damage have occurred.3 In 2001, LeRoy and Medsger proposed criteria for the ‘early’ diagnosis of SSc, with Raynaud's phenomenon as the single major criterion. These criteria incorporate SSc-specific autoimmune antibodies and microvascular techniques, more specifically the ‘scleroderma-type’ changes on capillaroscopy.4 Like the ACR criteria, the LeRoy and Medsger criteria allow patients with ‘clinically recognisable’ SSc (skin involvement), LcSSc and DcSSc to be included. In addition, however, they also allow patients to be included ‘earlier’, before skin involvement has occurred. This third group of patients is classified as limited systemic sclerosis (LSSc), and is characterised by the presence of Raynaud's phenomenon plus the presence of SSc-specific autoimmune antibodies and/or the presence of typical ‘scleroderma-type’ abnormalities. The combination of Raynaud's phenomenon with typical ‘scleroderma-type’ abnormalities as a criterion for ‘early’ diagnosis reflects the fact that Raynaud's phenomenon often precedes the diagnosis of ‘clinically recognisable’ SSc, and the fact that the typical ‘scleroderma-type’ abnormalities are specific in distinguishing scleroderma spectrum (SDS) (SSc, mixed connective tissue disease (CTD), dermatomyositis, undifferentiated CTD) from healthy controls or other CTDs. This diagnostic potential of capillaroscopy to distinguish SDS from other CTDs has already been established in several studies.5,–,10 Whether the capillaroscopic patterns of the SDS themselves are distinguishable is a matter of subjective opinion between experts. As yet, no large-scale blinded studies have been performed to resolve this question indisputably.11 12
The typical ‘scleroderma-type’ nailfold capillary changes of the SDS were initially described by Maricq using the wide-field microscopy technique.13 14 Recently, Cutolo et al used a newer technique, nailfold videocapillaroscopy (NVC), to qualitatively assess the nailfolds of a cohort of patients fulfilling the ACR criteria for SSc. This technique has the advantage that it consists of a handheld device which can be moved along the nailfold and which allows images of patients with SSc with severe flexion contractures to be taken. Cutolo et al classified the patterns of patients with SSc into ‘early’, ‘active’ and ‘late’ scleroderma patterns. To refer to the patterns of other SDS diseases, they used the terminology ‘scleroderma-like’.15 In addition to the qualitative assessment, Cutolo et al also proposed the semiquantitative scoring of giant capillaries, capillary loss, microhaemorrhages and ramifications.16,–,18 Both the qualitative and semiquantitative NVC assessments are currently being practised in examining clinical associations or as biomarkers.18 19 In order to use these assessments to study ‘early preclinical’ and ‘clinically recognisable’ disease uniformly in a number of centres, it is essential to have intrarater and inter-rater agreement between raters at different centres. This study investigates the reliability of the qualitative assessment and semiquantitative NVC scoring in an SSc cohort.
Seventy-one consecutive patients (mean age 52±14 years) presenting to the Scleroderma Clinic of Ghent University Hospital and fulfilling the criteria for ‘early’ SSc according to LeRoy and Medsger were studied.4 Sixty-six were included on the basis of the capillaroscopy criterion as ascertained by MC. The other five patients all had SSc-specific antibodies. The patients were subsequently divided into LSSc (no skin involvement), LcSSc or DcSSc (skin involvement) according to LeRoy et al.1 4
Collection and blinding of nailfold videocapillaroscopic images
NVC was performed in all 71 patients, as described previously.18 Briefly, the nailfold of the second, third, fourth and fifth fingers was examined bilaterally in each patient using an optical probe videocapillaroscope equipped with a ×200 magnification contact lens and connected to image analysis software (Videocap, DS MediGroup, Milan, Italy). Four consecutive fields extending over 1 mm in the middle of the nailfold were studied per finger. Two readers evaluated the same images in two different skilled centres, the Videocapillaroscopy Unit at the Academic Clinical Unit of Rheumatology at the University of Genova and at Ghent University Hospital. The images were made anonymous before being assessed by rater 1 (CP, Genova) and rater 2 (VS, Ghent). To assess inter-rater reliability, rater 1 and rater 2 both read and scored all images once. To assess intrarater reliability, rater 2 read and scored all images twice. The minimum time interval between the two evaluations was 3 months.
Qualitative NVC assessment in the SSc cohort
The following capillaroscopic definitions were used for the qualitative assessment. The ‘early’ NVC scleroderma pattern is defined as the combination of few enlarged/giant capillaries, few capillary microhaemorrhages, a relatively well-preserved capillary distribution and no evident loss of capillaries (figure 1A). The ‘active’ NVC scleroderma pattern involves frequent giant capillaries, frequent capillary microhaemorrhages, moderate loss of capillaries, mild disorganisation of the capillary architecture and absent or mild ramified capillaries (figure 1B). The ‘late’ scleroderma NVC pattern includes irregular enlargement of the capillaries, few or absent giant capillaries and microhaemorrhages, severe loss of capillaries with large avascular areas, disorganisation of the normal capillary array and ramified/bushy capillaries (figure 1C). The ‘scleroderma-like’ pattern is defined as a capillary pattern showing mixed microvascular markers of the scleroderma capillary patterns, but not fully fitting the definition for the single ‘early’, ‘active’ or ‘late’ scleroderma pattern.15,–,17
The SDS category is defined as having either an ‘early’, ‘active’ or ‘late’ scleroderma pattern or a ‘scleroderma-like’ pattern. The ‘normal’ category is defined as a regular distribution of the capillaries without capillary loss and a morphology without specific changes or non-specific changes (figure 1D).20
Semiquantitative NVC assessment in the SSc cohort
Semiquantitative scoring was performed as described previously.18 The following hallmark parameters were counted over 32 fields (4 fields per finger, fingers 2–5 of each hand, 8 fingers): total amount of capillaries (normal: 9/mm), giant capillaries (homogeneously enlarged loops with a diameter >50 μm), microhaemorrhages (dark mass due to haemosiderin deposit) and ramifications (branching, bushy or coiled capillaries, often originating from a single normal-sized capillary). The recently described semiquantitative rating scale (0 = no changes; 1 = <33% of capillary alterations/reduction; 2 = 33–66% of capillary alterations/reduction; 3 = >66% of capillary alterations/reduction) was adopted.17 18 The mean score value for each capillaroscopic parameter was calculated.
Antibodies were detected by indirect immunofluorescence on Hep-2000 cells at a 1:40 serum dilution according to the manufacturer's instructions (Immunoconcepts, Sacramento, California, USA). The SSc-specific antibodies (anticentromere and antitopoisomerase) were detected by the INNO-LIA ANA Update (Innogenetics, Ghent, Belgium). Anti-RNA polymerase III was detected by ELISA (MBL International, Woburn, Massachusetts, USA) and anti-PM-Scl by immunodiffusion (Immunoconcepts).
Inter-rater and intrarater agreement of dichotomous data was assessed by calculation of the proportion of agreement and by κ coefficients. When the proportion of agreement was relatively higher than the κ coefficient, explanation was sought by further investigating the separate indexes of agreement: positive and negative agreement. Rater agreement of mean score values of capillary loss, giant capillaries, microhaemorrhages and ramifications was assessed by intraclass correlation coefficients. For normal distributions, means and SDs are reported; otherwise, medians and ranges are reported.
Characteristics of patients
The study group consisted of 71 patients (52 women, 19 men) who complied with the LeRoy and Medsger criteria (see table S1a in online supplement). The capillaroscopy criterion was met by 66 patients (93%) as ascertained by MC. The five patients (7%) who did not meet the capillaroscopic criterion all had SSc-specific antibodies. Four were anticentromere positive and one was anti-RNA polymerase III positive. Forty-nine patients (69%) met the SSc-specific autoantibodies criterion (see table S1b in online supplement). Twenty-two patients (31%) had no skin involvement (LSSc) and 49 (69%) had skin involvement (35 LcSSc and 14 DcSSc).
Reliability of qualitative NVC assessment in patients with SSc
The qualitative assessment of the NVC images of the 71 patients is shown in table 1. The inter-rater proportion of agreement to assess an image as belonging to the SDS category or to the ‘normal’ category was 90% (table 2; rater 2, episode A) with a κ coefficient of 0.6. The proportion of positive agreement was 94% while the proportion of negative agreement was 63%. The intrarater proportion of agreement to assess an image as belonging to the SDS category or not was 96% (table 2; rater 2, episode B) with a κ coefficient of 0.8. The inter-rater and intrarater proportions of agreement to distinguish between the ‘early’, ‘active’ and ‘late’ scleroderma NVC patterns themselves were 62% (table 3; rater 2, episode A) and 81% (table 3; rater 2, episode B) with a κ coefficient of 0.5 and 0.7.
Reliability of the semiquantitative NVC assessment in patients with SSc
The reliability of the semiquantitative NVC assessment is shown in table 4. The inter-rater/intrarater agreement as assessed by intraclass correlation coefficient was 0.96/0.95 for capillary loss, 0.84/0.95 for giant capillaries, 0.90/0.95 for microhaemorrhages and 0.64/0.95 for capillary ramifications.
This is the first international two-centre study (Genova/Ghent) to investigate the reliability of qualitative and semiquantitative NVC assessment in a well-defined SSc cohort.
The results of this study show that there is a high inter-rater proportion of overall agreement (90%) and a high intrarater proportion of overall agreement (96%) in qualitatively assessing whether an image belongs to the ‘SDS’ category or to the ‘normal’ category. When evaluating the reliability in distinguishing only between the ‘early’, ‘active’ and ‘late’ scleroderma patterns, the inter-rater agreement is substantial (62%). The intrarater overall proportion of agreement in distinguishing between the scleroderma patterns themselves was high (81%).
Concerning the reliability of the semiquantitative scoring, the inter-rater agreement is excellent for loss of capillaries, giant capillaries, microhaemorrhages and fair for capillary ramifications. The intrarater agreement is excellent for all mentioned hallmark parameters.
Several conclusions can be drawn from this study. First, assessment of a capillaroscopic image as an ‘early’, ‘active’ or ‘late’ scleroderma pattern in an SSc cohort is reliable. This assessment may therefore be reliably used in multicentre trials. In this way, the recently published change of ‘late’ to ‘active’ capillaroscopic scleroderma pattern after stem cell transplantation and from a severe scleroderma pattern to almost ‘normal’ in mixed CTD is noteworthy.21 22 Second, the inter-rater and intrarater agreement of semiquantitative assessment of capillary loss, giant capillaries and haemorrhages is high. Subsequently, these may have a role as a biomarker in therapeutic trials or in the investigation of prognostic associations.
Two points may be made concerning this study. First, when investigating the inter-rater reliability of qualitatively assessing whether a capillaroscopic image belonged to the ‘SDS’ category or to the ‘normal’ category, the value (90%=‘high’) when tested with the overall proportion of agreement is higher than the value (0.6=‘good’) when tested with the κ statistic. Subsequent statistics to investigate and explain this difference in magnitude of value were performed. Next to the ‘overall’ proportion of agreement, the separate indexes of proportionate agreement in the observers' positive (positive proportion of agreement) and negative decisions (negative proportion of agreement) was investigated.23 The proportion of agreement in negative decisions between raters was substantially lower (63%) than the proportion of positive agreement (94%), and explains the relatively ‘lower’ value of κ than the value of ‘overall’ proportion of agreement. Logical explanations for this lower proportion of agreement in negative decisions were made clear by qualitative re-assessment of the images in which the raters did not agree. As table 2 (rater 2, episode A) shows, there were five cases in which rater 1 assessed an image as not belonging to the SDS category while rater 2 did. Of these, four of the five ‘disagreements’ were caused by ‘information’ bias. These four images were characterised by the combination of ramifications and capillary loss only. Ramifications themselves may occur in healthy subjects and in CTDs ‘other than’ SDS.20 24 Logically, as rater 1 was not part of the study design, she marked these changes as ‘aspecific’ findings (classified in the ‘normal’ category) while rater 2, who was informed that an SSc cohort was being investigated, consequently marked these cases either as ‘late’ or ‘scleroderma-like’ (classified in the ‘SDS’ category). The other case as well as the two cases scored by rater 2 as ‘not belonging to the SDS category’ and by rater 1 as ‘belonging to the SDS category’ were likewise cases that were interpreted by one rater as ‘aspecific changes’ and by the other as ‘active’ scleroderma pattern. These cases all had pathological haemorrhages and enlarged capillaries. The rater who interpreted the capillaries qualitatively as ‘enlarged’ classified the image as ‘aspecific finding’ (classified in the ‘normal’ category) as the combination of pathological haemorrhages can appear in CTDs ‘other than’ SDS diseases, while the rater who interpreted them as giant capillaries classified them as ‘active’ scleroderma pattern (classified in the ‘SDS’ category). These cases may be interpreted as true disagreements and marks the borders of qualitative assessment in which the interpretation of a capillary to be an ‘enlarged’ one or a ‘giant’ is made ‘on sight’ and not by quantitative measurement.
Second, patients were included according to the LeRoy and Medsger criteria. The rationale behind this is that future multicentre studies will aim to include patients ‘earlier’ than when they present with ‘clinically recognisable’ disease. The authors decided to use the LeRoy and Medsger criteria to select an ‘early’ population as, at present, these criteria are the only ones that are validated in ‘early’ SSc.25 Consequently, SDS diseases other than ‘clinically recognisable’ SSc may be included in this cohort. To acknowledge this, the study design allowed both raters to rate qualitatively one of the scleroderma patterns as ‘early’, ‘active’ or ‘late’ and also as ‘scleroderma-like’ pattern. This is logical when investigating the reliability of raters in assessing whether an image belongs to the SDS category or to the ‘normal’ category. But it also has one drawback—specifically (see table 3), when narrowing down the investigation to the reliability of ‘only’ the ‘early’, ‘active’ and ‘late’ scleroderma patterns, the ‘scleroderma-like’ pattern is classified into the ‘no SSc-specific’ category with ‘normal’. This is illogical from a clinical point of view, but is the only way to evaluate the reliability of only the ‘early’, ‘active’ and ‘late’ scleroderma pattern, given the current study design. A possible option to overcome this drawback in future studies is only to allow raters in multicentre trials to rate as ‘early’, ‘active’ or ‘late’ scleroderma pattern. The alternative is that this terminology is described on a ‘clinically recognisable’ SSc cohort. Consensus will need to be sought in the setting up of multicentre studies, balancing the pros and cons of both options.
In short, in order to unravel the interaction between the different components of SSc (microvascular complications, SSc-specific antibodies, internal organ involvement, disease severity and disease activity), large multicentre studies including both ‘early’ and ‘clinically recognisable’ SSc are warranted. Capillaroscopy is a pivotal tool in SSc as it is a cornerstone for ‘early’ diagnosis and it may have opportunities to be used as a biomarker. To be able to be uniformily used in multicentre studies, the reliability of NVC assessments first needs to be demonstrated. This is the first study to demonstrate the reliability of the qualitative and semiquantitative NVC assessment in an SSc cohort.
The authors acknowledge Melissa De Decker for her unconditional daily secretarial support; their colleagues for sending their patients to the Scleroderma Clinic; head nurse Annemie Herssens and nurse Karlien Claes for their continuous dedicated care of the patients; and their senior professors for making this collaboration possible.
VS, CP and FDK contributed equally to this work.
Funding JTVP is supported by a research grant from the Fund for Scientific Research-Flanders. SD is supported by a grant from the ‘Rotary – National Association to support Disabled People’.
Competing interests None.
Ethics approval This study was conducted with the approval of the ethical committee of Ghent University. All patients signed informed consent.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.