Objective: To evaluate the ability to teach scleroderma experts and young rheumatologists to perform the modified Rodnan skin score test.
Methods: Three international “teaching courses for teachers” were conducted with 6–9 experts who performed 3–9 skin score tests each. In addition, an international course for 90 young rheumatologists, in which 18 patients with systemic sclerosis (SSc) participated, was also organised. Finally, a local repeated training course for 5–9 rheumatologists was performed, in which 6–7 patients with SSc participated.
Results: When 6–9 scleroderma specialists investigated the patients, the intraclass correlation coefficient (ICC) showed “good” to “excellent” values (0.865 and 0.710, respectively). When 90 young rheumatologists were involved in one teaching course, the coefficient of variation (CV) was relatively satisfactory (35%) owing to the high number of investigators, and with a considerable within-patient SD value of 5.4.
Repeated teaching of 5–9 young rheumatologists in local courses clearly improved the consistency. The ICC increased from 0.496 to a “good” level of 0.722. The within-patient SD values for intraobserver variability ranged between 2.5 and 2.9. The intraobserver CV was about 20%.
Conclusions: This study strongly supports the need for standardisation among different centres when using skin scoring for clinical trials. The intraobserver variability and within-patient SD values can be significantly reduced by repeated teaching. For inexperienced rheumatologists, at least one repeated teaching course is needed.
- ICC, intraclass correlation coefficient
- MRSS, modified Rodnan skin score
- SSc, systemic sclerosis
Statistics from Altmetric.com
Systemic sclerosis (SSc) is characterised by tightening and thickening of the skin, and involvement of internal organs. Particularly early in the disease, more extensive skin involvement coincides with more severe internal organ manifestation(s), poor prognosis1,2 and increased disability.3,4 Derived from the original Rodnan skin score,5 several modified and simplified skin score measuring methods have been developed.2,6,7 The modified Rodnan skin score (MRSS) uses clinical palpation to estimate the skin thickness. At present, the MRSS is considered the most appropriate and reproducible technique for measuring skin involvement in SSc.2,6,8,9,10 The method is easily used and fully validated.8,9
The aims of this study were (1) to establish a method for teaching larger numbers of rheumatologists to appropriately perform the MRSS and (2) to evaluate the interobserver and intraobserver variabilities of skin scoring in a group of mixed patients with SSc with both limited and diffuse disease examined by different rheumatologists from centres all over Europe.
The fully validated8,9 modified version of the Rodnan skin thickness score5,9 was used. A total of 17 skin sites were evaluated, including the face, upper arms, forearms, dorsum of the hands, fingers, chest, abdomen, thighs, forearms and feet. The maximum total score is 51 (MRSS-51) and the grading is 0, normal skin; 1, thickened skin; 2, thickened and unable to pinch; and 3, thickened and unable to move.
Teaching course for teachers
The 1st course (Berlin, June 2004), 2nd course (Budapest, January 2005) and 3rd course (Vienna, June 2005) were organised in a similar fashion. First, the “master teacher” (DEF) explained and demonstrated the technique of MRSS to the nine examiners, and examined each patient to establish the standard. Then, the examiners performed the investigations on the patients with SSc. The proportion of patients with diffuse cutaneous SSc was 66–100%, with a range of 33–80% of early cases (disease duration <3 years).
Teaching of a large number of young rheumatologists
A total of 90 young rheumatologists participated from all over Europe at the training in Budapest (January 2005), guided by the same master trainer (DEF). The master trainer gave a further demonstration of the technique to the nine tutors, followed by a discussion of details. Next, with the assistance of a patient with scleroderma, the master trainer demonstrated the details of the technique to the 87 “trainee” rheumatologists. Meanwhile, all tutors evaluated their patients, and after that, reasons for the recorded variability were once more discussed with the master trainer. Thereafter, the 87 trainee rheumatologists were divided into nine groups, each consisting of 5–6 students. Each trainee group had two patients for practice. Following a discussion with the tutor, each trainee investigated another two patients.
Local courses to investigate the intraobserver variability
The first local teaching course (Pécs, April 2005) was organized as described above. Nine rheumatologists investigated 6 patients each a local teacher from the previous course. During the second local course (Pécs, June 2005) 5 examiners who had participated in the previous course in April, each investigated 7 SSc patients.
All calculations were made with SPSS/PC+ software. Box plot diagrams were constructed; the scores of the master trainer were treated as an external gold standard.
The coefficient of variation was calculated as described previously.7
In April 2005, in Pécs, seven examiners performed the MRSS three times on each of six patients as described above.7
Intraclass correlation coefficient
In fig 1, box plots of the three different measurements of the tutors (from Berlin, Budapest, and Vienna, respectively) are presented. In addition, the evaluations performed by European tutors were compared with the single evaluation of a US rheumatologist (DEF) with extensive experience in skin scoring, who was invited to act as the master teacher for the whole course programme.
As shown in table 1, repeated teaching reduced substantially the coefficient of variation for interobserver variability both for this international group of experts (from 50% to 35%) and for Hungarian rheumatologists in a local repeated course (54% vs 32%). With regard to the ICC, both values were good or excellent for experts (0.865 and 0.710, respectively) (for definition of good or excellent ICC, see Methods). The local repeated study indicated that both the interobserver variability and ICC can be substantially improved by repeated teaching. The within-patient SD decreased from 54% to 32%. The ICC increased from 0.496 to the good level of 0.722. The coefficient of variation for intraobserver variability was close to 20% (table 1). When time constraints were placed on the time to complete the examinations as in the second course in Budapest, results were less consistent than in Berlin and Vienna.
For the Budapest course, the box plots of MRSS indicate larger interquartile range values (fig 1B) as compared with the values of the two other courses (fig 1A,C), which suggests better consistency.
When the 90 young rheumatologists performed the skin score measurements, variability was relatively good (35%) due to high numbers, but within-patient SD remained high (5.4; table 1) and the ICC value was low. Similar results were obtained when local repeated courses were organised in Pécs for young rheumatologists (table 1, fig 2). The ICC was good or excellent in all but one case (table 1). The intraobserver within-patient variability was relatively good from the beginning and remained stable thereafter.
The aim of this EUSTAR (European Scleroderma Trials and Research group) project was to unify European medical practice in skin score measuring. Overall, the results on MRSS-51 measurements were comparable to two previous independent studies, where the mean (within-patient SD) was 18.3 (4.6) and 17.7 (4.6), respectively.8,9 Our three investigations for teachers showed similar results (table 1), thereby confirming both that MRSS is an appropriate tool for skin score measuring and that it is possible to teach a large number of rheumatologists the appropriate skin score measuring technique in a short time.
A previous study had demonstrated that the interobserver variability of the MRSS-51 could be reduced by approximately 50%.7 The results of our repeat courses likewise demonstrated that the coefficient of variation for interobserver variability can be substantially reduced by repeated teaching (table 1). Although the ICC increased from 0.50 to the good level of 0.72 in the limited local experience with less experienced rheumatologists in Pécs, the ICC of the experts was very good from the beginning, but did not increase (table 1). This underlines the importance of a second training process for less experienced rheumatologists in particular.
It is encouraging that the coefficient of variation for intraobserver variability was around 20%, similar to that found for joint swelling/tenderness counts in rheumatoid arthritis (table 2).14 Although a low ICC was observed even during the second repeated teaching course, the overall appropriateness of the two repeated courses appeared to be satisfactory (table 1). These findings coincide with previous experience.6
When, for the first EUSTAR/EULAR (European League Against Rheumatism) course, almost 100 young rheumatologists were trained, the value of the within-patient SD was relatively good (5.4) with low ICC value, mainly due to the high number of investigators. Interobserver variability after this single teaching course was still considerable, even though experienced experts trained the students. However, the other courses suggest that, following repeated training, interobserver variability could be substantially decreased. After completion, an overall value of 5.5 of within-patient SD for interobserver variability as well as good ICC were achieved in mixed patient populations. This observation is also in agreement with previous experience.7
Our study has certain limitations. Even repeated sessions of the experts did not bring them any closer to the main tutor with time (table 1). This may be a limitation of the teaching method, or it could reflect a systematic bias of the principal teacher or a systematic bias of the other experts. The definition of a gold standard remains a critical point in such teaching exercises.
On the other hand, this exercise demonstrates a significant strength—the feasibility of training a large group of relatively inexperienced rheumatologists and the feasibility of a follow-up which completes the training in an efficient and straightforward manner.
In conclusion, in view of a significant amount of observer variability, teaching courses should be repeated at least twice for teaching inexperienced rheumatologists.
Besides the authors of the paper, the following colleagues participated in the teaching course for teachers: Phil Clements (US), Christopher Denton (UK), Ivan Foeldvari, Ulf Mueller-Ladner (Germany), Alan Tyndall (Switzerland), Gabriele Valentini (Italy) and Frederick Wigley (US).
The participants of the local repeated courses were: Valéria Dudics, Gábor Kumánovics, Csaba G Kiss, Zoltán Nagy, Nóra Nusser, Gábor Sütő, Sándor Szántó, Zoltán Szekanecz, Gabriella Szűcs, Sándor Szántó and Cecília Varjú (Hungary).
Participants of the European training course for young rheumatologists were: Stummvoll G (Austria); Culo MI, Marasovic Krstulovic D, Novak S, Radic M, Soldo JD (Croatia); Sorensen IJ, Sondergaard K, Strauss GI, Ullman S (Denmark); Tuvik P (Estonia), Assous N, Ilie D, Launay D (France); Clarenbach R, Foeldvari I, Gensch K, Grundt B, Hanitsch LG, Himsel A, Müller A, Olski TM, Seidel M, Sunderkoetter C, Süβ A, Walzer K (Germany); Kampakis G, Mamoulaki M, Siakka P (Greece); Lee Ka Wing G (Hong Kong); Dudics V, Orbán I, Szűcs G, Varjú C (Hungary); Cavazzana I, Chialà A, Comina D, Frassi M, Guiducci S, Iannone F, Lo Monaco A, Maglione W, Miniati I, Pieropan S, Ruocco L, Tiso F (Italy); Jae-Bum J (Korea); Pileckyte M (Lithuania); Coleiro B (Malta); Grandaunet BH, Midtvedt Ø (Norway); Kotulska A, Kowal-Bielecka O, Krasowska DM (Poland); Figueira R, Cordeiro F, Oliveira R, Henriques MJ, Pinto SP, Resende C, Sequeira G (Portugal); Damjanovska Rajcevska L (Macedonia); Dragomir D, Isa ML, Micu M, Nicola M, Rednic S (Romania); Nevskaya T (Russia); Stamenkovic B, Ostojic P, Zlatanovic M (Serbia); Lukácová O (Slovak Republic); Sipek DA (Slovenia); Banegil Espinosa I, de la Peña G, Lefebre P, Joven B, Nuño Nuño L, Ortega de la O MdC, Rodríguez Rubio S (Spain); Tikly M (South Africa); Chizzolini C, Oehri M, Ribi C, von Muehlenen I (Switzerland); Vonk M (The Netherlands); Dass S, Prabu A, Solanki K (UK).
Funding: This work was supported by a EULAR educational grant provided for EUSTAR.
Competing interests: None declared.
EUSTAR is the EULAR Scleroderma Trials and Research group. Besides the authors, further participants in the skin score measurements are listed in the Acknowledgements.
Published Online First 18 January 2007
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.