Interobserver reliability of rheumatologists performing musculoskeletal ultrasonography: results from a EULAR "Train the Trainers" course

Alexander K Scheel ^1*, Wolfgang A Schmidt ², Kay-Geert A Hermann ³, George A Bruyn ⁴, Maria-Antonietta D'Agostino ⁵, Walter Grassi ⁶, Annamaria Iagnocco ⁷, Juhani M Koski ⁸, Klaus P Machold ⁹, Esperanza Naredo ¹⁰, Horst Sattler ¹¹, Nanno Swen ¹², Marcin Szkudlarek ¹³, Richard J Wakefield ¹⁴, Hans R Ziswiler ¹⁵, Daniel Pasewaldt ¹⁶, Carola Werner ¹⁷ and Marina Backhaus ¹⁸

¹ Department of Nephrology and Rheumatology, Georg-August-University Göttingen, Germany
² Medical Centre for Rheumatology Berlin-Buch, Berlin, Germany
³ Radiology, Charité University Hospital, Berlin, Germany
⁴ Rheumatology, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands
⁵ Rheumatology, Ambroise Paré Hospital, UVSQ University, Boulogne Billancourt, France
⁶ Rheumatology, Universita degli Studi di Ancona, Ospedale A. Murri, Jesi (Ancona), Italy
⁷ Rheumatology, University La Sapienza, Rome, Italy
⁸ Rheumatology, Mikkeli Central Hospital, Mikkeli, Finland
⁹ Rheumatology, Vienna General Hospital, University of Vienna, Austria
¹⁰ Rheumatology, the Research Unit, and the Epidemiology Unit, Severo Ochoa Hospital, Madrid, Spain
¹¹ Rheumatology, Parkklinik, Bad Durkheim, Germany
¹² Rheumatology, Medisch Centrum Alkmaar, Alkmaar, The Netherlands
¹³ Rheumatology, University of Copenhagen, Hvidovre Hospital, Denmark
¹⁴ Academic Department of Musculoskeletal Medicine, Leeds General Infirmary, United Kingdom
¹⁵ Rheumatology and Clinical Immunolog, Inselspital, Bern, Switzerland
¹⁶ Nephrology and Rheumatology, Georg-August-University Göttingen, Germany
¹⁷ Medical Statistics, Georg-August-University Göttingen, Göttingen, Germany
¹⁸ Rheumatology and Clinical Immunology, Charité University Hospital, Berlin, Germany

^* To whom correspondence should be addressed. E-mail: ascheel{at}gwdg.de.

Accepted 15 December 2004

Abstract

Objective: To evaluate the inter-observer reliability between 14 experts in musculoskeletal ultrasonography (US) and to determine the overall agreement of US results compared to magnetic resonance imaging (MRI) which served as imaging gold standard.

Methods: The clinically dominant joint regions (shoulder, knee, ankle/toe, wrist/finger) of 4 patients with inflammatory rheumatic diseases were ultrasonographically examined by the 14 experts. US results were compared to MRI. The overall agreements, sensitivities, specificities and inter-observer reliabilities were assessed.

Results: Taking an agreement in US examination of 10 out of 14 experts into account, the overall kappa for all examined joints was 0.76. Calculations with regard to each joint region revealed high kappa values for the knee (1), moderate values for the shoulder (0.76) and hand/finger (0.59), and low agreement for ankle/toe joints (0.28). Kappa values regarding bone lesions, bursitis and tendon tears were high (kappa=1). Compared to MRI, we found a relatively good agreement for most US findings for the shoulder (overall agreement 81%, sensitivity 76%, specificity 89%) and knee joint (overall agreement 88%, sensitivity 91%, specificity 88%). Sensitivities were lower for wrist/finger (overall agreement 73%, sensitivity 66%, specificity 88%) and ankle/toe joints (overall agreement 82%, sensitivity 61%, specificity 92%).

Conclusion: This is the first study that investigates inter-observer reliability for a large number of physician sonographers. Inter-observer reliabilities, sensitivities, and specificities in comparison with MRI were moderate to good. Further standardization of US scanning techniques and definitions of different pathologic US lesions are necessary to increase the inter-observer agreement in musculoskeletal US.

Keywords: interobserver reliability, magnetic resonance imaging, ultrasonography

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

• Filippucci, E, Aydin, S Z., Karadag, O, Salaffi, F, Gutierrez, M, Direskeneli, H, Grassi, W (2009). Reliability of high-resolution ultrasonography in the assessment of Achilles tendon enthesopathy in seronegative spondyloarthropathies. Ann Rheum Dis 68: 1850-1855 [Abstract] [Full Text]  
• Funck-Brentano, T., Etchepare, F., Joulin, S. J., Gandjbakch, F., Pensec, V. D., Cyteval, C., Miquel, A., Benhamou, M., Banal, F., Le Loet, X., Cantagrel, A., Bourgeois, P., Fautrel, B. (2009). Benefits of ultrasonography in the management of early arthritis: a cross-sectional study of baseline data from the ESPOIR cohort. Rheumatology (Oxford) 48: 1515-1519 [Abstract] [Full Text]  
• Scire, C. A., Montecucco, C., Codullo, V., Epis, O., Todoerti, M., Caporali, R. (2009). Ultrasonographic evaluation of joint involvement in early rheumatoid arthritis in clinical remission: power Doppler signal predicts short-term relapse. Rheumatology (Oxford) 48: 1092-1097 [Abstract] [Full Text]  
• D'AGOSTINO, M.-A., CONAGHAN, P. G., NAREDO, E., AEGERTER, P., IAGNOCCO, A., FREESTON, J. E., FILIPPUCCI, E., MOLLER, I., PINEDA, C., JOSHUA, F., BACKHAUS, M., KEEN, H. I., KAELEY, G., ZISWEILER, H.-R., SCHMIDT, W. A., BALINT, P. V., BRUYN, G. A.W., JOUSSE-JOULIN, S., KANE, D., MOLLER, I., SZKUDLAREK, M., TERSLEV, L., WAKEFIELD, R. J. (2009). The OMERACT Ultrasound Task Force -- Advances and Priorities. The Journal of Rheumatology 36: 1829-1832 [Abstract] [Full Text]  
• Bruyn, G A W, Naredo, E, Moller, I, Moragues, C, Garrido, J, de Bock, G H, d'Agostino, M-A, Filippucci, E, Iagnocco, A, Backhaus, M, Swen, W A A, Balint, P, Pineda, C, Milutinovic, S, Kane, D, Kaeley, G, Narvaez, F J, Wakefield, R J, Narvaez, J A, de Augustin, J, Schmidt, W A (2009). Reliability of ultrasonography in detecting shoulder disease in patients with rheumatoid arthritis. Ann Rheum Dis 68: 357-361 [Abstract] [Full Text]  
• Wakefield, R J, Freeston, J E, O'Connor, P, Reay, N, Budgen, A, Hensor, E M A, Helliwell, P S, Emery, P, Woodburn, J (2008). The optimal assessment of the rheumatoid arthritis hindfoot: a comparative study of clinical examination, ultrasound and high field MRI. Ann Rheum Dis 67: 1678-1682 [Abstract] [Full Text]  
• Albrecht, K., Grob, K., Lange, U., Muller-Ladner, U., Strunk, J. (2008). Reliability of different Doppler ultrasound quantification methods and devices in the assessment of therapeutic response in arthritis. Rheumatology (Oxford) 47: 1521-1526 [Abstract] [Full Text]  
• Naredo, E, Bijlsma, J W J, Conaghan, P G, Acebes, C, Balint, P, Berner-Hammer, H, Bruyn, G A W, Collado, P, D'Agostino, M A, de Agustin, J J, de Miguel, E, Filippucci, E, Grassi, W, Iagnocco, A, Kane, D, Koski, J M, Manger, B, Mayordomo, L, Moller, I, Moragues, C, Rejon, E, Szkudlarek, M, Terslev, L, Uson, J, Wakefield, R J, Schmidt, W A (2008). Recommendations for the content and conduct of European League Against Rheumatism (EULAR) musculoskeletal ultrasound courses. Ann Rheum Dis 67: 1017-1022 [Abstract] [Full Text]  
• Violante, F S, Bonfiglioli, R, Graziosi, F, Caso, A, Isolani, L, Fiorentini, C, Mattioli, S (2007). Potential of ultrasonography for epidemiological study of work-related wrist tenosynovitis. Occup. Environ. Med. 64: 82-86 [Abstract] [Full Text]  
• Meenagh, G., Filippucci, E., Kane, D., Taggart, A., Grassi, W. (2007). Ultrasonography in rheumatology: developing its potential in clinical practice and research. Rheumatology (Oxford) 46: 3-5 [Full Text]  
• Koski, J M, Saarakkala, S, Helle, M, Hakulinen, U, Heikkinen, J O, Hermunen, H, Balint, P, Bruyn, G A, Filippucci, E, Grassi, W, Iagnocco, A, Luosujarvi, R, Manger, B, De Miguel, E, Naredo, E, Scheel, A K, Schmidt, W A, Soini, I, Szkudlarek, M, Terslev, L, Uson, J, Vuoristo, S, Ziswiler, H R (2006). Assessing the intra- and inter-reader reliability of dynamic ultrasound images in power Doppler ultrasonography. Ann Rheum Dis 65: 1658-1660 [Abstract] [Full Text]  
• Scheel, A K, Hermann, K-G A, Ohrndorf, S, Werner, C, Schirmer, C, Detert, J, Bollow, M, Hamm, B, Muller, G A, Burmester, G R, Backhaus, M (2006). Prospective 7 year follow up imaging study comparing radiography, ultrasonography, and magnetic resonance imaging in rheumatoid arthritis finger joints. Ann Rheum Dis 65: 595-600 [Abstract] [Full Text]  
• Naredo, E, Moller, I, Moragues, C, de Agustin, J J, Scheel, A K, Grassi, W, de Miguel, E, Backhaus, M, Balint, P, Bruyn, G A W, D'Agostino, M A, Filippucci, E, Iagnocco, A, Kane, D, Koski, J M, Mayordomo, L, Schmidt, W A, Swen, W A A, Szkudlarek, M, Terslev, L, Torp-Pedersen, S, Uson, J, Wakefield, R J, Werner, C, EULAR Working Group for Musculoskeletal Ultrasound, (2006). Interobserver reliability in musculoskeletal ultrasonography: results from a "Teach the Teachers" rheumatologist course. Ann Rheum Dis 65: 14-19 [Abstract] [Full Text]