Article Text

Download PDFPDF

First external validation of sensitivity and specificity of the European League Against Rheumatism (EULAR)/American College of Rheumatology (ACR) classification criteria for idiopathic inflammatory myopathies with a Japanese cohort
  1. Masatoshi Jinnin1,
  2. Akiko Ohta2,
  3. Shoichiro Ishihara3,
  4. Hirofumi Amano4,
  5. Tatsuya Atsumi5,
  6. Manabu Fujimoto6,
  7. Takashi Kanda7,
  8. Yasushi Kawaguchi8,
  9. Atsushi Kawakami9,
  10. Akio Mimori10,
  11. Tsuneyo Mimori11,
  12. Toshihide Mimura12,
  13. Yoshinao Muro13,
  14. Hajime Sano14,
  15. Jun Shimizu15,
  16. Tsutomu Takeuchi16,
  17. Yoshiya Tanaka17,
  18. Kazuhiko Yamamoto18,
  19. Takayuki Sumida19,
  20. Hitoshi Kohsaka20
  21. The Research Team for Autoimmune Diseases, The Research Program for Intractable Disease of the Ministry of Health, Labor and Welfare
  1. 1 Department of Dermatology and Plastic Surgery, Faculty of Life Sciences, Kumamoto University, Kumamoto, Japan
  2. 2 Division of Public Health, Department of Social Medicine, Faculty of Medicine, Saitama Medical University, Saitama, Japan
  3. 3 Department of Neurology and Neurological Science, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan
  4. 4 Department of Internal Medicine and Rheumatology, Juntendo University School of Medicine, Tokyo, Japan
  5. 5 Department of Rheumatology, Endocrinology and Nephrology, Faculty of Medicine and Graduate School of Medicine, Hokkaido University, Hokkaido, Japan
  6. 6 Department of Dermatology, University of Tsukuba, Ibaraki, Japan
  7. 7 Department of Neurology and Clinical Neuroscience, Yamaguchi University Graduate School of Medicine, Yamaguchi, Japan
  8. 8 Institute of Rheumatology, Tokyo Women's Medical University, Tokyo, Japan
  9. 9 Department of Immunology and Rheumatology, Unit of Advanced Preventive Medical Sciences, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan
  10. 10 Division of Rheumatic Diseases, National Center for Global Health and Medicine, Tokyo, Japan
  11. 11 Department of Rheumatology and Clinical Immunology, Graduate School of Medicine, Kyoto University, Kyoto, Japan
  12. 12 Department of Rheumatology and Applied Immunology, Faculty of Medicine, Saitama Medical University, Saitama, Japan
  13. 13 Department of Dermatology, Nagoya University Graduate School of Medicine, Aichi, Japan
  14. 14 Department of Rheumatology, Hyogo College of Medicine, Hyogo, Japan
  15. 15 Department of Neurology, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
  16. 16 Division of Rheumatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan
  17. 17 The First Department of Internal Medicine, School of Medicine, University of Occupational and Environmental Health, Fukuoka, Japan
  18. 18 Department of Allergy and Rheumatology, The University of Tokyo Hospital, Tokyo, Japan
  19. 19 Department of Internal Medicine, Faculty of Medicine, University of Tsukuba, Ibaraki, Japan
  20. 20 Department of Rheumatology, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan
  1. Correspondence to Dr Hitoshi Kohsaka, Tokyo, Japan; kohsaka-tmd{at}umin.ac.jp

Abstract

Objective To externally validate the performance of the new European League Against Rheumatism (EULAR)/American College of Rheumatology (ACR) classification criteria set for idiopathic inflammatory myopathies (IIM) with a Japanese cohort.

Methods This study included 420 IIM and 402 non-IIM cases. Probability of having IIM in each patient was calculated using the collected data set. The cut-off probability was set at 55%, as recommended by EULAR/ACR. Patients classified as IIM by the criteria were further subclassified with classification trees.

Results When the probability cut-off was set at 55%, the sensitivity/specificity of the new criteria to diagnose IIM were 89.3%/91.0% in the total cohort, 88.1%/95.1% without muscle biopsy data and 90.4%/65.5% with biopsy data. The cohort included 12 overlap syndrome patients with biopsy data, who were included as non-IIM cases in accordance with traditional Japanese methods. When they were included in the IIM cases, the specificity in patients with biopsy increased to 74.4%. The sensitivity/specificity of the new criteria to diagnose polymyositis/dermatomyositis (PM/DM) plus juvenile and amyopathic DM in the Japanese cohort was 87.4%/92.4%, which were greater than those of the Tanimoto’s criteria revised to enable classification of amyopathic DM (ADM) (71.2%/87.8%) and were comparable with those of Bohan & Peter’s criteria to diagnose those diseases except for ADM (88.4%/88.3%).

Conclusions Our study externally validated high specificity of the new criteria for the first time, although with several limitations, including low percentage of child patients. The new criteria have higher sensitivity and/or specificity in classification of PM/DM than the previously reported criteria, demonstrating its usefulness for interethnic patients.

  • dermatomyositis
  • polymyositis
  • autoimmune diseases
View Full Text

Statistics from Altmetric.com

Key messages

What is already known about this subject?

  • New classification criteria for idiopathic inflammatory myopathies were proposed on the basis of the data analyses by the International Myositis Classification Criteria Project. Examined for sensitivity validation with external cohorts, they were recently approved by European League Against Rheumatism and American College of Rheumatology.

What does this study add?

  • Our study externally validated high specificity of the new criteria for the first time.

How might this impact on clinical practice or future developments?

  • The new criteria have higher sensitivity and/or specificity in classification of polymyositis/dermatomyositis than the previously reported criteria, demonstrating its usefulness for inter-ethnic patients. Setting probability cut-off at 55% in the new criteria together with use of the classification tree was acceptable for the Japanese cohort.

Introduction

Idiopathic inflammatory myopathies (IIM) are heterogeneous disorders characterised by muscle weakness and muscle inflammation and include polymyositis (PM), dermatomyositis (DM) and inclusion body myositis (IBM).1 Immune-mediated necrotising myopathy (IMNM) could be an independent disease entity in IIM. Various classification criteria for IIM and its subgroups have been published since 1970.2 Among them, the Bohan & Peter’s criteria and Tanimoto’s criteria for PM/DM have been commonly used in Japan.3–5

Bohan & Peter’s criteria, described in 1975, include five variables. There are some limitations, however; IBM and muscular dystrophies with inflammatory changes cannot be excluded. In addition, the skin rashes are not specified.

Tanimoto’s criteria, published in 1995, include nine variables. Confusingly, definitions of the Gottron’s sign and linear extensor erythema are different from those in other criteria. Only anti-Jo-1 antibody is included as the myositis-specific autoantibody, not more recently identified antibodies. Furthermore, because Tanimoto’s criteria rely mainly on muscle abnormalities and related systemic rheumatic symptoms, amyopathic DM (ADM) cannot be diagnosed. Since the criteria have been used to identify patients for the purpose of subsidisation of medical expenses in Japan, there were some problems, especially among patients with ADM with severe interstitial lung diseases, until the recent revision in 2015 (online supplementary table 1).

Unlike previously published criteria, which were defined primarily on the basis of clinical impressions of experts and without validation by external cohorts, a new set of criteria were proposed on the basis of data analyses by the International Myositis Classification Criteria Project (IMCCP). Demographic, clinical and laboratory data of the 976 patients with IIM and consisting of 75 variables were collected and compared with those of 624 comparators which included patients with a broad spectrum of mimicking conditions, such as autoimmune myopathy or non-inflammatory myopathy. Data analysis revealed that 16 variables could distinguish the IIM cases from the comparators most effectively. The new criteria were examined for sensitivity validation with external cohorts and recently approved by European League Against Rheumatism (EULAR) and American College of Rheumatology (ACR) (http://www.imm.ki.se/biostatistics/calculators/iim).6 7

The new criteria have three distinct features, and the first is that they were developed differentially for patients who underwent muscle biopsy and those who did not. This consideration is primarily for juvenile cases and made the criteria applicable to adult and juvenile patients. The second feature of the new criteria is that each variable of the criteria was assigned a score based on its ability to discriminate IIM from the comparators. The criteria provide total aggregated scores for each patient, and the probability of IIM in each patient can be calculated from the total aggregated score using a formula. Last, while the original paper suggested a recommended cut-off probability for IIM as 55%,6 7 there is users’ flexibility in the setting of cut-offs according to the type of study; for example, the cut-off points can be increased to facilitate high specificity for clinical trials that should require stringent classification.

Taken together, IMCCP concluded that at present the new EULAR/ACR-approved criteria are the most reliable. Yet to be done for the criteria are validation of specificity with external cohorts and also for cases with ethnicities that may be underrepresented in the original cohort. Accordingly, in the current study, we validated the specificity as well as the sensitivity of the new criteria using a Japanese cohort. This is one of the research projects of subcommittee for PM/DM in the Japanese National Research Committee on Autoimmune Diseases supported by the Ministry of Health, Labour and Welfare of Japan. We discuss the usefulness and limitations of the criteria when used for Japanese patients with IIM.

Materials and methods

​Study subjects and data collection

The subjects of the present study include 420 IIM cases and 402 non-IIM cases with definitive diagnosis of IIM or other diseases made by experts including the authors at 19 institutes participating in this national project between 2007 and 2013. The project was approved by research ethics committees of individual institutes and carried out by expert rheumatologists, neurologists, dermatologists and epidemiologists in these institutes. As inclusion criteria, the clinical data of allocated number of IIM cases assigned to each facility were retrospectively collected in the order of new patient visits from medical records using questionnaires (online supplementary table 2), which were distributed to the experts. There were no exclusion criteria.

Non-IIM cases were also collected from each institute, in the order of patient visits (online supplementary table 3). Diagnosis of all cases was validated by an expert rheumatologist, neurologist and dermatologist.

​Methodology of new EULAR/ACR classification

The new EULAR/ACR classification criteria show the probability of a patient having IIM for use in clinical and research settings.6 7 In brief, the new criteria consist of the muscle findings, skin findings and laboratory measures.6 7 Each item is assigned a weighted score, and the aggregated scores are calculated by summing the score points. The aggregated scores can be converted into a probability of IIM using the following formulas:

Probability of IIM = 1/[1 + exponential (6.49 – score)] when muscle biopsy data are present or

Probability of IIM = 1/[1 + exponential (5.33 – score)] when muscle biopsy data are absent

Unknown or missing data were regarded as ‘score 0’. Among the information included in the questionnaires, only the criteria items were used for classification of patients and comparators. The web-based calculator and the Excel file that can be downloaded from criteria web page were not used in this study. The correlation between total aggregated scores and varying probability of having IIM are shown in the previous paper.6 7 A patient is classified as an IIM case if the patient’s probability is above a specified cut-off value.

Furthermore, a patient classified as IIM by the EULAR/ACR classification criteria can be further subclassified by a classification tree into six groups: PM, DM, ADM, juvenile DM (JDM), other juvenile myositis and IBM.6 7

Other methods are described in online supplementary text.

Results

​Demographic characteristics of IIM and non-IIM cases in the study population

Data from 420 IIM and 402 non-IIM cases were collected at the 19 institutes that participated in the present study. The IIM cases consisted of patients with five subgroups: PM (n=137; three patients had IMNM), DM (n=188), ADM (n=64), JDM (n=6) and IBM (n=7) (online supplementary table 4). The non-IIM cases consisted of patients with 31 different diseases including systemic lupus erythematosus (SLE; n=88), systemic vasculitis (n=63), systemic sclerosis (n=60), mitochondrial myopathy (n=8) and contact dermatitis (n=9).

The IIM cases consisted of 127 male and 293 female patients, whereas the non-IIM cases consisted of 106 male and 296 female patients (table 1). The mean age of onset was 53.9±15.4 years in the IIM cases and 46.4±19.2 years in the non-IIM cases, whereas the mean ages at diagnosis was 55.1±15.4 years in IIM and 48.8±18.9 years in non-IIM. The number of cases under 18 years of age at diagnosis was five (1.2%) in IIM and 12 (3.0%) in non-IIM.

Table 1

Demographic characteristics of IIM cases and non-IIM cases in the study population

The frequency of each variable in IIM cases and non-IIM cases are shown in online supplementary table 5. L2 (elevated serum levels of lactate dehydrogenase (LDH)) was the most common (93.8%) in patients with IIM among items included in the new criteria. Among the other items, skin biopsy was performed in 174 (41.4%) out of 420 IIM and 87 (21.6%) out of 402 non-IIM. Electromyogram and muscle MRI were performed in 67.9% and 70.2% of patients with IIM, respectively.

Sensitivity and specificity of the EULAR/ACR classification criteria for the Japanese cohort

The distribution of the probabilities of both IIM and non-IIM in all Japanese cases, the cases without muscle biopsy data and the cases with muscle biopsy data are shown in online supplementary figure 1.

The IMCCP team concluded that the best balance between sensitivity and specificity to diagnose IIM was found at probability of 55% in patients both with and without muscle biopsy data.6 7 As was done by the team, sensitivity and specificity of the new criteria in the all Japanese cohort for varying probability cut-offs were calculated (figure 1A). Those in the patients with IIM without the muscle biopsy data and with the muscle biopsy data were also calculated separately (figure 1B, C). The optimal probability cut-off for sensitivity and specificity was found at 50% in all Japanese cases; sensitivity and specificity were 91.0% (95% CI: 87.79 to 93.52) and 89.8% (86.42–92.58), respectively. The optimal cut-offs were found at 40% in patients without muscle biopsy data; sensitivity and specificity were 91.6% (86.87–95.02) and 91.9% (88.55–94.57), respectively and at 65% in patients with muscle biopsy data; sensitivity and specificity were 87.6% (82.49–91.68) and 70.9% (57.10–82.37), respectively.

Figure 1

Sensitivity/specificity and probability of the EULAR/ACR classification criteria for IIM in the all Japanese cases (A), the cases without muscle biopsy data (B) and the cases with muscle biopsy data (C). Blue line and red lines indicate sensitivity and specificity, respectively. ACR, American College of Rheumatology; EULAR, European League Against Rheumatism; IIM, idiopathic inflammatory myopathies.

When the cut-off of 55% (recommended by the IMCCP team) was applied, the sensitivity and specificity in the all Japanese patients were 89.3% and 91.0%, respectively (table 2). In patients without muscle biopsy data, they were 88.1% and 95.1% respectively, and in patients with muscle biopsy data they were 90.4% and 65.5%, respectively (table 2). Since shift of the probability cut-off from the optimal values to 55% makes subtle changes (<6%) in the sensitivity and specificity, 55% should also be acceptable for Japanese patients. The new criteria were therefore validated with a Japanese cohort. Receiver operating characteristics curve analysis indicated that the area under the curve for all Japanese cases, the cases without muscle biopsy data and the cases with muscle biopsy data was 0.97, 0.87 and 0.97, respectively (online supplementary figure 2). The specificity in the Japanese patients with muscle biopsy data being lower than that in the IMCCP cohort was addressed in the following separate analyses.

Table 2

Sensitivity and specificity of EULAR/ACR classification criteria for IIM

Sensitivity and specificity for subgroup classification of the Japanese cohort with the EULAR/ACR classification criteria and classification tree

The patients classified as having IIM according to the EULAR/ACR criteria were further classified into subgroups with the associated classification tree.6 7 The percentages of the patients who were subclassified correctly with the classification tree into the ‘true’ classification made by their physicians were 70.9% in PM, 73.2% in DM, 82.9% in ADM, 100% in JDM and 71.4% in IBM (table 3). The subgroup classification performed well as in the IMCCP analysis.6 7 Difficult cases included a 77-year-old patient, with serum CK activation but not with objective muscle weakness. The patient had erythematous papules on the extensor surfaces of extremities. The IMCCP definition of Gottron’s papules is ‘Erythematous to violaceous papules over the extensor surfaces of joints’ and can be skin lesions of the other diseases including hand eczema and verruca. Thus, due to absence of any other cutaneous manifestations, the eruption of the patient was not judged as skin manifestation specific to IIM by a clinician. The patient was diagnosed as having PM, which was validated by experts, but fell into ADM according to the tree.

Table 3

Comparison of physician-diagnosed IIM subgroups with IIM subgroups defined according to the classification tree among patients meeting the EULAR/ACR classification criteria for IIM

​Comparison of sensitivity and specificity of the EULAR/ACR criteria with other classification criteria

Sensitivity and specificity of the new classification criteria with cut-off probability 55% in Japanese patients with PM/DM/JDM/ADM were 87.4% and 92.4%, respectively (table 4). They were then compared with previously published criteria (Bohan & Peter’s and Tanimoto’s criteria) using the same dataset (table 4). Since Bohan & Peter’s and Tanimoto’s criteria are for classification of PM/DM/JDM/other juvenile myositis, but not of ADM, the ADM cases were excluded in their specificity and sensitivity studies: actually only 26.3% (20 of 76) of patients with ADM were captured using Bohan and Peter’s criteria. The specificity and specificity of the new criteria were comparable with those of Bohan & Peter’s criteria (88.4% and 88.3%, respectively) and were higher than those of Tanimoto’s criteria (82.2% and 87.8%, respectively).

Table 4

Sensitivity and specificity of EULAR/ACR classification criteria and the classification tree for PM/DM

In Japan, Tanimoto’s criteria were partly revised in 2015 to include ADM by adding the following note to the criteria; an amyopathic patient whose histopathological findings of the eruptions are compatible with DM can be diagnosed as having ADM (online supplementary table 1). This change was made for expansion of medical expense subsidisation for patients with ADM. When patients with ADM were included for analyses of the revised Tanimoto’s criteria, the sensitivity and specificity turned out to be 71.2% and 87.8%, respectively. Thus, the EULAR/ACR classification criteria have the highest sensitivity and/or specificity for the Japanese cohort, regardless of whether ADM is included or not.

Sensitivity and specificity of the EULAR/ACR classification criteria for IIM after overlap syndrome were included in IIM

The specificity for patients with the muscle biopsy data was low (65.5%) (table 2) in the Japanese cohort. This was because 19 out of 55 patients with non-IIM with muscle biopsy data were diagnosed as having IIM according to the new criteria. The diagnosis of the 19 cases included rimmed vacuolar distal myopathy (all three patients were misclassified), mitochondrial myopathy (1 of 6 patients), sarcoidosis (2 of 3 patients), systemic vasculitis (1 of 11 patients), other dystrophy (4 of 10 patients), systemic sclerosis (2 of 3 patients) and mixed connective tissue disease (MCTD, 6 of 8 patients). We suggest that Japanese physicians have differing definition of myositis from those from other countries when patients have other autoimmune diseases. When patients have myositis with systemic sclerosis or MCTD, they are diagnosed as having scleroderma myopathy or MCTD itself, but not IIM. In contrast, in the EULAR/ACR criteria, this is considered as overlap syndrome.

Suspecting that the low specificity may be due to the differing definition of myositis, we changed the definition. When patients with ‘overlap syndrome’ were defined as those with SLE, MCTD and systemic sclerosis, who had abnormalities in any of four items of muscle findings or elevation of serum skeletal muscle enzymes, 12 patients (1 SLE, 8 MCTD and 3 systemic sclerosis) with muscle biopsy data were diagnosed as having overlap syndrome. If these patients were transferred to IIM, the specificity in classification of the patients with biopsy increased up to 74.4% (table 5).

Table 5

Sensitivity and specificity of EULAR/ACR classification criteria for IIM after overlap syndrome was included in IIM

Discussion

The present analysis, involving Japanese 420 IIM cases and 402 non-IIM comparator cases, is important as the first external validation of specificity of the new classification criteria. The sensitivity was also validated within a cohort of Asian ethnicity. There was some question as to whether the new criteria could be validated in Asian and African populations, since these groups were underrepresented in the IMCCP study.6 7

Another significance of this study is the comparison of the new criteria with the previous criteria. Bohan & Peter’s criteria and Tanimoto’s original or revised criteria have been commonly used in Japan. They were not designed for classification of IBM. Thus, we calculated the sensitivity and specificity of Tanimoto’s criteria and Bohan & Peter’s criteria for our Japanese patients and directly compared the usefulness of the new criteria and previous criteria for the classification of PM/DM/JDM/ADM. The specificity and/or sensitivity in the new criteria was higher than that of Bohan & Peter’s criteria and Tanimoto’s original or revised criteria, confirming that the new criteria should be applied in classification of Japanese patients with IIM instead of Tanimoto’s revised criteria.

The low specificity of the new criteria for Japanese patients was due to myositis associated with other autoimmune diseases not being defined as PM in overlap syndromes. This is in sharp contrast with the classification in the EULAR/ACR areas, where these conditions are defined as overlap syndromes. Transfer of patients with those conditions from the comparator group to the IIM group actually made the specificity higher. To apply the new criteria in Japan, it appears that Japanese physicians should use the same definitions as those in the EULAR/ACR countries.

Other different points between the IMCCP and Japanese cohorts include the composition of diseases in non-IIM. The Japanese cohort had more rheumatic diseases (74% vs 36%), more dermatological diseases (15% vs 5%) and less neurological diseases (9% vs 58%) than the IMCCP cohort. The new criteria may be therefore better in ruling out neurological comparators than rheumatological and dermatological comparators when muscle biopsy was performed. Notably, the report of the EULAR/ACR criteria also described the sensitivity and specificity of Bohan & Peter’s criteria (98% and 55%, respectively) and Tanimoto’s original criteria (96% and 31%, respectively) for the classification of IIM in their cohort and concluded that the new criteria are superior in specificity. Low specificity of the previous criteria in the IMCCP analysis may be affected by the percentage of neurological comparators. Similarly, in the current project, very few children were enrolled in both IIM cases and comparators; thus, the results may mainly refer to the adult population.

Second, patients with IBM were underrepresented in the Japanese cohort (1.7% vs 18%). Fewer patients with IBM were enrolled in this study because we have less patients with IBM in Japan. The national registry of intractable diseases, which should cover all patients requiring subsidisation of medical expenses, had only 350 patients with IBM registered among the Japanese population of 127 million people in 2017. The new criteria might identify IBM more readily than other IIM.

Third, muscle biopsies were performed in IMCCP cohort mostly if skin rashes were absent. Thus, muscle biopsy data were available in 80% of cases and comparators.6 7 In Japan, however, muscle biopsy is generally performed only when differential diagnosis is difficult. There may therefore be a greater number of difficult cases in the patient group with muscle biopsy data. The low number of individuals with available muscle biopsy data particularly in the comparator cases (only approximately 15%) may affect the statistics and the validity of the results. The present study is retrospective, so we could not control the number of patients with muscle biopsy data.

The present study has some limitations, as described in online supplementary text. Moreover, the new criteria did not include recently identified myositis-specific autoantibodies such as anti-MDA5, anti-Mi-2 and TIF1γ autoantibodies. In our analysis, several patients with DM could not be diagnosed correctly by the new criteria, but were positive for anti-ARS antibodies, which indicates that the new criteria could be improved by adding these specific antibodies. As was discussed in the IMCCP report together with the MRI and electromyogram findings,6 contribution of these new antibodies to sensitivity and specificity should be validated in the future.

Acknowledgments

This study was supported in part by project research on intractable diseases from the Japanese Ministry of Health, Labour and Welfare.

References

View Abstract

Footnotes

  • Handling editor Josef S Smolen

  • Contributors MJ, AO, SI and HK performed the research, analysed the data and wrote the paper. Other authors contributed to collect patients’ data. MJ and AO equally contributed to this work.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement statement The clinical data of patients were retrospectively collected from the medical records using questionnaires.

  • Patient consent for publication Not required.

  • Ethics approval The project was approved by research ethics committees of individual institutes (No. 1580).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.

  • Author note Authors’ affiliations listed were current at the time that the data collection was performed. The current address of Hajime Sano is: Department of Rheumatology, Kyoto Okamoto Memorial Hospital, Kyoto, Japan.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.