Rheumatoid arthritis (RA) is a chronic disabling disease that is often associated with limitation in physical, mental, and social function,^1,2 with potential work disability.^3,4 In order to describe and assess daily functioning and disability from a bio-psychosocial perspective in all aspects of health, the framework of the World Health Organization International Classification of Functioning, Disability and Health (ICF) can be used. This framework provides a unified and standard language for the description of health and health related conditions and a common framework among all health professions.⁵ The ICF structure includes the two parts: (1) functioning; and (2) disability and contextual factors. Each part has two components: (i) Body Functions and Structures, Activities and Participation; (ii) Environmental Factors and personal factors. Components are further defined by the so called ICF categories. To rate the magnitude or the severity of the problem in each of the ICF categories, WHO proposes the so called qualifiers’ scale. The categories are scored by a health professional during an interview with the patient.

In order to facilitate the application of the ICF in clinical practice, specific core sets were developed for specific diseases as short lists of ICF categories that are important for patients. They serve as minimal standards for the reporting of functioning and health for clinical and epidemiological studies as well as in clinical encounters (brief ICF core set) or as standards for multiprofessional, comprehensive assessment (comprehensive ICF Core Set) under consideration of influential environmental factors. The comprehensive ICF Core Set for RA represents the typical spectrum in functioning of patients with RA with a selection of 96 categories,⁶ in the four components—Body Functions (b), Body Structures (s), Activities and Participation (d), and Environmental Factors (e). The preliminary version of the ICF Core Set for RA⁶ (hereafter stated when referring to the comprehensive ICF Core Set for RA) was developed by experts consisting of rheumatology health professionals in a formal decision making and consensus process. The process included a Delphi exercise,⁷ a systematic literature review,⁸ and an empirical data collection with the ICF checklist.⁹

The ICF Core Set for RA has undergone content validation from a patient perspective,^10,11 but the reliability of the qualifiers’ scale of the ICF Core Set for RA has not yet been studied. This study had the objective to assess the observer reliability of the ICF Core Set for RA in rheumatological practice. The specific aims were (1) to estimate the inter-rater and intra-rater reliability of the ICF Core Set for RA when applied by health professionals in a specialised rheumatology facility, and (2) to study the metric of the qualifiers’ scale.

PATIENTS AND METHODS

Study design

We conducted a reliability study with assessments at two time points one week apart in RA patients with stable disease activity.

Patients

A sample of 25 patients with RA¹² was included in the study after written informed consent. Patients were recruited consecutively from the rheumatology outpatient department at Diakonhjemmet Hospital in Oslo, Norway. The recent disease history was objectified from patients and hospital records.

Data collection

Patients were interviewed twice, at the first time point (T1) and then 6–10 days later (second time point (T2)). Interviews at T1 were performed independently by an occupational therapist (SL) and by a physiotherapist (RHM). Patients were assessed only by one therapist assigned randomly at T2. Both health professionals were working at the department of rheumatology and were trained according to a video issued by the ICF branch of the WHO at the Ludwig-Maximilian University in Munich, Germany.

The health professionals rated the magnitude or the severity of the problem in each of the ICF categories primarily based on the information obtained during the interview and only if necessary and in very few cases from available medical information in the patient charts.

A short introduction to the concepts of the ICF was given in lay terms to all patients at the beginning of each interview. Open ended questions were used to ask patients about their level of functioning in each of the areas referred to in each of the ICF categories contained in the ICF Core Set for RA. Patients also filled in self administered questionnaires while waiting for the interview. The rheumatology professionals had no insight into the questionnaires.

Measures

ICF Core Set for RA

The ICF Core Set for RA includes 25 categories from the component Body functions, 18 from the component Body Structures, 32 from the component Activities and Participation, and 21 from the component Environmental Factors.⁶ By error, one of the categories (b715 stability of joints function) was omitted. The severity of the patients’ problems in each of the ICF categories is quantified with the qualifiers’ scale. The qualifiers’ scale of the components Body Functions and Structures, and Activities and Participation has five response levels, each ranging from 0 to 4, corresponding to no/mild/moderate/severe/complete impairment. For example, a moderate problem with walking would give a score of 2. The qualifiers’ scale of the component environmental factors has nine response levels ranging from −4 to +4. A specific environmental factor can be a barrier (−4 to −1), a facilitator (1 to 4), or can have no influence (0) on the patient’s functioning. If the factor has an influence, the extent of the influence (either positive or negative) can be coded with mild/moderate/severe/complete. In addition, there are the response options “8, not specified” and “9, not applicable” for each category.

Health status

Measures to describe the study population and the stability of the disease between the two time points included visual analogue scales (0–100) for pain, fatigue, and patient global assessment of disease activity, the modified Health Assessment Questionnaire (MHAQ),¹³ short form SF-36,¹⁴ and Rheumatoid Arthritis Disease Activity Index (RADAI).¹⁵

Data analysis and statistics

We used descriptive statistics to describe the study population. To examine the stability of patients between both assessments the limits of agreement statistics by Bland and Altman were applied.¹⁶ Patients were excluded from the intra-rater reliability analysis if the difference in health status exceeded limits of agreement in two of the seven measures (pain, fatigue, patient global, RADAI, MHAQ, mental health, and physical function from SF-36) over the 6–10 day period between the assessments.

Since the response options of the qualifiers’ scale “not specified”⁸ and “not applicable”⁹ are not part of the ordinal scale ranging from 0 to 4 for the components of functioning (−4 to 4 for environmental factors) they were not included for kappa statistics. Thus, the response options 8 and 9 were considered missing data.

To examine the inter-rater and intra-rater reliability of the ICF categories raw agreement was calculated and weighted kappa statistics were applied to investigate whether there is more agreement than might occur by chance given random guessing. The choice of this approach was based on recent suggestions by, for example, Landis and Koch,¹⁷ to placing more weight on the raw data than on the kappa coefficient, in agreement statistics.^18,19 ICC analyses were not applicable for the categorical analyses in this study.

To explore the metric of the qualifiers’ scale, the Rasch model for ordered response levels was used.²⁰ Reversed threshold estimates provide sufficient evidence to conclude that the empirical ordering is not consistent with the intended ordering. Thresholds define the boundaries between response categories. Collapsing response levels may reveal the effective number and ordering of levels post hoc.²⁰ Therefore, in case of reversed threshold estimates, response levels were collapsed to obtain an effective number and ordering of response categories post hoc.

Two different dimensions, respectively, were studied based on the Rasch model—namely, the dimension “functioning” and the dimension “environmental factors.” In line with the bio-psychosocial model on which the ICF is based, the ICF categories of the components Body Functions and Structures, Activities and Participation were included in the Rasch analysis to study the dimension functioning. The environmental factors categories were included in the Rasch analysis to study the dimension environmental factors. After collapsing response categories agreement statistics were recalculated.

The data analysis regarding descriptive statistics and Bland–Altman plots was performed using SPSS 12.0. Weighted kappa values were analysed using the Statistical Analysis System (SAS version 9.1.3). Rasch analyses were performed with the program RUMM 2020.²¹ When applicable, the level of statistical significance was set to p<0.05. The study was approved by the regional ethics committee.

RESULTS

Demographic data and clinical characteristics of the participants are shown in table 1. For the analyses of the intra-rater reliability, two patients were excluded because the disease was not stable between the two time points of assessment. Mean time for assessments with the ICF Core Set for RA was 34.2 (SD 9.1, range 20–75) minutes.

Missing values of more than 5%, the result of response options “not specified” and “not applicable,” were present in 9/95 categories within raters (9%) and 23/95 categories (24%) between raters.

Tables 2–5 list the ICF categories in the different components Body Functions, Body Structures, Activities and Participation, and Environmental Factors and present intra-rater and inter-rater reliability with percentages for complete agreement and kappa statistics.

Mean intra-rater agreement for all ICF categories was 59% which increased to 72% after collapsing of qualifiers, ranging from 29% (e340) to 96% (b510) before, and from 44% (e450) to 96% (b510) after collapsing of qualifiers (tables 2–5).

Mean inter-rater agreement for all ICF categories was 47% and increased to 61% after collapsing of qualifiers, ranging from 0% (e450) to 80% (d560) before, and from 8% (d415) to 88% (d560) after collapsing of qualifiers (tables 2–5).

The mean intra-rater agreement per component was 61% for Body Functions, 62% for Body Structures, 60% for Activities and Participation, and 52% in the component Environmental Factors. The mean inter-rater agreement was for Body Functions 55%, for Body Structures 46%, for Activities and Participation 51%, and 31% in the component Environmental Factors. Between raters 52% of the ICF categories showed at least 50% agreement (78% after collapsing), and in 77% within raters (99% after collapsing).

Weighted kappa statistics showed reliability of 0.4 or higher in 82/95 ICF categories (86%) within raters, but only in 41/95 ICF categories (43%) between raters (table 6).

Rasch analyses suggested that reduction of the number of qualifiers from five to three—and from nine to three for environmental factors—improved both inter-rater and intra-rater agreement. According to these results, the response levels 1–2 and 3–4 of the ICF categories belonging to Body Functions, Body Structures, and Activities and Participation were collapsed, respectively. In the component Environmental Factors, the response levels from −4 to −1 and from 1 to 4 were collapsed.

Several considerations were thereby taken into account: Firstly, the number of response categories that does not follow the consecutive order intended was considered. Secondly, a further collapsing strategy was studied—namely, the collapsing of the response categories 3 and 4. However, this strategy did not yield satisfactory results as most of the ICF categories still presented response categories that did not have a consecutive order (results not shown). Also, owing to the low frequencies in response categories 3 and 4, no further collapsing strategies, such as collapsing response categories 1 and 2 and 2 and 3, were considered. Finally, the same response format was intended for all ICF categories. The proposed collapsing strategy is clinically intuitive for judging the severity of a problem in the corresponding ICF categories. After collapsing the response categories, only four ICF categories in the functioning component did not follow a consecutive order and five in the component environmental factors.

DISCUSSION

In this study the reliability of the ICF Core Set for RA demonstrated only low and at best moderate agreement. Agreement for the individual categories between different health professionals was lower than within the same person. Based on explorative analyses a reduction of the scale qualifiers from five to three (and from nine to three for environmental factors) could improve both inter-rater and intra-rater reliability.

This is the first study to explore and establish the reliability of an ICF Core Set for a specific disease. Extensive testing of the ICF is necessary as the ICF is a WHO adopted classification for global application. Reliability testing of the ICF has so far only been performed in one other study where not a core set, but specified ICF categories were tested for inter-rater reliability in geriatric patients, and a moderate reliability was reported.²²

ICF categories are not self assessed and a patient’s report is being interpreted and scored by an interviewer, leading necessarily to a discrepancy between scores and reduced reliability. Health professionals with different specialties—such as an occupational therapist and a physiotherapist in our study—also have different focus on and awareness for the individual ICF categories. Therefore, one expects better inter-rater agreement if similar professionals have been involved in the judgments.

Within this context it is important to remember that limits of agreement in clinical and health status measures are generally quite wide. This was for example demonstrated when we examined the test-retest reliability for different data collection methods of self reported health status in patients with RA.²³ In rheumatology, low reliability when grading tender joints is known,²⁴ and inter-rater agreement hardly exceeds the level of chance.²⁵ Therefore, it does not surprise us that determined ICF categories of the ICF Core Set for RA show low reliability.

In this study, low reliability was demonstrated especially for categories concerning the environmental factors, in particular inter-rater agreement; categories from this component of the ICF are scored by attributing a positive or negative graded weight. The patient provides the information during the assessment if a factor (for example, family) is a facilitator or a barrier or both. Because an environmental factor may be expressed as both a facilitator and a barrier by the patient, the interviewer may meet the challenge to decide on one overall final judgment when scoring the individual category. In addition, environmental factors may be difficult to assess owing to difficult technical terms that do not always have associations in everyday life or do not have a clear meaning. Another problem of testing the ICF is given by the qualifiers on a mixed scale 0–4 and recording if categories are not specified or not applicable. The ICF qualifiers “8, not specified” and “9, not applicable” are very useful from a clinical point of view but they represent a “barrier” that is difficult to overcome when performing statistical analyses.

Assessment of patients with the ICF Core Set is time consuming. Lack of clarity in wording of the ICF categories may also have produced a need for explanation, leading to variance in time required to complete the assessment (range 20–75 minutes) and to a potential for variance in agreement. Increasing experience of the health professionals in this study did not considerably shorten the time (data not shown), but time use was constant at all three assessments. The reliability of the core set in RA might be improved by means of a training manual.

Our analysis suggests reducing the number of qualifiers when applying the ICF Core Set for RA, because the metric of the five qualifier scale was not sound. In addition to improved reliability, the feasibility of the ICF Core Set in RA could be improved when fewer qualifiers need to be considered by the assessor. The suggested reduction of response qualifiers needs further confirmation in other studies as our finding of improved reliability with fewer response categories was driven by our own data. In addition, the sample size of our study has to be considered. This sample size was comparable with other studies examining reliability.^23,26 The number of ICF categories considered in the dimension functioning and in the dimension environmental factors for the Rasch analyses, respectively, is fairly high in relation to the number of patients in our sample. In future studies, a larger sample size should be included in the Rasch analyses to augment the precision of the parameter estimations of the Rasch model.

A limitation of this study is that it was conducted at one centre and our findings may not be extrapolated to other environments of patients and health professionals. In addition, the results of the Rasch analyses were analysed to derive conclusions regarding the category response functions of the qualifiers’ scale and were not further considered to obtain information regarding the fit of the ICF categories to the Rasch model. Further studies are needed to investigate whether the ICF Core Set for RA from the viewpoint of the modern test theory has contributed to building a measure.

A strength of this study is that we rigorously sought to examine reliability of the comprehensive ICF Core Set for RA for both inter-rater and intra-rater agreement. Patients in our sample represent the typical average age group of patients with RA in our outpatient clinic²⁷ with age comparable to patients in a recent validation of the patient perspective of the ICF Core Set for RA.¹⁰ Both assessors in this study were experienced clinicians with interest in and knowledge of the ICF concept.

In summary, the reliability of the ICF RA core set is now established, being low to moderate in this study and varied considerably across categories and between raters. Our analysis suggests reducing the number of qualifiers when applying the ICF Core Set in RA. It remains to be shown, whether this empirical observation of improved agreement with reduced number of three response qualifiers in all dimensions may enhance the acceptance and feasibility of the ICF Cores Set in RA. The importance of testing the ICF Core Set for RA is apparent as the ICF has been adopted by WHO, and research on the applicability of the ICF in RA and in other rheumatic diseases is warranted to confirm our findings in this new field. ICF is an ongoing process where results from evaluations and suggestions for further improvement are continuously integrated.