Article Text

Extended report
Measurement invariance of the Illness Invalidation Inventory (3*I) across language, rheumatic disease and gender
  1. Marianne Belia Kool1,2,
  2. Rens van de Schoot3,
  3. Isabel López-Chicheri García4,
  4. Ricarda Mewes5,
  5. José A P Da Silva6,
  6. Karoline Vangronsveld7,
  7. Andreas A J Wismeijer8,
  8. Mark A Lumley9,
  9. Henriët van Middendorp10,
  10. Johannes W J Bijlsma2,
  11. Geert Crombez7,
  12. Winfried Rief5,
  13. Rinie Geenen1,2
  1. 1Department of Clinical and Health Psychology, Utrecht University, Utrecht, The Netherlands
  2. 2Department of Rheumatology and Clinical Immunology, University Medical Center Utrecht, Utrecht, The Netherlands
  3. 3Department of Methods and Statistics, Utrecht University, Utrecht, The Netherlands
  4. 4Department of Psychology, Catholic University San Antonio, Murcia, Spain
  5. 5Department of Clinical Psychology and Psychotherapy, Philipps University Marburg, Marburg, Germany
  6. 6Department Rheumatology, University Hospital Coimbra, Coimbra, Portugal
  7. 7Department of Experimental Clinical and Health Psychology, Ghent University, Ghent, Belgium
  8. 8Department of Developmental and Clinical Psychology, Tilburg University, Tilburg, The Netherlands
  9. 9Department of Psychology, Wayne State University, Detroit, Michigan, USA
  10. 10Department of Medical Psychology, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
  1. Correspondence to Marianne Belia Kool, Department of Clinical and Health Psychology, Utrecht University, PO Box 80140, Utrecht 3508 TC, The Netherlands; m.b.kool{at}uu.nl

Abstract

Objectives The Illness Invalidation Inventory (3*I) assesses patients’ perception of responses of others that are perceived as denying, lecturing, not supporting and not acknowledging the condition of the patient. It includes two factors: ‘discounting’ and ‘lack of understanding’. In order to use the 3*I to compare and pool scores across groups and countries, the questionnaire must have measurement invariance; that is, it should measure identical concepts with the same factor structure across groups. The aim of this study was to examine measurement invariance of the 3*I across rheumatic diseases, gender and languages.

Methods Participants with rheumatic disease from various countries completed an online study using the 3*I, which was presented in Dutch, English, French, German, Portuguese and Spanish; 6057 people with rheumatic diseases participated. Single and multiple group confirmatory factor analyses were used to test the factorial structure and measurement invariance of the 3*I with Mplus.

Results The model with strong measurement invariance, that is, equal factor loadings and thresholds (distribution cut-points) across gender and rheumatic disease (fibromyalgia vs other rheumatic diseases) had the best fit estimates for the Dutch version, and good fit estimates across the six language versions.

Conclusions The 3*I showed measurement invariance across gender, rheumatic disease and language. Therefore, it is appropriate to compare and pool scores of the 3*I across groups. Future research may use the questionnaire to examine antecedents and consequences of invalidation as well as the effect of treatments targeting invalidation.

  • Psychology
  • Fibromyalgis/Pain Syndromes
  • Osteoarthritis
  • Rheumatoid Arthritis

Statistics from Altmetric.com

Introduction

Invalidation, defined as the perception of cognitive, affective and behavioural responses of others that are judged to be denying, lecturing, not supporting and not acknowledging the condition of the patient,1 is problematic for some patients with rheumatic diseases. Symptoms of rheumatic diseases such as pain, fatigue and stiffness are mostly invisible and because of this people in the social environment of the patient might forget or misjudge the burden and consequences of the illness.2 When there is no clinical or laboratory evidence to account for the symptoms of the rheumatic illness, such as in fibromyalgia,3 this can provoke even more serious disbelief and distrust towards the patients.4 Indeed, patients with fibromyalgia reported that invalidation is a major issue in their lives, adding a burden to the symptoms.1 ,5 Moreover, invalidation could increase the risk of becoming more physically impaired and depressed,6 thus highlighting the need for attending to invalidation in research and clinical settings.7

The Illness Invalidation Inventory (3*I) is a self-report questionnaire that assesses patients’ invalidation. An initial evaluation suggested internal consistency and concurrent validity of the inventory.6 To be able to use the 3*I in epidemiological studies, to make comparisons of invalidation across patient groups and countries and to examine its antecedents and consequences, the questionnaire must measure identical constructs with the same structure across groups. This means that the factor structure, factor loadings and thresholds should be comparable across groups,8 which is called measurement invariance. Thresholds are the points on the unobserved normal distribution where, on average, respondents vary between two different response options.9 When measurement invariance is absent, groups or subjects respond differently to items, and factor means cannot validly be compared across groups. However, it is unclear whether the 3*I shows measurement invariance across rheumatic diseases, gender and languages.

The 3*I assesses invalidation by the spouse, family, medical professionals, work environment and social services using eight items for each of these five social sources (figure 1). The structure for each social source comprises two factors: discounting (five items) and lack of understanding (three items). Discounting represents active negative social responses including disbelieving, admonishing, dismissing inability to work, not acknowledging symptom fluctuations and offering unusable advice. Lack of understanding reflects a lack of positive social responses such as not recognising, comprehending and emotionally supporting the patient or illness. To date, the Dutch version of the 3*I has been validated in patients with rheumatoid arthritis and fibromyalgia,6 but measurement invariance across rheumatic diseases, gender and language versions of the 3*I has not been tested. Furthermore, qualitative studies suggest that invalidating experiences of patients with rheumatic diseases are common in many countries.4 ,5 ,10 ,11 However, the study of invalidation across countries is only legitimately possible after the equivalence of the language versions (other than the Dutch version) of the 3*I has been established.

Figure 1

Factor structure of the Illness Invalidation Inventory (3*I). Note: Items for the factor ‘lack of understanding’ have reversed response scores.

The first aim of the current study was to test the factor structure in a large sample of Dutch patients with several rheumatic diseases and to examine measurement invariance between patients with fibromyalgia and patients with other rheumatic diseases and between men and women. The second aim was to examine measurement invariance between different language versions (Dutch, English, French, German, Portuguese and Spanish).

Methods

Participants

The participants comprised 6057 people with rheumatic diseases from different countries. Inclusion criteria were: (1) the self-report of a rheumatic disease; (2) the report that the disease was diagnosed by a medical specialist, general practitioner or nurse; (3) being 18 years or older; and (4) speaking Dutch, English, French, German, Portuguese or Spanish. Table 1 shows the characteristics of the participants, who had a mean age of 45.7 years and were mostly female (88%). With the exception of the German respondents, patients with fibromyalgia constituted the largest percentage of respondents (40%–76% across different languages). The German version of the questionnaire was completed by patients with a wide range of conditions, particularly systemic lupus erythematosus (29%) and ankylosing spondylitis (25%). Overall, patients with osteoarthritis (8%–28%), rheumatoid arthritis (6%–20%) and Sjögren's syndrome (4%–21%) were well represented in the study.

Table 1

Demographic characteristics of patients with rheumatic diseases for the six language versions

Procedure

The study was conducted according to the principles of the Declaration of Helsinki12 and was tested and approved by the Medical Ethical Committee of the University Medical Center Utrecht.

The software program ‘Netquestionnaires’13 was used to develop the online international questionnaires. An online version was first developed for the Dutch version, which was pre-tested among a small sample of Dutch patients. After translation, the other language versions were developed uniformly and pre-tested by a small expert group.

Participants were invited via a recruitment notice on websites of patient associations for rheumatic diseases. Patient associations in Dutch, English, French, German, Portuguese and Spanish language countries were asked to put the recruitment notice on their website. The text of this notice was similar across countries and patient associations. It is unknown whether (some) patient organisations took additional actions to bring the call to the notice of patients. On the website, participants could choose one of the six language versions of the study. The recruitment notice included information about the aim and content of the study, inclusion criteria, duration of participation (about 20 min) and a hyperlink to the online questionnaire. Participants could decide to participate after being informed about the study, and were able to stop at any point if they desired. Of the 8293 participants who began filling out the questionnaire, 6057 (77%) completed it and provided the data analysed here. About 80% of the dropouts were participants who stopped filling out the questionnaire while responding to the first demographic questions.

Instruments

The online study included items about demographic characteristics and the 3*I.

The 3*I6 measures invalidation by each of five sources (spouse, family, medical professionals, work environment and social services). It consists of two factors: discounting (five items) and lack of understanding (three items; see figure 1). Participants indicate on a 5-point scale (1, never; 2, seldom; 3, sometimes; 4, often; 5, very often) how frequently during the past year people within each category responded to them in the described way. A source category that does not apply can be skipped. An initial validation study in Dutch patients demonstrated good reliability and validity of the 3*I.6

Translation of the 3*I was done by the forward-and-backward translation method. First, the Dutch version of the 3*I was translated by a native English and Dutch speaker to English. When consensus was reached about the English version, another native English and Dutch speaker translated the English version back to Dutch. After consensus was reached between translators and researchers, the final English version of the questionnaire was determined. The English version of the questionnaire was used for the translation to other languages. For each translation, the forward-and-backward translation method was used by two native English speakers and two native speakers of the other language.

Measurement invariance

The first step to determine measurement invariance is to specify the model (adequate structure) of the instrument.14 With confirmatory factor analyses (CFAs), the factor structure of the model is tested for each group. The second step is to check whether the best fitting factor model is adequate and equal across groups. This can be tested, first, by comparing the numerical values of the factor loadings across groups, which should be similar (weak invariance), and, second, by comparing whether the thresholds are similar across groups (scalar invariance).15 For straightforward interpretation of latent variable means and patterns of correlations across groups, both the factor loadings and the thresholds should be similar across groups (strong invariance).16

When weak invariance is not supported, this could mean that one or more of the common factors have different meanings across the population groups,17 or that a subset of the factor loading estimates for a group is biased due to extreme response style. When strong invariance is not supported, either differential additive response bias or differential acquiescence response styles might be the problem.18

Statistical analysis

Single and multiple group CFAs were used to test the factorial structure and measurement invariance by means of Mplus V.6.11.19

First, the Dutch data were used to check whether the model in figure 1 with two factors fits the data better than a one-factor model. Furthermore, we investigated whether the model better fits the data with continuous or ordered categorical indicators. Because a model including all sources of the 3*I with categorical indicators was too complex for Mplus to compute, the model was run for each source separately (spouse, family, medical professionals, work environment and social services). In the Results section, the statistics for the source ‘family’ are presented because this source applies to almost all participants. The results for the other four sources are presented in an online supplementary file.

Due to non-normally distributed item scores, a robust weighted least squares means and variance (WLSMV) estimator was used. Full information maximum likelihood estimation was used to include participants with a score on at least one item of a subscale.20 To assess model fit, we used the comparative fit index (CFI), Tucker–Lewis index (TLI) and root mean square error of approximation (RMSEA). Cut-off values for fit were considered adequate if CFI and TLI values are >0.90 and RMSEA<0.08. The Bayesian information criterion (BIC) was used to compare competing models. A lower BIC indicates a better trade-off between model fit and model complexity. Because the BIC value is not estimated by the WLSMV estimator, all models were repeated using a maximum likelihood estimator, again using a correction for non-normality.

In the second step of the analyses, measurement invariance across rheumatic diseases was tested in the Dutch data for patients with only fibromyalgia versus patients with one other rheumatic disease (n=890). The other 548 patients who reported having two or more comorbid rheumatic diseases were excluded from this analysis. Third, measurement invariance across gender was tested in the Dutch data. Fourth, measurement invariance was tested across languages in the international data using a similar procedure.

In each test of measurement invariance, three models were analysed: Model 1 includes constrained factor loadings and thresholds free (weak invariance); Model 2 includes constrained thresholds and factor loadings free (scalar invariance); Model 3 includes constrained factor loadings and thresholds (strong invariance).21

In the last step of the analysis, standardised factor loadings, threshold values and internal consistency (Cronbach's α) were examined.

Results

Factor structure of the 3*I

For several items of the 3*I, the score distribution across the response categories were skewed. Therefore, one- and two-factor models with continuous factor indicators were compared with a one- and two-factor model where the items were defined as being categorical. Table 2 shows outcomes of the four CFAs for the 3*I source ‘family’ factor structure in the Dutch data. The two-factor model with categorical items provided the better trade-off between model fit and model complexity according to the BIC value and it had adequate fit estimates. Analyses for the four other social sources were repeated and the two-factor model with categorical items also showed the best fit for each source (see online supplementary table S1). This model has been used in subsequent analyses.

Table 2

Fit statistics of confirmatory factor analyses of the 3*I source ‘family’ for the one- and two-factor solution including continuous or categorical factor indicators in the Dutch sample (n=1855)

Measurement invariance of the 3*I across rheumatic diseases

Table 3 shows outcomes of the CFA for the source ‘family’ of the 3*I for fibromyalgia versus other rheumatic diseases in the Dutch data. The three models were tested for each source of the 3*I. Although Model 1a had the lowest χ2 value and RMSEA, and the largest CFI and TLI, the fit indices were also acceptable for Model 3a. The BIC value was lowest for Model 3a, which shows that Model 3a with both the factor loadings and the thresholds constrained is simpler than Model 1a; it is the preferred model because it has a better trade-off between model fit and model complexity. The BIC value was also lowest for Model 3a among the other sources (see online supplementary table S2). Therefore, Model 3a was chosen as the best model indicating measurement invariance across rheumatic diseases.

Table 3

Test of measurement invariance of source ‘family’ of the 3*I of rheumatic disease (fibromyalgia vs other rheumatic disease) and gender in the Dutch sample, and of language in the international sample

Measurement invariance of the 3*I across gender

Measurement invariance was also tested across gender in the Dutch data. Table 3 shows the fit estimates for the source ‘family’. Model 3b (strong invariance) best fitted the data: it had adequate CFI, TLI and RMSEA values and the lowest BIC value. Model 3b also showed the lowest BIC value among the other sources (see online supplementary table S3). This supports measurement invariance across gender.

Measurement invariance of the 3*I across languages

In the separate English, French, German, Portuguese and Spanish data sets, CFA was conducted to examine the two-factor structure of the 3*I for each language version of the 3*I as compared with the Dutch factor structure. The two-factor model showed adequate fit estimates for each language version of the 3*I (results are not shown). Because this result confirmed the equivalence of factor structures of the 3*I across languages, it was feasible to study measurement invariance of the 3*I across the languages.

A multiple group model including all data sets was created to compare Models 1c, 2c and 3c across languages. Fit estimates are shown in table 3 for the source ‘family’ of the 3*I. All models (1c, 2c and 3c) showed adequate CFI and TLI estimates. The fit estimates showed the best fit for Model 1c (weak invariance); in general, it had the lowest χ2, RMSEA and BIC values and the largest CFI and TLI. Model 1c (constrained factor loading, thresholds free) indicates that the factor loadings are invariant across languages, but that threshold values are non-invariant across languages.

To illustrate this finding, consider the standard factor loadings obtained with Model 1c (factor loadings free; table 4). The factor loadings differ barely between language versions of the 3*I. Thus, constraining these factor loadings to be equal resulted in a simpler model (ie, less parameters have to be estimated) and it yields a better fit (table 3).

Table 4

Standardised factor loadings for 3*I for each language version

Table 5 shows the thresholds of items 1 and 2 for the source ‘family’ of the 3*I in each language version. The results indicate differences between languages; for example, item 2 threshold 2 shows −0.25 for the French version and 0.40 for the German version. If we constrain the thresholds to be equal (Model 3c), the BIC indicates that, although Model 3c is simpler, the fit has worsened.

Table 5

Thresholds of items 1 and 2 of the source ‘family’ of the 3*I of each language version

The fit of Model 3c could be improved through partial measurement invariance (setting some thresholds ‘free’, but constraining others). To identify non-invariant thresholds, both the modification index (MI) for a parameter, which gives the expected drop in the model's χ2 value if the parameter is freely estimated, and the value of thresholds across languages can be studied. Ideally, one or two items or one or two languages would be identified, explaining the non-invariant thresholds. In that case, a solution would be to delete these items or to deal with computation of factors in this language in a different manner. In our study, the German data seem most different. However, close inspection of the MI and the (5 times 32) threshold values across languages did not show significant MIs or specific threshold patterns. Therefore, it was not possible to determine the degree of partial measurement invariance. Because the model assuming strong measurement invariance still has a good fit to the data (ie, most model fit indices are well above the cut-off values) and differences in threshold values probably cancel each other out between the different languages,9 ,22 we concluded that Model 3c applies for all social sources (see online supplementary table S4). The Cronbach's α per language indicated that the internal consistency of the two 3*I factors is good: between 0.76 and 0.95 for the factor ‘discounting’ and between 0.78 and 0.93 for the factor ‘lack of understanding’. This provides additional support for using Model 3c.

Discussion

This study investigated the measurement invariance of the 3*I across rheumatic disease, gender and language version. Our results confirm the validity of a two-factor structure of the 3*I comprising discounting and lack of understanding for each source (spouse, family, medical professionals, work environment and social services) in patients with rheumatic diseases. Strong measurement invariance (ie, constrained factor loadings and thresholds) across rheumatic disease (fibromyalgia vs other rheumatic disease) and across gender was established. Strong measurement invariance across the six language versions of the 3*I was also supported by adequate fit estimates and by good internal consistency of the factors for each language. In most cases, the BIC supported strong measurement invariance, but not all estimates showed the best fit for this model. Because most models under investigation had a moderate to high RMSEA (0.08–0.14) and about equal CFI and TLI fit statistics, our model selection was largely based on BIC. Although the fit of the model with constrained factor loadings and thresholds was not the best across language, measurement invariance of the 3*I for languages was not rejected. Because all models (weak, scalar and strong invariance) showed good fit estimates, it is acceptable to conclude that the different language versions of the 3*I are comparable.

This study has some limitations. First, diagnoses were self-reported by patients, and there was no certification by a medical specialist. Second, participants were recruited through the internet, which may have led to a younger sample from a higher social economic class. However, this is less of a problem for the examination of measurement invariance, because the instrument should be invariant whether patients do or do not have the disease, are young or old, etc. Third, no expert committee was involved in the translation procedure of the 3*I.

The 3*I is the first instrument to quantify invalidation in patients with rheumatic diseases. Results of our study indicate that comparisons between diseases, gender and language versions of the 3*I are possible, and are likely to be not affected by different response styles or different interpretations of items. This is an advantage when examining antecedents and consequences of invalidation as well as the effects of treatments targeting invalidation. In clinical practice, the questionnaire can be used to assess invalidation in individual patients, which will help therapists to understand and treat patients’ problems.

Acknowledgments

We would like to thank all participants for their contribution to this study, the patient associations for help in recruiting participants and Joop Hox for his expertise and suggestions.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:

Footnotes

  • Handling editor Tore K Kvien

  • Funding This study was funded by a grant from the Dutch Arthritis Association (Reumafonds, DAA 06-2-401).

  • Competing interests None.

  • Ethics approval Medical Ethical Committee of the University Medical Center Utrecht.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.