Article Text

Download PDFPDF

  1. Bart Seppen1,
  2. Merel J. L’ami1,
  3. Sharon Rico1,
  4. Marieke Ter Wee2,
  5. Franktien Turkstra1,
  6. Leo D. Roorda3,
  7. Fabio Catarinella4,
  8. Dirkjan van Schaardenburg1,
  9. Michael Nurmohamed1,
  10. Maarten Boers1,
  11. W.H. Bos1
  1. 1ARC | Reade, Rheumatology, Amsterdam, Netherlands
  2. 2VUmc | Amsterdam UMC, Department of Epidemiology and Biostatistics, Amsterdam, Netherlands
  3. 3ARC | Reade, Rehabilitation, Amsterdam, Netherlands
  4. 4Brightfish B.V., Hoofddorp, Netherlands


Background: Several mobile applications (apps) exist to monitor symptoms of rheumatoid arthritis (RA), but high-quality apps are lacking.1 We developed an app with patients, following the Medical Research Council guidance for developing and evaluating complex interventions.2 The app facilitates telemonitoring through a weekly MDHAQ/RAPID33 questionnaire and has been integrated with the local electronic medical record.

Objectives: To evaluate the users’ satisfaction, usability and engagement of the application, as a first and preparatory step toward the ultimate goal of telemonitoring and self-initiated care.

Methods: In a first, one month, pilot study 42 RA patients used the app. User’s satisfaction was measured using an 11-point Likert scale. Usability was measured using the three concepts of the Technology Acceptance Model (TAM).4 According to TAM, perceived usefulness (PU), perceived ease of use (PEOU), (behavioral) intention to use (ITU) predict how and when a product will be used. The PU and PEOU were measured using the system usability scale (0-100). A score of 68, or higher, indicates sufficient system usability.5 The ITU was measured on a 11-point Likert scale. Engagement was measured by the percentage of patients completing all weekly questionnaires. General feedback was collected and implemented.

In a second, one month, pilot study, the improved app was evaluated in 24 RA patients using 5 themes derived from the Mobile App Rating Score6: “engagement”, “functionality”, “aesthetics”, “information” and “subjective quality.” Participants rated the questions on a 5 - point Likert scale. All responses were categorized into three categories: negative (1-2), neutral (3) and positive (4-5). Afterwards, nine participants (three users, three dropout users, and three non-users) agreed to participate in a semi-structured interview to get feedback on the App.

Results: In the first study, the ReumaMeter scored an overall median score of 8.0 (interquartile range (IQR) 7.0-9.0), a mean system usability score of 76 (SD 14.8) and participants intended to keep using the ReumaMeter in the future (median 7.0, IQR 5.0-9.0). Engagement decreased to 61% in week 4. During the second study, the number of positive responses for each category was at least twice as high as the number of negative responses (Figure 1). Feedback that emerged during the interviews matched these responses. In addition, several participants stated that app usage declined due to low disease activity.

Conclusion: The participants’ overall feedback was positive in terms of users’ satisfaction and usability. Engagement dropped, which may be due to lack of internal triggers to measure disease activity when patients are in remission. To assess the overall impact of the app on RA patient care, a randomized controlled trial is planned.

References [1] Grainger R, et al. JMIR Mhealth Uhealth. 2017. 2:e7.

[2] Craig P., et al. BMJ. 2008. doi:10.1136/bmj.a1655

[3] Pincus, The Journal of Rheumatology. 2008. 35,2136-37.

[4] Davis, F.D. MIS Quarterly. 1989. 13(3):319-339.

[5] Bangor, A. International Journal of Human-Computer Intervction. 2008. 24(6) 574-594.

[6] Stoyanov, S.R., et al. JMIR Mhealth Uhealth, 2015. 3(1): p.e27.

Figure 1

Qualitative results of app-evaluation - 79% of participants responding

Disclosure of Interests: Bart Seppen: None declared, Merel J. l’Ami: None declared, Sharon Rico: None declared, Marieke ter Wee Grant/research support from: Nonrestricted grant from Lilly Netherlands BV, Speakers bureau: ARC Preceptorship program, Franktien Turkstra: None declared, Leo D. Roorda: None declared, Fabio Catarinella: None declared, Dirkjan van Schaardenburg: None declared, Michael Nurmohamed Grant/research support from: AbbVie, Bristol-Myers Squibb, Celgene, Eli Lilly, Janssen, Menarini, MSD, Mundipharma, Pfizer, Roche, Sanofi and UCB, Consultant for: AbbVie, Bristol-Myers Squibb, Celgene, Eli Lilly, Janssen, Menarini, MSD, Mundipharma, Pfizer, Roche, Sanofi and UCB, Speakers bureau: AbbVie, Bristol-Myers Squibb, Celgene, Eli Lilly, Janssen, Menarini, MSD, Mundipharma, Pfizer, Roche, Sanofi and UCB, Maarten Boers: None declared, W.H. Bos Grant/research support from: Abbvie, Sanofi, Novartis, UCB, Celgene, Roche, Speakers bureau: Abbvie, Sanofi, Eli Lilly

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.