Article Text
PDF
Abstract
Objective To produce European League Against Rheumatism (EULAR) recommendations for the reporting of ultrasound studies in rheumatic and musculoskeletal diseases (RMDs).
Methods Based on the literature reviews and expert opinion (through Delphi surveys), a taskforce of 23 members (12 experts in ultrasound in RMDs, 9 in methodology and biostatistics together with a patient research partner and a health professional in rheumatology) developed a checklist of items to be reported in every RMD study using ultrasound. This checklist was further refined by involving a panel of 79 external experts (musculoskeletal imaging experts, methodologists, journal editors), who evaluated its comprehensibility, feasibility and comprehensiveness. Agreement on each proposed item was assessed with an 11-point Likert scale, grading from 0 (total disagreement) to 10 (full agreement).
Results Two face-to-face meetings, as well as two Delphi rounds of voting, resulted in a final checklist of 23 items, including a glossary of terminology. Twenty-one of these were considered ‘mandatory’ items to be reported in every study (such as blinding, development of scoring systems, definition of target pathologies) and 2 ‘optional’ to be reported only if applicable, such as possible confounding factors (ie, ambient conditions) or experience of the sonographers.
Conclusion An EULAR taskforce developed a checklist to ensure transparent and comprehensive reporting of aspects concerning research and procedures that need to be presented in studies using ultrasound in RMDs. This checklist, if widely adopted by authors and editors, will greatly improve the interpretability of study development and results, including the assessment of validity, generalisability and applicability.
- ultrasonography
- outcome and process assessment
- healthcare
- quality indicators
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Statistics from Altmetric.com
Key messages
What is already known about this subject?
Nomenclature, definitions of ultrasound-detected pathologies, scoring systems and technical issues may affect the validity and generalisability of results of ultrasound studies in rheumatic and musculoskeletal diseases.
These aspects, along with critical design characteristics, are often suboptimally reported in current ultrasound studies.
What does this study add?
A 23-item recommendation checklist was developed by a European League Against Rheumatism taskforce to ensure transparent and comprehensive reporting of ultrasound research.
This is the first reporting checklist focused on how to report characteristics of imaging measurement tools.
How might this impact on clinical practice or future developments?
The use of this checklist may improve the interpretability, reproducibility and generalisability of study results.
Ultrasound is an imaging technique widely used in patients with rheumatic and musculoskeletal diseases (RMDs) to detect signs of inflammation and destructive changes.1 Despite an increased use in clinical practice facilitated by major technical advances in the resolution of soft tissue contrast (B-mode or grey scale (GS)) and of vascular perfusion (Doppler techniques), a relatively long learning curve2 and, until recently, the absence of agreed scoring systems have hampered its utilisation for research.3 4
The European League Against Rheumatism (EULAR) and the Outcome Measures in Rheumatology (OMERACT) Ultrasound Working Group have actively worked towards the standardisation of the technique by developing educational programmes and by performing several studies evaluating its reliability, validity and feasibility.5–8 These initiatives have underlined that factors such as nomenclature, definitions of ultrasound-detected pathologies, scoring systems and technical issues with the ultrasound equipment may affect the validity and generalisability of these results. These aspects, along with critical design characteristics, such as reproducibility, blinding, patient selection and clearly defined purposes of the ultrasound evaluation, are often suboptimally reported in the current ultrasound studies.5 6 9 10
Complete and accurate reporting is necessary to detect potential biases in the study (internal validity) and to assess the generalisability and applicability of the results (external validity). Over the last 20 years, many guidelines have been developed to improve the quality of reporting of research articles, including those for randomised controlled trials (RCT) (Consolidated Standards of Reporting Trials)11 and diagnostic accuracy studies (Standards for Reporting Diagnostic accuracy studies).12 13 EULAR has also contributed by developing recommendations for reporting registers and clinical trial extension studies.14 15 We are not aware of recommendations focused on how to report characteristics of imaging measurement tools such as the equipment characteristics, procedures or scoring, which can influence the validity and generalisability of study results. Therefore, an EULAR taskforce was convened to propose recommendations for the reporting of such aspects in ultrasound studies in RMDs.
Methods
The convenor (MADA), EULAR methodologist (LC) and project fellow (FC) led a multidisciplinary taskforce in accordance with the EULAR Standardised Operating Procedures (SOPs).16 The taskforce included 23 members from 11 European countries and from the USA and was composed as follows: 11 experts in ultrasound in RMDs, 7 in methodology, 1 in both ultrasound and methodology, 2 in biostatistics, 1 patient research partner and 1 health professional in rheumatology. Three of the 23 members were members of EMEUNET and 13 of them were also part of an editorial board.
The taskforce employed a stepwise process summarised in figure 1, including two face-to-face meetings and several Delphi rounds. First the EULAR methodologist, convenor and fellow searched for evidence of quality of reporting of ultrasound studies in RMDs. The choice was made to focus on an extensively studied topic, that is, ultrasound assessment of synovitis in rheumatoid arthritis. In PubMed Clinical Queries, a broad search was performed; 80 studies were randomly selected and divided in four categories: diagnosis, aetiology, prognosis and therapy. The articles were summarised in table format to highlight objective, design, technical data, measures and outcomes (online supplemental file 1). These tables were sent to each member of the taskforce prior to the first face-to-face meeting, with the request to identify possible sources of bias and error and the absence of information considered important for the generalisability of the results. During the first face-to-face meeting, the members of the taskforce discussed the results and the unmet requirements in the selected literature, agreed on the format of presentation of the project (checklist or statement document) and elaborated a first list of items to be included. Other objectives of this meeting were the definition of a target audience and the need for systematic reviews. After the meeting, a number of focused literature reviews addressed specific issues; a summary of their results, along with the total list of items, were subsequently sent to the taskforce members. Relevance and comprehensibility of each proposed item were tested in a Delphi exercise, first by the taskforce members (excluding the convenor, EULAR methodologist and fellow), then by a panel of external experts chosen from the fields of musculoskeletal imaging, epidemiology and methodology, including journal editors. External experts were also asked if no key aspects were missed (comprehensiveness). During the second face-to-face meeting, the optimal format of the checklist document was established. Inclusion of each item was either supported by empirical evidence, when available, or by consensus within the task force, that the information requested by the item was methodologically important to assess in a study, as recommended by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) ‘guidance for developers of health research reporting guidelines’.17 In the same way, it was agreed not to include a level of evidence for each proposed item. The external experts were then invited to apply the checklist to a selection of ultrasound articles and to comment on its feasibility and comprehensibility; this resulted in minor modifications to the items. Finally, an online Delphi survey was performed among the taskforce experts to obtain their level of agreement with each final item, including each term of an accompanying glossary, included to define the checklist terminology. Agreement was assessed with a Likert scale, grading from 0 (total disagreement) to 10 (full agreement). Consensus was defined as a mean agreement ≥7 and with at least eight responders (2/3 of participants) having an agreement ≥7.
Supplemental material
Process flowchart of the project.
Results
Figure 1 shows the flowchart of the project. During the first face-to-face meeting, a preliminary checklist of 43 items was established, and three scoping reviews were requested on factors potentially influencing the ultrasound evaluation and therefore the generalisability of the results: (a) contextual factors (eg, smoking, temperature), (b) machine quality (eg, device, settings) and acquisition methods (eg, joint or transducer position) (online supplemental file 2).
Supplemental material
A first Delphi exercise helped decide which of the 43 items should be considered as ‘mandatory’ (always reported in every ultrasound study) or ‘optional’ (reported only according to specific study designs). After two voting rounds, several items were rephrased, deleted or combined, resulting in a checklist of 17 ‘mandatory’ and 8 ‘optional’ items (figure 1).
This new checklist was distributed among 218 external panellists (external Delphi exercise): 123 experts in muscoloskeletal imaging, 67 in epidemiology, 7 in methodology as well as 21 journal editors. Seventy-nine of them (36%) were participated. The external experts rated the initiative as very important (96%), the checklist comprehensive (95%), and all items were considered clear by the majority of them (median: 96%, range: 86%–96%). Additional suggestions were made to clarify some terminology. The results were discussed during the second face-to-face meeting, where the format of the checklist was agreed. Each item was verified to ensure comprehensibility, and a preliminary glossary including 12 terms was prepared. After this process, the checklist included 23 items (21 ‘mandatory’ and 2 ‘optional’) organised into 13 categories. This version and the glossary were distributed to the 79 external experts who had participated in the previous evaluation. Twenty-nine (37%) of them agreed to test the new checklist on additional selected articles and to comment on the comprehensibility and comprehensiveness of the items and the glossary as well as on the feasibility of applying the checklist. The median time needed to assess the articles for reporting the checklist items was 30 min (range 10–240). Comprehensibility was assessed as good with a median of 8 (range 1–10); and additional suggestions on the glossary terminology were made. The convenor and fellow incorporated all suggestions, and the final product (checklist with accompanying recommendation guidelines and glossary) was submitted to the taskforce members. Two rounds of an additional Delphi exercise (the second) were needed to obtain a final agreement on each item of the checklist and each term of the glossary.
The final checklist, composed of 23 items (21 ‘mandatory’ and 2 ‘optional’), organised into 13 categories, along with the level of agreement for each item, is reported in table 1; the accompanying glossary and the recommendation guidelines on how to use and interpret each item in the following paragraphs are presented in table 2.
Recommendations checklist for reporting studies using ultrasound in rheumatic and musculoskeletal diseases
Glossary
Target audience and when to apply the recommendations
The target audience comprises health or scientific researchers reporting or assessing observational and interventional studies using ultrasound in RMDs: that is, rheumatologists, radiologists and healthcare professionals using ultrasound, manuscript reviewers, grant applicants and reviewers, journal editors. Each mandatory item of the checklist should be considered as essential to be reported in every ultrasound study regardless of the purpose of the study. Such a report will allow proper appraisal of the validity and applicability of the results. The checklist is meant to be applied whenever ultrasound is used (investigation of measurement properties, diagnostic or prognostic accuracy and therapeutic studies). It is focused specifically on ultrasound issues and is neither intended to be totally comprehensive for all study design issues, nor intended to replace other existing reporting guidelines (eg, RCT, observational diagnostic studies, etc).
General items
The first six items on ‘objective’, ‘design’, ‘participants’ and ‘blinding’ as well as items 20 (‘statistical analysis’) and 21 (‘disclosures’) are not specifically related to ultrasound and some of them have already been included in other reporting checklists. However, the taskforce members felt it was essential to include them in this checklist to emphasise their importance. For example, the objective of the ultrasound measurement in a study might be different from the main objective of the study, and then such difference should be clearly described. Another example is the blinding of the ultrasound evaluation. Blinding is of utmost importance, especially in diagnostic or therapeutic studies, since the lack of patient or operator blinding can influence the results.18 19 Item 20 (‘statistical analysis’) refers mainly to the way in which the analysis of ultrasound variables should be performed, for example, the importance to clearly state whether analyses are performed at patient or at joint/site level.
Ultrasound features
Item 7 refers to the ultrasound definition of the pathological lesions under study. It is crucial to be able to check for consistency between what authors say they want to measure (eg, erosions (target domain) as a measure of structural damage (broad domain of interest)) and what was really measured with ultrasound. The concepts of broad and target domains are explained in table 2. In such cases, reporting the domain components (ie, elementary lesions) really measured by ultrasound will help evaluate consistency. A precise definition of ultrasound elementary lesions used in the study helps to check whether ultrasound is able to measure what it is supposed to measure (domain match). Here we used the terminology proposed by OMERACT for the development of imaging outcome measurement instruments,20 and in particular of ultrasound.5 In recent years, the OMERACT ultrasound working group, frequently in collaboration with EULAR, has undertaken considerable efforts to develop and improve definitions of ultrasound elementary lesions for a defined pathology (eg, synovitis, enthesitis, bone erosion).5 8
Scanning/acquisition procedures
Several sources of variability may affect the reliability of ultrasound measurements and generalisability of the study results. These include the quality of the equipment, positioning of patient and transducer and training of the examiner. One of the scoping reviews (online supplemental file 2) studied whether acquisition methods (ie, joint or transducer position and dynamic acquisitions) affected the reliability and accuracy of ultrasound. All retrieved studies confirmed the importance of a standardised joint position for the reliability and generalisability of the results; this applies to all anatomical sites (eg, knee, wrist, Achilles tendon, etc) and all target pathologies under study (eg, synovitis, joint effusion, etc),.21–26 In addition, appropriate transducer manipulation is needed to avoid artefacts.23–25 For example, transducer pressure may cause the synovial hypertrophy or Doppler signal to disappear or be reduced.27 The 2001 EULAR guidelines for performing ultrasound in rheumatology (updated in 2017) addresses both joint (and patient) positioning as well as transducer use.28 29 Authors are invited to refer to these latest guidelines in their studies. Items 8 and 9 describe what details should be provided about the scanning and acquisition procedures to assess compliance with these guidelines.
Ultrasound scoring systems
Items 10, 11 and 12 focus on a clear description of ultrasound scoring systems, especially if a newly developed scoring system is used. Special attention should be paid to the documentation of the development of the scoring system. As ultrasound is frequently considered the most operator-dependent imaging technique, intra-rater and inter-rater reliability is an important concern and a strong argument for standardisation. For new scoring systems, results of intra-sonographer and inter-sonographer/reader reliability studies should be reported. For existing scoring systems, reference to previous reliability studies should be given as well as the results of reliability assessments among the sonographers/readers in the context of the study.
Sonographers/readers
Depending on the setting, the person who performs the ultrasound acquisition of the images (sonographer) can be a healthcare professional or a medical doctor (radiologist or rheumatologist). The images can be interpreted at the time of acquisition or later, by the same or another person. Choices made here affect ultrasound scores and generalisability, so details on who performs the acquisition and the interpretation are a mandatory reporting requirement. Item 14 on the experience of sonographer(s) and/or reader(s) is optional, mainly because no consensus exists on how to report such experience. EULAR and American College of Rheumatology suggest a competency assessment in ultrasound to improve the quality of the examination.30 31
Equipment
Technical characteristics of the imaging device (item 15), ultrasound modalities and settings used (item 16) may affect the intrasonographer and intersonographer/reader reliability and generalisability of the results. A second scoping review (online supplemental file 2) addressed this question, that is, whether the ultrasound device (model, age, acquisition software, transducers and settings) affect the reliability or accuracy of the ultrasound examination. We found no study investigating the influence of device age or software on ultrasound results. However, seven studies assessed the influence of the machine (eg, ultrasound device, transducer frequency, settings) on ultrasound results whatever the anatomical site studied, the ultrasound modality used (ie, Doppler, GS) or the target pathology under study (eg, joint effusion, synovitis, erosion).32–38 Three studies used a phantom to compare the ultrasound devices.34 36 37 Five of the seven studies showed differences in the performance between machines and therefore an influence on the study results.35 36 However, in these studies, the relevance and the magnitude of such differences were reported, but no sensitivity analysis was conducted.
Images, pictures and figures
Since ultrasound is a tomographic imaging modality, the appearance of the structures may change following the orientation and position of the transducer. Standardised images should always be presented (item 17) so that the reader can easily recognise the anatomical structures as well as the target pathology and elementary lesions described in the study. The use of drawings can facilitate the interpretation of the images for readers not experienced in ultrasound. Images should never contain patient information and should be accompanied by clear legends and points of reference.
Contextual factors
Item 18 deals with the feasibility of ultrasound, in particular, the time necessary for the evaluation, which depends on the number of sites (or joints) examined and the number of ultrasound examinations performed over the duration of the study. Although these aspects are highly important for the acceptability of the technique, the taskforce members felt that time spent should be reported only if relevant to the study question.
Item 19 refers to additional potential sources of variability: ambient and patient conditions. This item was made optional because the third scoping review (online supplemental file 2) failed to find strong evidence of the influence of these factors on the ultrasound results. It reviewed the effect of three ambient conditions (room temperature, atmospheric pressure and time of the day) and five patient conditions (exercise, skin temperature, smoking, alcohol consumption and caffeine) on ultrasound measurements. There was a potential influence of time of the day on Doppler signal evaluation (with contradictory effects)39 40 and on GS results41; and potential influences on Doppler signal following the application of cold (ice and cold water)42–44 and after physical exercise.45–47
Discussion
This EULAR taskforce developed a recommendation checklist to ensure transparent and comprehensive reporting of ultrasound research and procedures aspects, which may affect the interpretation and generalisability of the results. The checklist consists of 23 items (21 ‘mandatory’ and 2 ‘optional’), organised into 13 categories. Its organisation allows authors to choose the order and format for presenting information, depending on their preferences and on journal style. Content validity of the recommendations checklist was confirmed by a panel of external experts, who considered each item of the checklist an essential reporting point, crucial to make an informed judgement on the quality of the scientific report. Moreover, all items were considered comprehensible and the checklist as a whole was considered to comprehensively cover all relevant reporting issues.
Along with sufficient content validity of this checklist, additional strengths include the development process that followed EULAR SOPs for a stepwise consensual approach16 and the guidance from the EQUATOR network,17 also, the panel members reflected a wide range of expertise and stakeholders. In addition, agreement on comprehensiveness and comprehensibility of the checklist was obtained in the first round of voting for all items of the checklist and all definitions of the glossary.
A possible limitation of this project is the fact that the face-to-face meetings comprised mostly Europeans, with only one colleague from USA, and only one patient. We partially overcame this in the external Delphi panels, including more international experts, including several radiologists.
The checklist was purposefully focused and is complementary to other existing guidelines, depending on the study design. It has not been developed as a tool to assess the quality of published research; however, it can certainly serve as a basis to develop such a tool, and its use may improve the quality of studies, as seen with other reporting recommendations.48 49 We hope that this reporting checklist will be widely adopted by authors and editors, which, in turn, will greatly improve the interpretability, reproducibility and generalisability of the study results.
Supplemental material
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Acknowledgments
The authors want to thank all contributing experts who participated in the online survey (see online supplemental file 3 for the full list).
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Handling editor Josef S Smolen
Twitter @carmona_loreto, @FerreiraRJO
MB, PVB, GAB, RC, PGC, RJOF, JLG-C, FG, HBH, DvdH, AI, MCK, RBL, PM, EN, WAS, LT, CBT and RT contributed equally.
Contributors Full-text review, data abstraction and Delphi assessments were performed by FC, supervised by MADA and independently double-checked by LC. MADA and LC supervised the methodology of the scoping literature review and FC prepared the evidence report. FC and MADA prepared the first draft of recommendations, and all authors participated in the discussion and formulation of recommendations. MADA supervised the project and FC, MADA, LC, MB and PGC drafted the manuscript. All authors reviewed the manuscript and approved its final version.
Funding This project was funded by the European League Against Rheumatism (EULAR). PGC is supported in part by the UK National Institute for Health Research (NIHR) Leeds Biomedical Research Centre. RC is supported by a core grant from the Oak Foundation (OCAY-18–774-OFIL).
Disclaimer The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
Competing interests FC reports personal fees from Lilly and Novartis France. MB reports personal fees from Novartis, BMS and Pfizer. RC is a founding member of the Technical Advisory Group of OMERACT, an organization that develops outcome measures in rheumatology and receives arms-length funding from 12 companies. DvdH reports personal fees from AbbVie, Amgen, Astellas, AstraZeneca, Bayer, BMS, Boehringer Ingelheim, Celgene, Cyxone, Daiichi, Eisai, Eli-Lilly, Galapagos, Gilead, Glaxo-Smith-Kline, Janssen, Merck, Novartis, Pfizer, Regeneron, Roche, Sanofi, Takeda, UCB Pharma and is Director of Imaging Rheumatology bv. AI reports grants from Abbvie, MSD, and Alfasigma and personal fees from AbbVie, Abiogen, Alfasigma, Biogen, BMS, Celgene, Eli-Lilly, Janssen, MSD, Novartis, Sanofi, Sanofi Genzyme. RBL reports personal fees from AbbVie, Galapagos, Gilead, Jansen, Eli-Lilly, Novartis, Pfizer, UCB. PM reports grants and personal fees from AbbVie, Novartis, Janssen, personal fees from Celgene, grants from Merck Sharp & Dohme, UCB, Roche. RT reports personal fees from Amgen, AbbVie, Novartis. MADA reports personal fees from Abbvie, BMS, Novartis, Celgene, Janssen and grants from Pfizer.
Provenance and peer review Not commissioned; internally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.