Article Text
PDF
Abstract
In this article, the European League Against Rheumatism (EULAR) standardised operating procedures for the elaboration, evaluation, dissemination and implementation of recommendations endorsed by the EULAR standing committees published in 2004 have been updated. The various steps from the application to implementation have been described in detail.
- Epidemiology
- Qualitative research
- Outcomes research
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
In 2004, standardised operating procedures (SOPs) for the elaboration, evaluation, dissemination and implementation of recommendations endorsed by the European League Against Rheumatism (EULAR) standing committees were published.1 These SOPs applied to recommendations on classification and/or diagnostic criteria, clinical trial conduct, management of specific musculoskeletal disorders and standardisation of procedures. Many sets of recommendations have been published. For example, 27 sets of management recommendations have been published, 25 since 2005. Since their introduction, the SOPs have operated ‘to maintain and to homogenise a high level of intrinsic quality and comparability of such studies’.1 A recent review of the quality of published EULAR management recommendations has concluded that the overall quality of recommendations has been good with improvements seen over the last decade. However, the review also identified potential areas for improvement especially in patient representation and the provision of implementation tools.2 This review of the quality of EULAR recommendations, the milestone of a decade since publication of the first SOP, the experience from many recommendations and evolving insight stimulated the EULAR executive committee (EC) to update the SOPs.
The overall organisation of the SOP has been maintained in its original form. For clarity, the new version is presented here in total, even where there is no change from the original SOP. These SOPs are intended for musculoskeletal disorders, either in general (eg, inflammatory diseases) or specific (eg, rheumatoid arthritis).
Projects to define recommendations with the ultimate goal of obtaining EULAR endorsement can be submitted to the appropriate EULAR standing committee. In general, management and monitoring recommendations, classification/diagnostic criteria, diagnostic approaches, clinical trial conduct and outcome measures should be submitted to the EULAR Standing Committee of Clinical Affairs (ESCCA). Imaging-specific projects should be submitted to the EULAR Standing Committee of Imaging, whereas projects related to standardisation of (laboratory) procedures should be submitted to the EULAR Standing Committee of Investigative Rheumatology. However, depending on the exact topic of the project, applications can also be submitted to the EULAR Standing Committees of Epidemiology, Paediatrics, Patients and Health Professionals. At the beginning of a project it should be considered whether the project is best organised as an independent EULAR project or that collaboration with another society would improve its quality and reach.
The most current version of the SOP for the application process, including budgeting, and application forms are available on the EULAR website (http://www.EULAR.org). These should always be checked and used before submission of a proposal. The EULAR website also contains a list of all recommendations that have been published or are under development.
The first draft of the update has been prepared by the chair and past-chair of ESCCA together with the first author of the original SOP. Thereafter, input was sought from all members of the EULAR EC, ensuring that representatives of all stakeholders (rheumatologists, health professionals, patients, methodologists and basic scientists) were consulted.
We describe one by one the key elements required for a project to develop recommendations. The steps are also summarised in a flow diagram (figure 1).
Flowchart of various steps during development of recommendations. AGREE, Appraisal of Guidelines for Research & Evaluation; ARD, Annals of Rheumatic Diseases.
Which wording?
There are two options for the wording: ‘recommendations’ and ‘points to consider’. Most of the projects in the past fell under the category of recommendations. This is selected for projects that can be viewed as providing advice, and if applied, as a measure of good quality. Usually, this is used for a project when there is sufficient evidence from the literature, so that the recommendations can be, at least partly, data driven. Points to consider are preferred for those projects where the advice cannot be substantiated by data or for clinical areas where the term ‘recommendation’ is simply avoided. In the first SOP the term ‘guidelines’ was offered. However, ‘guidelines’ are perceived as too strict, which might lead to legal implications if not followed. This wording has never been used and is also no longer an option. In the remainder of the document, the term ‘recommendations’ is used but this is equally applicable to ‘points to consider’.
Which category?
The SOPs could be applied to the following categories:
recommendations for conducting clinical studies, including clinical trials and observational studies
recommendations for management and/or monitoring in clinical practice
recommendations for diagnostic approaches and/or imaging
recommendations for standardisation of procedures
recommendations for implementation.
In the previous SOPs, projects on classification/diagnostic criteria, and definition of outcome measures, for example, responder criteria were included. As the methodology to define such outcomes is so different from the recommendation projects mentioned here, we have decided not to include these in the current SOP. However, it should be acknowledged that several areas of guidance within this SOP also apply to these projects (such as the representation of countries in a project). The fact that these projects are not referred to in the SOP does not in anyway suggests that these are not of interest to EULAR. Consequently, they can still be submitted to the respective standing committee for consideration.
Target population
From the start of the project the target population should be determined. The target population is the group of people that will be interested in the recommendations and that should be made aware of their existence. So the definition of the target population will facilitate the project from design to implementation. A project might have multiple target populations. Examples are rheumatologists, general practitioners, health professionals (in the entire document health professionals are professionals managing patients with musculoskeletal diseases excluding medical doctors), medical students, patients, (inter)national drug agencies, pharmaceutical companies, policy makers and health insurance companies. Although the content of the recommendations will not vary, the presentation, dissemination and implementation may need to be adjusted for the various target populations (eg, lay versions of recommendations for patients).
Task force
Each task force assembled to develop a recommendation consists of the convenor, methodologist, fellow, clinical experts, health professionals and patients. Typically, the total task force will have 18–24 members. To ensure full transparency, each member of the task force must disclose potential conflicts of interest before the start of the project, regardless of whether they may be deemed a conflict with regard to the specific project. The requirements for each member of the task force will be detailed below.
The convenor of the project is usually a person with a high level of expertise in the topic and is the chair of the task force. This person provides liaison between EULAR and the task force and bears responsibility for the content and financial aspects of the project. While complying with the rules mentioned below, the convenor has the full freedom to select the individuals for the task force. If needed, a co-convenor can be appointed, but the convenor remains the single responsible person. It is preferred that the convenor is not a member of the EULAR EC.
The methodologist works in close collaboration with the convenor. The convenor and methodologist submit the application together as a sign of the agreement with the proposed design by the methodologist. The methodologist is a person who has expertise in performing both a systematic literature search and the development of recommendations, which can be proven by a scientific track record, and—if applicable—experience in the data analysis techniques required for a project. The methodologist, preferably not a member of the EULAR EC, is not necessarily an expert in the topic of the recommendations. The methodologist supervises the systematic literature search and will co-chair all meetings and is in charge of the methodological aspects of the various steps of the project.
At least one fellow will be an integral part of the task force and will perform the systematic literature search under supervision of the methodologist, and—if applicable—perform data analysis in relation to the project. The fellow is usually a junior researcher, doctor or health professional, working in the institute of the convenor or methodologist.
Clinical experts are invited by the convenor and should be representative of the European rheumatological community. At least five different European countries should be represented, well spread over Europe. However, experts outside Europe may also be asked to take part. Rheumatologists will form the majority of those invited, but non-rheumatologists, such as other medical specialists, health economists or representatives of national agencies, can be invited depending on the expertise needed in the specific task force. The clinical experts should be top leaders in the field. But in addition, each task force should include two young rheumatologists, members of EMEUNET (The Emerging EULAR Network), to promote education in the methodology of the development of recommendations (http://www.emeunet.eular.org/). Selection should be made after consultation with the chair of EMEUNET.
A minimum of one health professional should be part of the task force. Based on the topic of the project, there may be a preference for a nurse, physiotherapist or other health professional. This health professional will take part in all meetings and steps of the project.
A minimum of two patients, preferably with the disease under study, should be part of the task force and will take part in all meetings and steps of the project.3 Identification of eligible patients can take place through the clinics of participating task force members or through the EULAR network of patient research partners.
Collection of evidence
The GRADE system should be used for guidance throughout the elaboration of the recommendations.4 A systematic literature review (SLR) should form the basis of the project. Preferably, guidance as provided by the Cochrane group or the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement will be followed for the various steps of the SLR.5 ,6
The first step is to define the research questions. This is usually done during the first meeting of the task force, for example, if the project is on management, it needs to be defined which management aspects will be addressed (eg, treatment (pharmacological and/or non-pharmacological); monitoring). After the meeting these clinical questions will be changed into epidemiological questions by the convenor, methodologist and fellow. In addition, the outcome measures that will be extracted need to be defined (eg, ACR20 response rates, mean change in Simple Disease Activity Index (SDAI), etc.). The databases that will be used for the search should be defined with a minimum inclusion of MedLine/PubMed and the Cochrane Library (the latter in case of management recommendations). In addition, further evidence may need to be collected prior to the first meeting (eg, compilation of existing guidelines), to provide all participants with a background to start discussions.
Special attention should be paid to collecting data on cost-effectiveness when developing management recommendations and should be an integral part of the SLR.
The quality of all retrieved papers needs to be assessed by validated assessments tools. Examples are the Cochrane Risk of Bias tool for all studies and the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) II for diagnostic accuracy studies and the Quality In Prognosis Studies (QUIPS) for observational studies.5 ,7 ,8
Depending on the type of project, various analyses to quantify the results should be performed to be able to aggregate results from the included studies and obtain a pooled effect size (meta-analysis) and to compare results or explain heterogeneity (meta-regression or sensitivity analysis).
Formulation of recommendations
During the second meeting of the task force, the results of the SLR and—if applicable—of the original data analysis will be presented. The next step will be the wording of the recommendations. Normally, a maximum of 10 recommendations will be defined. This might be adjusted depending on the project, but too many recommendations are likely to result in a loss of focus and might be problematic in the implementation process. Recommendations should in principle be based on the data obtained from the SLR. The evidence for each recommendation should be categorised. This has been clearly defined for the evaluation of treatment in table 1. This labelling will mostly apply to the SLR; any recommendation driven by original data analysis will most likely fall into category 3, as they typically represent secondary analysis of available datasets. The GRADE system can also be used to rate the evidence for recommendations related to efficacy.4 The evidence for other types of recommendations should be rated by the Oxford Levels of Evidence, which define the level of evidence based on the type of research question.9
Categories of evidence9
The next step is to define the ‘strength of recommendation’ for each recommendation (table 2).9 This is a combination of the information from the SLR (categories of evidence) and expert opinion. Especially, in situations when there are no published data available for a recommendation (there are no positive and no negative data, the topic has not been thoroughly investigated), expert opinion plays an important role. If it turns out that for a recommendation project there is little data-driven evidence, it is better to downgrade the recommendations to the level of ‘points to consider’.
Strength of recommendations
Presentation of recommendations
The main text of the manuscript of the recommendations will describe the entire process of development and all recommendations in detail. However, for dissemination purposes it is important to present the recommendations in an easy to understand way (eg, slide presentations). Recommendations on management or diagnosis/monitoring can be presented as (1) different short sentences (bullets) or as (2) an algorithm. The bullet list is most appropriate for management recommendations while the algorithm can be useful for recommendations on a diagnostic process. A special consideration is the development of a lay version of the recommendations for patients. Ideally, this would be part of the original project, but frequently this can only be completed as a separate project afterwards with involvement of a larger group of patients. EULAR has taken the initiative to develop recommendations for the process of adapting EULAR recommendations into lay versions.
Evaluation of the recommendations
The first evaluation of the recommendations is performed as part of the original development of the recommendations. After the recommendations have been defined and the level of evidence and strength of recommendation have been added to each recommendation, every member of the task force will indicate their level of agreement with each recommendation. This is scored on a numerical rating scale ranging from 0 (completely disagree) to 10 (completely agree). This step can be done by electronic communication. The average (SD) and the range of scores are presented in the manuscript. An additional way of presenting the data is the percentage of task force members providing a score above a certain threshold, for example, 6 or 7.
External evaluation, by persons not part of the task force, can be performed following the Appraisal of Guidelines for Research & Evaluation (AGREE) instrument and is frequently separate from the original publication.10 The latest version is AGREE II, which is a refinement of the original AGREE instrument. This is a list of six domains with in total 23 aspects that are checked to assess the quality and robustness of the recommendations (box 1). The AGREE instrument has two final questions, rating the overall quality of the recommendations (score range from 1 to 7) and advice to use the recommendations (yes, yes with modifications, no). There is a user's manual with clear instructions on how to use the instrument. Although the final evaluation according to the AGREE instrument should be performed by independent reviewers, it is strongly recommended that the AGREE instrument is used in the planning of the project and also in the writing of the manuscript. Frequently, a task force adheres to quality standards, but this is often not well documented in the manuscript and consequently rated as missing in the external evaluation.
AGREE II instrument
AGREE II domains and items
Domain 1. Scope and Purpose
The overall objective(s) of the guideline is (are) specifically described.
The health question(s) covered by the guideline is (are) specifically described.
The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described.
Domain 2. Stakeholder Involvement
The guideline development group includes individuals from all the relevant professional groups.
The views and preferences of the target population (patients, public, etc.) have been sought.
The target users of the guideline are clearly defined.
Domain 3. Rigour of Development
Systematic methods were used to search for evidence.
The criteria for selecting the evidence are clearly described.
The strengths and limitations of the body of evidence are clearly described.
The methods for formulating the recommendations are clearly described.
The health benefits, side effects and risks have been considered in formulating the recommendations.
There is an explicit link between the recommendations and the supporting evidence.
The guideline has been externally reviewed by experts prior to its publication.
A procedure for updating the guideline is provided.
Domain 4. Clarity of Presentation
The recommendations are specific and unambiguous.
The different options for management of the condition are clearly presented.
Key recommendations are easily identifiable.
Domain 5. Applicability
The guideline describes facilitators and barriers to its application.
The guideline provides advice and/or tools on how the recommendations can be put into practice.
The potential resource implications of applying the recommendations have been considered.
The guideline presents monitoring and/or auditing criteria.
Domain 6. Editorial Independence
The views of the funding body have not influenced the content of the guideline.
Competing interests of guideline development group members have been recorded and addressed.
AGREE, Appraisal of Guidelines for Research & Evaluation.
Finally, the target population should also perform an evaluation. In case of management recommendations, these could be rheumatologists working in various countries in academic and non-academic centres. For recommendations on diagnosis, general practitioners could be appropriate to evaluate the recommendations. And in the case of recommendations on the conduct of clinical studies, stakeholders such as pharmaceutical companies and registration agencies are important to include in the evaluation process. Another aspect to consider is the potential use of the recommendations in undergraduate and postgraduate education of doctors and other health professionals.
Dissemination of the recommendations
A strategy to disseminate the recommendations should be part of the original proposal. The minimum dissemination that is required is the submission of an abstract to the EULAR annual congress and the submission of a manuscript to the Annals of Rheumatic Diseases, the EULAR journal, for consideration of publication. The convenor is the single person responsible for the list and ranking of the authors of the publication and contact with the editor. It is important to discuss at an early stage with the editor how the recommendations should be submitted. In principle, there should be a single manuscript including all the different aspects. It might be necessary to publish the details of the underlying SLR online. If there is an intention to publish the SLR as a separate manuscript, this should be discussed with the editor upfront and in consultation with the editor the best publication strategy should be defined. If the manuscript on the recommendations is accepted for publication in the Annals of Rheumatic Diseases, open access will be offered free of charge.
Another dissemination option is to send the recommendations to a high number of the target population, for example, rheumatologists in various countries in case of management recommendations. The rheumatologists can be asked about their agreement with each recommendation in a similar way as described for the task force members. In addition, they can be asked to indicate whether the specific recommendation will change their practice, and if it will not change their practice, is this because they disagree or because they already apply this in clinical practice. Finally, expected or actual barriers to the implementation can be listed. This will provide essential information to develop an implementation project.
Other ways of dissemination to be considered are presentation by key opinion leaders, the inclusion in national recommendations and continuing education programmes.
Implementation of the recommendation
Implementation is the process by which targeted users integrate the recommended actions into daily clinical practice. Only with successful implementation, can the recommendations lead to the desired changes of increased quality of care and harmonisation of approach. Implementation starts with knowledge about the recommendations. Therefore, dissemination is the crucial first step. However, although knowledge is essential, implementation needs more active involvement when a fundamental change in the habits of rheumatologists is needed. For successful implementation, a separate project is usually necessary, which selects one or more established implementation strategies. Examples are inclusion of recommendations in quality indicators, reimbursement dependent on fulfilment of certain recommendations and audit feedback. There is a specific science on implementation and examples can be found in the literature.11
Update policy of the recommendations
It is useful to indicate at the end of the development of the recommendations process, when it is expected that an update of the recommendations is needed. This can be defined in years, but this might be difficult to define upfront. A better solution is probably to indicate a specific reason for an update being necessary, for example, when one (or more) new treatment options become available. Or when there are other important developments, which might change clinical practice. This should be monitored by the convenor, but also members of the original task force or experts in the field can indicate the need for an update and apply for this to EULAR.
Practical and financial aspects of the project
The most up-to-date information on the budget and practical information for running a project can be found on the EULAR website. In principle, there will be two meetings of the task force. Typically, the meeting starts in the afternoon or evening of day 1 and finishes in the afternoon of day 2. There is a maximum budget to organise each meeting. The list of attendees should fulfil the requirements as specified under ‘D’ and should be approved by the chair of the responsible standing committee. The EULAR secretariat will take care of the practical aspects of these meetings (hotel and meeting room reservation, meals and travel expenses).
The budget also covers a stipend for the fellow of the project. This is a limited amount and usually only covers a part-time salary for the time of the project. There will be no additional payments for any of the task force members. The most up-to-date information on what is covered by the budget can be found on the EULAR website (http://www.EULAR.org).
EULAR endorsement policy
Endorsement by EULAR can be sought after completion of the project. In principle, only projects that have been approved by EULAR from the beginning as described above, and therefore received financial support, can be submitted for endorsement. Basically, every project that has been approved and executed according to the protocol will receive endorsement. However, any deviation, which has not been approved, can lead to refusal of endorsement of the final set of recommendations. The convenor together with the methodologist should send an annual report on the progress of the project to the chair of the standing committee. The chair of the standing committee should approve the abstract intended for submission to EULAR before submission. It is expected that this abstract be sent to the chair by at least 2 weeks before the submission deadline for EULAR to allow changes to be made if necessary. The final manuscript should also be sent to the chair of the committee for approval. The chair will check that the correct procedures were followed. After this approval has been obtained, the EULAR secretariat will send the manuscript to all members of the EULAR EC. These members have a maximum of 2 weeks to answer two questions: (1) does the paper confer with EULAR rules and objectives? (2) does the paper reflect the objectives of the task force? If there is no response, it is assumed that there is no objection.
We hope that these updated SOPs will further strengthen the quality and harmonisation of future EULAR recommendations.
Acknowledgments
EULAR secretariat for all logistical support.
Footnotes
Handling editor Hans WJ Bijlsma
Contributors DvdH prepared the first version of the manuscript. All the authors reviewed the draft versions and gave their approval of the final version of the manuscript.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.