You are here

Article Text

Article menu

Download PDFPDF

Original research
Effects of evidence-based clinical practice guidelines on quality of care: a systematic review
  1. M Lugtenberg1,
  2. J S Burgers2,
  3. G P Westert1,3
  1. 1
    Department of Tranzo, Tilburg University, Tilburg, The Netherlands
  2. 2
    Scientific Institute for Quality of Healthcare (IQ Healthcare), University Medical Centre St. Radboud, Nijmegen, The Netherlands
  3. 3
    National Institute for Public Health and the Environment (RIVM), Bilthoven, The Netherlands
  1. Correspondence to Marjolein Lugtenberg Msc, Department of Tranzo, Tilburg University, PO Box 90153, 5000 LE Tilburg, The Netherlands; m.lugtenberg{at}uvt.nl

Abstract

Background: Evidence-based clinical guidelines aim to improve the quality of care. In The Netherlands, considerable time and effort have been invested in the development and implementation of evidence-based guidelines since the 1990s. Thus far, no reviews are available on their effectiveness. The primary aim of this article was to assess the evidence for the effectiveness of Dutch evidence-based clinical guidelines in improving the quality of care.

Methods: A systematic review of studies evaluating the effects of Dutch evidence-based guidelines on both the process and structure of care and patient outcomes was conducted. The electronic databases Medline and Embase (1990–2007) and relevant scientific journals were searched. Studies were only selected if they included a controlled trial, an interrupted time series design or a before and after design.

Results: A total of 20 studies were included. In 17 of 19 studies that measured the effects on the process or structure of care, significant improvements were reported. Thirteen of these studies reported improvement with respect to some of the recommendations studied. In addition, the size of the observed effects varied largely across the recommendations within guidelines. Six of nine studies that measured patient health outcomes showed significant but small improvements as a result of the use of clinical guidelines.

Conclusions: This review demonstrates that Dutch evidence-based clinical guidelines can be effective in improving the process and structure of care. The effects of guidelines on patient health outcomes were studied far less and data are less convincing. The high level of variation in effects across recommendations suggests that implementation strategies tailored to individual recommendations within the guideline are needed to establish relevant improvements in healthcare. Moreover, the results highlight the need for well-designed studies focusing on the level of the recommendations to determine which factors influence guideline utilisation and improved patient outcomes.

https://doi.org/10.1136/qshc.2008.028043

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Increasingly, clinical practice guidelines (CPGs) are being developed in all areas of medicine as a means to improve the quality of care. By translating the best available scientific evidence into specific recommendations, guidelines can serve as useful tools to achieve effective and efficient patient care.1 Whereas guidelines initially were based on consensus among experts, guideline development has been gradually formalised and evidence-based guidelines—linking the individual recommendations with their supporting evidence—are becoming standard practice.2 Developing evidence-based guidelines, however, does not guarantee improved quality of care. Effective implementation should ensure guideline adherence in practice and subsequently lead to improved patient outcomes.

    Studies measuring the effects of guidelines on quality of care have predominantly focused on effects on clinical practice. Several international reviews showed that the majority of guideline studies resulted in significant improvements with respect to the process of care.345 Fewer studies have focused on the effects of guidelines on patient health outcomes. One review indicated improved patient health outcomes,3 whereas a second review, focusing on primary care, did not find a positive effect.6 However, most of the studies that were included in these reviews used older guidelines that were not developed according to the current standards of evidence-based medicine.7

    Guideline utilisation is complex and many factors may influence the impact of guidelines on care. Factors linked to the guideline itself are the strength of the evidence,89 the method of development and transparency of the guideline10 and the perceived adoptability, complexity and triability of the recommendations.1112 Apart from guideline factors, the nature of the implementation strategy can contribute to guideline utilisation. Passive strategies, such as educational material and meetings, generally have a small effect and multifaceted strategies are not necessarily better than single interventions.113 Finally, contextual, organisational and cultural factors may impede or limit guideline adoption regardless of how thoroughly they are implemented.1415

    The Netherlands has been a forerunner in evidence-based guideline development and guideline implementation research, compared with other European countries.16 Since 1982, more than 200 guidelines have been developed by the Dutch Institute for Healthcare Improvement (CBO) and the Dutch College of General Practitioners (NHG), the two most prominent guideline organisations in The Netherlands. Historically, the CBO focused on secondary care and the NHG on primary care, since there is a clear-cut distinction between primary and secondary care in the Dutch healthcare system. In the last decade, other organisations have also become active in guideline development. Partly because of the role of the Centre for Quality of Care Research (since June 2008 Scientific Institute for Quality of Healthcare), many implementation studies have been conducted to measure the effectiveness of the Dutch guidelines.16

    In spite of a considerable investment in the area of evidence-based clinical guidelines in The Netherlands, thus far, it is unclear to what extent these activities have been successful in improving compliance with guidelines and patient health outcomes. By examining the impact of evidence-based guidelines in a country the size of The Netherlands, which features well-defined organisations responsible for guideline development, unique observations can be made. The primary aim of this study is, therefore, to provide an overview of the effectiveness of Dutch evidence-based guidelines in improving the quality of care. In addition, we want to explore which factors are associated with guideline utilisation and improved patient outcomes.

    Methods

    Concepts and definitions

    In this review, CPGs were defined as “systematically developed statements to assist practitioner decisions about appropriate healthcare for specific clinical circumstances.”17 Guidelines that use the results of systematic literature reviews in formulating the recommendations and that link the individual recommendations with their supporting evidence were regarded as evidence-based CPGs. A recommendation was defined as “any statement that promotes or advocates a particular course of action in clinical care.”18

    Implementation was defined as “a planned process and systematic introduction of innovations or changes of proven value; the aim being that these are given a structural place in professional practice, in the functioning of organisations or in the health care structure.”19 Dissemination, on the other hand, is regarded as more passive than implementation and involves strategies such as distributing guidelines or publication of guidelines in scientific journals.

    To evaluate effects on quality of care, we used Donabedian’s model, which distinguished the structure, processes and outcomes of care.20 Structure of care refers to “human, physical and financial resources that are needed to provide medical care” (eg, the presence of spirometry in general practice).21 Process of care refers to “the set of activities that go on within and between practitioners and patient” (eg, prescription of medication),21 whereas “the change in a patient’s current and future health status that can be attributed to antecedent health care” (eg, blood pressure) is defined as outcome of care.21

    Search strategy

    A systematic literature search was conducted in Medline, Embase and relevant Dutch scientific journals. Searches were performed in Medline and Embase of literature published from 1990 to May 2007 using several combinations of key-words (Appendix A). We did not include studies published before 1990, as evidence-based guideline development in The Netherlands started in the early 1990s. To identify Dutch-language publications we performed a sensitive search in Medline (1990–2007) with the free text word “guideline*”, limited to Dutch language. In addition, two relevant Dutch scientific journals, Huisarts & Wetenschap and Nederlands Tijdschrift voor de Geneeskunde, were searched for additional studies.

    Two reviewers (ML and JB) independently screened the titles and abstracts of the articles and selected 163 potentially relevant articles. Discrepancies were resolved by discussion and consensus. These articles were further selected according to the following inclusion criteria (fig 1).

    Figure 1
    Figure 1

    Flow diagram of study selection process (QUOROM)22

    1. The study concerned (a) clinical Dutch guideline(s). Drug formularies, patient guidelines and European guidelines were excluded.

    2. The study addressed the adherence to recommendations related to the process and structure of care and/or the effects of guidelines on patient health outcomes.

    3. The study concerned (an) evidence-based guideline(s). Local or regional protocols and guidelines that were derived from evidence-based national guidelines were also included.

    4. The study included a controlled trial (randomised controlled trial, controlled clinical trial), an interrupted time series or a before and after design. Studies that evaluated the effectiveness of different guideline dissemination and implementation strategies were included as well as studies that measured the effect of a guideline against a non-intervention control group.

    In addition, we consulted reference lists from all articles that were retrieved for more detailed information.

    Data extraction

    Three categories of studies were distinguished based on the target users of the guideline(s): (1) general practitioners; (2) medical specialists and (3) other healthcare providers, such as physiotherapists and midwives. The following data were collected from each study: type of guideline (national or local/regional); clinical area; setting; study design; number of included patients and physicians; type of intervention; process and structure measures; patient outcome measures; effects on process and structure of care; and effects on patient health outcomes.

    Methodological quality

    The methodological quality of the eligible studies was assessed by one reviewer (ML) and checked by a second reviewer (JB) using the quality criteria of the Cochrane Effective Practice and Organisation of Care Group (EPOC).23 The EPOC quality criteria checklist includes seven criteria for randomised controlled trials, seven criteria for controlled before and after studies and seven criteria for interrupted time series. Although the EPOC criteria were not developed to assess the methodological quality of uncontrolled before and after studies, we used them for these studies as well, since there are no high-quality checklists available to measure the quality of these type of studies. The quality criteria, such as concealment of allocation, follow-up of professionals and follow-up of patients or episodes of care, were scored as “done”, ”not done”, ”not clear” and in some cases as “not applicable.”

    Data synthesis

    Due to the heterogeneity of the studies, pooling of the results and calculating an overall estimate of the effects were not possible. Instead, we summarised the effectiveness in three categories: mostly effective (if the study demonstrated a significant effect on more than half of the outcome measures), partly effective (if the study reported a significant effect on half or less than half of the outcome measures) and not effective (if no significant effect was demonstrated). If a study evaluated the effectiveness of different guideline dissemination and implementation strategies, effectiveness was determined by the observed significant improvement in either of the study groups, rather than by a significant improvement in the intervention group compared with the control group.

    Results

    Description of the studies

    Setting, study design and type of intervention

    Thirty articles referring to 20 different studies fulfilled the inclusion criteria and were included in the selection. Most of the included studies targeted general practitioners (table 1).242526272829303132333435363738394041424344 Three studies were conducted among medical specialists454647 and four studies targeted other healthcare providers.484950515253

    View this table:
    Table 1

    Characteristics of included studies (n = 20)

    The majority of the studies concerned guidelines that had been developed at the national level.242526272829313234353637383940414243444547484950 Most studies focused on diagnosis and treatment,24313233383940434445464748495053 while five studies concerned preventive care25262728293042 and three a combination of both.34353637415152 The most common studied medical conditions were cardiovascular diseases, influenza, diabetes mellitus type II, low-back pain, asthma and COPD.

    Most of the studies were cluster randomised controlled trials.2433343536374041434446495053 The studies used predominantly multifaceted intervention strategies to implement the guideline(s).242526272829313233343536373839404142434446474950515253 The intervention strategies most often used as part of multifaceted interventions were educational meetings, distribution of educational material, and audit and feedback.

    Methodological quality

    Overall, the quality of the included studies was moderate. A priori calculations of sample size were reported in seven studies.24343536374344474950515253 In the majority of the studies that allocated study groups, practices or groups of healthcare providers were the unit of allocation,25263031323334353637404142464950 thereby protecting participants against contamination. In four studies, allocation was by individual provider.243839434453

    Five studies reported that more than 20% of providers dropped out.24334243444950 In four studies, data from less than 80% of patients were reported for at least one of the outcome measures or points in time.313233343536373839 Two studies reported an agreement in (some of the) outcomes between raters of less than 90% (or κ<0.8).464950 In most of the studies, however, reliability of outcome measures was not reported at all.242526272829303132343536374041424344454748515253

    Effects on quality of care

    Four studies evaluated the effectiveness of different dissemination or implementation strategies,304143444950 while 16 studies evaluated the effectiveness of a single implementation strategy (table 2).

    View this table:
    Table 2

    Summary of selected Dutch guideline studies addressing the effects on quality of care

    Effects on process and structure of care

    Of the 19 studies that examined effects of the guideline(s) on the process or structure of care, 17 showed significant improvements (table 2). The majority of these studies reported improvements with respect to some of the recommendations studied.242526282930313234353637404243444546515253 Only four studies showed an effect on all outcomes measuring adherence to the guideline.273839474950 In six studies, improvements were observed in half or less than half of the assessed process or structure of care measures.2431324043444653 Two studies failed to demonstrate any effect on the process or structure of care.3341

    Overall, the size of the effects varied largely across the recommendations within a guideline. Significant improvements in adherence to recommendations ranged from 7.2% to 88% in the 17 studies and varied by 76% across recommendations within one study. In studies including a control group, odds ratios ranged from 0.2 (95% CI 0.1 to 0.6) to 27.13 (95% CI 12.86 to 57.24).

    Effects on patient health outcomes

    Nine studies assessed the effects of guidelines in terms of patient health outcomes. Six of these studies reported significant improvements in at least some of the outcomes studied (table 2).31323435363743444849505152 Wolters et al4344 and De Laat et al5152 reported improvements in all outcome measures, while four studies313234353637484950 demonstrated modest improvements in some of the assessed patient outcome measures. In three studies,38394047 no effect on patient health outcomes was observed.

    Characteristics of the studies and effects on quality of care

    All studies focusing on preventive care were mostly effective in terms of the process or structure of care (table 3).25262728293042 With respect to type of design, all uncontrolled before and after studies that measured effects on process or structure of care272829455152 or on patient health outcomes485152 were categorised as mostly effective. In contrast, both studies that failed to demonstrate an effect on clinical practice were cluster randomised controlled trials.3341

    View this table:
    Table 3

    Characteristics of included studies (n = 20) and effects on quality of care

    There were no differences in effects on quality of care with respect to type of healthcare provider, type of guideline or between studies that used multifaceted intervention strategies and studies that used a single intervention.

    Discussion

    Overall, the results of this study demonstrate that there is evidence for the effectiveness of Dutch evidence-based guidelines on the process and structure of care in The Netherlands. The majority of the studies reported improvement with respect to some of the recommendations studied. In addition, the size of the effects varied largely across recommendations within the guidelines. The effects of guidelines on patient health outcomes were studied far less and data are less convincing. Two-thirds of the studies that measured patient outcomes reported significant improvements. However, the observed changes in patient outcomes were generally modest and only found for some of the outcomes studied.

    Findings from our review in terms of the process of care are comparable with those of previous international reviews which demonstrated small to moderate improvements.345 With respect to patient outcomes, results from earlier reviews were inconsistent.36 However, measuring patient outcomes is complex due to many factors such as long delays and confounding of many outcomes.5455 Our review provides some evidence for the effects of guidelines on patient outcomes. It also suggests that guidelines focusing on preventive care are particularly effective in improving the process or structure of care. Nevertheless, the number of studies in our review does not allow us to draw firm conclusions on the effects of guidelines on patient outcomes or on factors that contribute to improved quality of care.

    An important finding of our study is that the observed effects varied largely across recommendations. The variation could be explained by barriers related to individual recommendations rather than barriers that apply to the guideline as a whole. For example, although we selected only evidence-based guidelines in our study, the strength of the evidence may vary across recommendations, thereby influencing their impact.89 Also, a recommendation may not be performed because of other factors such as healthcare professional issues (eg, lack of motivation) or environmental factors (eg, lack of resources). Future research should focus on barriers related to both the guideline and its specific recommendations when exploring the association with effects on quality of care.

    Furthermore, the fact that the effects of guidelines varied largely across recommendations might suggest that guideline implementation should focus more on individual recommendations rather than the guideline as a whole. Whereas the majority of studies included in our review used multifaceted strategies to implement the guideline, these were generally not tailored to individual recommendations. The nature of the implementation strategy is often the same for all recommendations within a guideline. A more focused approach, based on the results of an analysis of barriers of adhering to individual recommendations could improve the use and effectiveness of guidelines in practice.

    One of the strengths of the present study is that it focuses on the effectiveness of evidence-based guidelines on quality of care. Previous reviews also considered guidelines that were not developed according to the standards of evidence-based medicine. In addition, in our study the effects of guidelines within one healthcare system were analysed. The guidelines in the studies included in this review were produced by well-known and credible organisations in The Netherlands and these guidelines generally have acceptable quality scores and are adequate tools for healthcare improvement.5657

    As stated before, The Netherlands has been a forerunner in evidence-based guideline development and implementation research in Europe. Based on the current literature, we have no reason to believe that the effectiveness of evidence-based guidelines, in terms of quality of care, is different in other countries or that more robust designs are generally used to assess these effects. Contextual and country-specific factors may, however, influence the effectiveness of evidence-based guidelines in terms of quality of care. In some countries, such as Scotland and New Zealand, guidelines are produced in a similar context as in The Netherlands, making the conclusions applicable to these countries as well.2

    Several limitations to the present study can be mentioned. First, despite the long tradition of evidence-based guideline development in The Netherlands, the number of studies measuring the effects of guidelines with a robust design was limited. Therefore, we decided to include uncontrolled before and after studies as well, while taking into account the weaknesses of these designs. However, our results demonstrate that these studies tended to have more positive results than studies using more robust designs. Because of the relatively small number of uncontrolled before and after studies in our review, our conclusions regarding the effectiveness of guidelines were not much affected by it. Nevertheless, this finding highlights the need for well-designed studies measuring the effectiveness of Dutch evidence-based guidelines.

    Second, the studies included in our review were very heterogeneous, not allowing pooling of the results. On the other hand, heterogeneity of studies may increase the generalisability of findings as a wider range of different settings, study populations and behaviours are included.58 Third, to determine effectiveness of guidelines on quality of care, we counted the number of measures in each study that showed a significant result. In this assessment, we did not take into account the effect sizes of the individual measures (equal weights are given to improvements of 1 or 70%).58 As an alternative, we categorised the effectiveness of an intervention in mostly, partly and not effective, which may provide more insight than a dichotomy.

    In conclusion, there is a huge misbalance between the number of guidelines developed and the number of high-quality studies that assess their effectiveness. Despite this, our review demonstrates that Dutch evidence-based guidelines can be effective in improving the process and structure of care. Evidence on the effectiveness of guidelines on patient outcomes is less convincing. The variation in effects across recommendations suggests that it is useful to focus on recommendations when analysing barriers to guideline adherence and to design implementation strategies tailored to individual recommendations instead of to the guideline as a whole. Further research is needed to determine which factors linked to the guideline and its specific recommendations are important in predicting guideline utilisation and improved patient outcomes.

    Acknowledgments

    The authors wish to thank Rikie Deurenberg, information specialist at the Dutch Institute for Healthcare Improvement (CBO), for developing and performing the search strategy.

    Appendix A: Search strategy

    Description: A detailed description of the search strategy used in this review is available online.

    REFERENCES

    1. ↵
      1. Grimshaw JM,
      2. Thomas RE,
      3. MacLennan G,
      4. et al
      . Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8:1–72.
      OpenUrlPubMed
    2. ↵
      1. Burgers JS,
      2. Grol R,
      3. Klazinga NS,
      4. et al
      . Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care 2003;15:31–45.
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Grimshaw JM,
      2. Russell IT
      . Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342:1317–22.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Grimshaw J,
      2. Freemantle N,
      3. Wallace S,
      4. et al
      . Developing and implementing clinical practice guidelines. Quality Health Care 1995;4:55–64.
      OpenUrlFREE Full Text
    5. ↵
      1. Grimshaw J,
      2. Eccles M,
      3. Ruth T,
      4. et al
      . Toward evidence-based quality improvement: evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med 2006;21(Suppl 2):14–20.
      OpenUrl
    6. ↵
      1. Worrall G,
      2. Chaulk P,
      3. Freake D
      . The effects of clinical practice guidelines on patient outcomes in primary care: a systematic review. CMAJ 1997;156:1705–12.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Shekelle PG,
      2. Woolf SH,
      3. Eccles M,
      4. et al
      . Clinical guidelines: developing guidelines. BMJ 1999;318:593–6.
      OpenUrlFREE Full Text
    8. ↵
      1. Grol R,
      2. Dalhuijsen J,
      3. Thomas S,
      4. et al
      . Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ 1998;317:858–61.
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Burgers JS,
      2. Grol RPTM,
      3. Zaat JOM,
      4. et al
      . Characteristics of effective clinical guidelines for general practice. Br J Gen Pract 2003;53:15–19.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003;12:18–23.
      OpenUrlAbstract/FREE Full Text
    11. ↵
      1. Rogers EM
      . Diffusion of innovations. 4th edn. New York: Free Press, 1995.
    12. ↵
      1. Grilli R,
      2. Lomas J
      . Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Med Care 1994;32:202–13.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Grimshaw J,
      2. Eccles M,
      3. Tetroe J
      . Implementing clinical guidelines: current evidence and future implications. J Contin Educ Health Prof 2004;24(Suppl 1):31–7.
      OpenUrlCrossRefPubMed
    14. ↵
      1. Vaughn TE,
      2. McCoy KD,
      3. BootsMiller BJ,
      4. et al
      . Organizational predictors of adherence to ambulatory care screening guidelines. Med Care 2002;40:1172–85.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Eisinger F,
      2. Geller G,
      3. Burke W,
      4. et al
      . Cultural basis for differences between US and French clinical recommendations for women at increased risk of breast and ovarian cancer. Lancet 1999;353:919–20.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Grol R,
      2. Jones R
      . Twenty years of implementation research. Fam Pract 2000;17(Suppl 1):32–5.
      OpenUrlCrossRef
    17. ↵
      1. Field MJ,
      2. Lohr KN
      . Clinical practice guidelines: directions for a new program. Washington, DC: National Academy Press, 1990.
    18. ↵
      1. Burgers J
      . Quality of clinical practice guidelines [dissertation]. Nijmegen: UMC St. Radboud, 2002.
    19. ↵
      1. Grol R,
      2. Wensing M,
      3. Eccles M
      1. Grol R
      . Implementation of changes in practice. In: Grol R, Wensing M, Eccles M, eds. Improving patient care: the implementation of change in clinical practice. Edinburgh: Elsevier Butterworth Heinemann, 2005:6–14.
    20. ↵
      1. Donabedian A
      . Evaluating the quality of medical care. Milbank Mem Fund Q 1966;44(Suppl):166–206.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Donabedian A
      . Explorations in quality assessment and monitoring, volume 1. The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press, 1980.
    22. ↵
      1. Moher D,
      2. Cook DJ,
      3. Eastwood S,
      4. et al
      . Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Br J Surg 2000;87:1448–54.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      Effective Practice and Organization of Care Group (EPOC). The data collection checklist. http://www.epoc.uottawa.ca/checklist2002.doc (accessed Sept 2007).
    24. ↵
      1. Engers AJ,
      2. Wensing M,
      3. Van Tulder MW,
      4. et al
      . Implementation of the Dutch low back pain guideline for general practitioners: a cluster randomized controlled trial. Spine 2005;30:559–600.
      OpenUrlCrossRefPubMed
    25. ↵
      1. Frijling B,
      2. Hulscher MEJL,
      3. Van Leest LATM,
      4. et al
      . Multifaceted support to improve preventive cardiovascular care: a nationwide, controlled trial in general practice. Br J Gen Pract 2003;53:934–41.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Frijling BD,
      2. Hulscher MEJL,
      3. Drenthen AJM,
      4. et al
      . Preventie van hart- en vaatziekten: resultaten van het LHV-NHG-project ‘Preventie: maatwerk’ [Prevention of cardiovascular diseases: results from the LHV-NHG project ‘Prevention: tailor-made’] (In Dutch). Huisarts Wet 2004;47:449–53.
      OpenUrl
    27. ↵
      1. Hak E,
      2. Hermens RPMG,
      3. Hoes AW,
      4. et al
      . Effectiveness of a co-ordinated nation-wide programme to improve influenza immunisation rates in The Netherlands. Scand J Prim Health Care 2000;18:237–41.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Hermens RPMG,
      2. Hak E,
      3. Hulscher MEJL,
      4. et al
      . Improving population-based cervical cancer screening in general practice: effects of a national strategy. Int J Qual Health Care 1999;11:193–200.
      OpenUrlAbstract/FREE Full Text
    29. ↵
      1. Hermens RPMG,
      2. Hak E,
      3. Hulscher MEJL,
      4. et al
      . Adherence to guidelines on cervical cancer screening in general practice: programme elements of successful implementation. Br J Gen Pract 2001;51:897–903.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Hulscher MEJL,
      2. Van Drenth BB,
      3. Van der Wouden JC,
      4. et al
      . Changing preventive practice: a controlled trial on the effects of outreach visits to organise prevention of cardiovascular disease. Qual Health Care 1997;6:19–24.
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Jans MP,
      2. Schellevis FG,
      3. Van Hensbergen W,
      4. et al
      . Improving general practice care of patients with asthma or chronic obstructive pulmonary disease: evaluation of a quality system. Eff Clin Pract 2000;3:16–24.
      OpenUrlPubMed
    32. ↵
      1. Jans MP,
      2. Schellevis FG,
      3. Le Coq EM,
      4. et al
      . Health outcomes of asthma and COPD patients: the evaluation of a project to implement guidelines in general practice. Int J Qual Health Care 2001;13:17–25.
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Kasje WN,
      2. Denig P,
      3. Stewart RE,
      4. et al
      . An educational programme for peer review groups to improve treatment of chronic heart failure and diabetes mellitus type 2 in general practice. J Eval Clin Pract 2006;12:613–21.
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Lobo CM,
      2. Frijling BD,
      3. Hulscher MEJL,
      4. et al
      . Improving quality of organizing cardiovascular preventive care in general practice by outreach visitors: a randomized controlled trial. Prev Med 2002;35:422–9.
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Frijling BD,
      2. Lobo CM,
      3. Hulscher MEJL,
      4. et al
      . Intensive support to improve clinical decision making in cardiovascular care: a randomised controlled trial in general practice. Qual Saf Health Care 2003;12:181–7.
      OpenUrlAbstract/FREE Full Text
    36. ↵
      1. Frijling BD,
      2. Lobo CM,
      3. Hulscher MEJL,
      4. et al
      . Multifaceted support to improve clinical decision making in diabetes care: a randomized controlled trial in general practice. Diabet Med 2002;19:836–42.
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Lobo CM,
      2. Frijling BD,
      3. Hulscher MEJL,
      4. et al
      . Effect of a comprehensive intervention program targeting general practice staff on quality of life in patients at high cardiovascular risk: a randomized controlled trial. Quality Life Res 2004;13:73–80.
      OpenUrlCrossRef
    38. ↵
      1. Renders CM,
      2. Valk GD,
      3. Franse LV,
      4. et al
      . Long-term effectiveness of a quality improvement program for patients with type 2 diabetes in general practice. Diabetes Care 2001;24:1365–70.
      OpenUrlAbstract/FREE Full Text
    39. ↵
      1. Renders CM,
      2. Valk GD,
      3. Van de Poll-Franse LV,
      4. et al
      . (in Dutch) De effectiviteit van een kwaliteitsbevorderingsprogramma op de zorg voor diabetespatiënten in de eerste lijn [Effectiveness of a quality improvement program on care of diabetes patients in primary care]. Huisarts Wet 2002;45:512–7.
      OpenUrl
    40. ↵
      1. Smeele IJ,
      2. Grol RP,
      3. Van Schayck CP,
      4. et al
      . Can small group education and peer review improve care for patients with asthma/chronic obstructive pulmonary disease? Qual Saf Health Care 1999;8:92–8.
      OpenUrlAbstract
    41. ↵
      1. Van der Weijden T,
      2. Grol RPTM,
      3. Knottnerus JA
      . Feasibility of a national cholesterol guideline in daily practice: a randomized controlled trial in 20 general practices. Int J Qual Health Care 1999;11:131–7.
      OpenUrlAbstract/FREE Full Text
    42. ↵
      1. Van Essen GA,
      2. Kuyvenhoven MM,
      3. De Melker RA
      . Implementing the Dutch College of General Practitioners’ guidelines for influenza vaccination: an intervention study. Br J of Gen Pract 1997;47:25–9.
      OpenUrl
    43. ↵
      1. Wolters R,
      2. Wensing M,
      3. Klomp M,
      4. et al
      . Effects of distance learning on clinical management of LUTS in primary care: a randomised trial. Patient Educ Couns 2005;59:212–8.
      OpenUrlCrossRefPubMedWeb of Science
    44. ↵
      1. Wolters R,
      2. Grol R,
      3. Schermer T,
      4. et al
      . Improving initial management of lower urinary tract symptoms in primary care: costs and patient outcomes. Scand J Urol Nephrol 2006;40:300–6.
      OpenUrlCrossRefPubMedWeb of Science
    45. ↵
      1. Kamphuisen PW,
      2. Jacobs EMG,
      3. Mol JJ,
      4. et al
      . Resultaten van het invoeren van diagnostiek naar longembolie volgens de CBO-richtlijn in een algemeen opleidingsziekenhuis [Results from a general training hospital for the implementation of a diagnostic workup for pulmonary emobolism according to the Dutch Institute for Health Care Improvement] (In Dutch). Ned Tijdschr Geneeskd 2002;146:2083–6.
      OpenUrlPubMed
    46. ↵
      1. Schouten JA,
      2. Huischer MEJL,
      3. Trap-Liefers J,
      4. et al
      . Tailored Interventions to improve antibiotic use for lower respiratory tract infections in hospitals: a cluster-randomized, controlled trial. Clin Infect Dis 2007;44:931–41.
      OpenUrlAbstract/FREE Full Text
    47. ↵
      1. Van Kasteren MEE,
      2. Mannien J,
      3. Kullberg B-J,
      4. et al
      . Quality improvement of surgical prophylaxis in Dutch hospitals: evaluation of a multi-site intervention by time series analysis. J Antimicrob Chemother 2005;56:1094–102.
      OpenUrlAbstract/FREE Full Text
    48. ↵
      1. Bakker WM,
      2. Houweling ST,
      3. Bilo HJG,
      4. et al
      . Naar polikliniek verwezen patiënten met diabetes mellitus type 2 goed af met behandeling volgens NHG-Standaard [Diabetes mellitus type 2 patients referred to the polyclinic benefit from treatment given according the NHG-guideline] (In Dutch). Huisarts Wet 2006;49:15–19.
      OpenUrl
    49. ↵
      1. Bekkering GE,
      2. Hendriks HJM,
      3. Van Tulder MW,
      4. et al
      . Effect on the process of care of an active strategy to implement clinical guidelines on physiotherapy for low back pain: a cluster randomised controlled trial. Qual Saf Health Care 2005;14:107–12.
      OpenUrlAbstract/FREE Full Text
    50. ↵
      1. Bekkering GE,
      2. Van Tulder MW,
      3. Hendriks EJM,
      4. et al
      . Implementation of clinical guidelines on physical therapy for patients with low back pain: randomized trial comparing patient outcomes after a standard and active implementation strategy. Phys Ther 2005;85:544–55.
      OpenUrlAbstract/FREE Full Text
    51. ↵
      1. De Laat EH,
      2. Schoonhoven L,
      3. Pickkers P,
      4. et al
      . Implementation of a new policy results in a decrease of pressure ulcer frequency. Int J Qual Health Care 2006;18:107–12.
      OpenUrlAbstract/FREE Full Text
    52. ↵
      1. De Laat E,
      2. Pickkers P,
      3. Schoonhoven L,
      4. et al
      . Guideline implementation results in a decrease of pressure ulcer incidence in critically ill patients. Crit Care Med 2007;35:815–20.
      OpenUrlCrossRefPubMedWeb of Science
    53. ↵
      1. Van der Sanden WJM,
      2. Mettes DG,
      3. Plasschaert AJM,
      4. et al
      . Effectiveness of clinical practice guideline implementation on lower third molar management in improving clinical decision-making: a randomized controlled trial. Eur J Oral Sci 2005;113:349–54.
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Malin JL,
      2. Asch SM,
      3. Kerr EA,
      4. et al
      . Evaluating the quality of cancer care: development of cancer quality indicators for a global quality assessment tool. Cancer 2000;88:701–7.
      OpenUrlCrossRefPubMedWeb of Science
    55. ↵
      1. Eddy DM
      . Performance measurement: problems and solutions. Health Aff 1998;17:7–25.
      OpenUrlAbstract/FREE Full Text
    56. ↵
      AGREE Collaboration. Guideline development in Europe: an international comparison. Int J Technol Assess Health Care 2000;16:1039–49.
      OpenUrlCrossRefPubMedWeb of Science
    57. ↵
      1. Burgers JS,
      2. Boluyt N
      . De kwaliteit van CBO-richtlijnen en NHG-Standaarden beoordeeld met het AGREE-instrument [An assessment using the AGREE instrument of the quality of clinical practice guidelines produced by the Dutch Institute for Healthcare Improvement (CBO) and the Dutch College of General Practitioners (NHG) evaluation] (In Dutch). Huisarts Wet 2004;47:394–9.
      OpenUrl
    58. ↵
      1. Grimshaw J,
      2. McAuley LM,
      3. Bero LA,
      4. et al
      . Systematic reviews of the effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 2003;12:298–303.
      OpenUrlAbstract/FREE Full Text

    Footnotes