3. The implementation of Clinical Practice Guidelines: a systematic review

Authors

¹ Enrique Sanchis-Sánchez
² María Montserrat Sánchez-Lorente
³ Mª Rosario Salvador-Palmer
⁴ Rosa Mª Cibrián Ortiz-de Anda
⁵ An LD Boone
⁶ María Isabel Mármol-López

PhD. Assistant Professor. Faculty of Physiotherapy. University of Valencia. Spain
PhD. Assistant Professor. Nursing School of the University and Polytechnic Hospital La Fe, Valencia. University of Valencia. Spain
PhD. Associate Professor. Faculty of Medicine. University of Valencia. Spain
PhD. Full Professor. Faculty of Medicine. University of Valencia. Spain
MD. Preventive Medicine and Public Health specialist. University of Oviedo. Spain
PhD. Nursing School of the University and Polyt

Email: Esta dirección de correo electrónico está siendo protegida contra los robots de spam. Necesita tener JavaScript habilitado para poder verlo.

Abstract

Aim: the purpose of this review is to identify and appraise the available literature on Clinical Practice Guidelines (CPGs) implementation.

Methodological culture to introduce evidence into clinical practice is not standard. Dissemination of CPGs and passive training are the most used yet insufficient strategies.

CPGs implementation is a holistic process requiring accurate planning, enthusiasm and an implementation plan that facilitates local adaptation and evaluation. Nursing managers should participate in organizing implementation. Most barriers and facilitators are related to health professionals and to the social and organizational context.

Methods: a systematic search of online search engines MEDLINE, EMBASE, CINAHL, CUIDEN and CPGs development and clearinghouses was conducted.

Results: of the 284 records identified by the search engines, only 20 were relevant. Additionally, 9 registers on implementation were retrieved from development and clearinghouses. A dynamic and particular implementation plan, organizational involvement, a multidisciplinary implementing team and local adaptation are crucial elements for successful implementation.

Conclusions: the quality and number of studies found in MEDLINE, EMBASE and CINAHL were similar. The information from the Scottish Intercollegiate Guidelines Network (SIGN), the Guidelines International Network (GIN) and GuíaSalud stands out for its quality. The relationship between evidence, context and facilitation should be strengthened. There are no unique formulas to develop an implementation process.

Keywords: clinical practice guidelines; health, implementation; planning medical care.

Título: La implementación de Guías de Práctica Clínica: una revisión sistemática

Resumen

Objetivo: el propósito de esta revisión es identificar y evaluar la literatura disponible sobre la implementación de las Guías de Práctica Clínica (GPCs).

La cultura metodológica para introducir la evidencia en la práctica clínica no es estándar. La difusión de GPCs y la capacitación pasiva son las estrategias más utilizadas pero son insuficientes.

La implementación de GPCs es un proceso holístico que requiere una planificación precisa, entusiasmo y un plan de implementación que facilite la adaptación y evaluación local. Los gerentes de enfermería deben participar en la organización de la implementación. La mayoría de las barreras y facilitadores están relacionados con los profesionales de la salud y con el contexto social y organizativo.

Métodos: búsquedas electrónicas exhaustivas y sistemáticas de perfil genérico en bases de datos MEDLINE, EMBASE, CINAHL y CUIDEN, y en centros elaboradores y compiladores de GPCs.

Resultados: de 284 registros de la búsqueda en las bases de datos, solo 20 registros fueron considerados como relevantes. De la búsqueda en los centros elaboradores de GPCs y centros compiladores se obtuvieron nueve documentos que contenían apartados específicos sobre implementación. Los planes de implementación tienen que ser dinámicos y específicos. La participación de la organización, un equipo de implementación multidisciplinario y la adaptación local son elementos cruciales para una implementación exitosa.

Conclusiones: la calidad y el número de estudios sobre implementación en las bbdd Medline, EMBASE y CINAHL son similares. Destaca la información sobre implementación del centro elaborador Scottish Intercollegiate Guidelines Network (SIGN) y de los centros compiladores de GPC Guidelines International Network (GIN) y GuíaSalud. La relación entre la evidencia, el contexto y la facilitación debe ser fortalecida. No hay fórmulas únicas para desarrollar un proceso de implementación.

Palabras clave: guías de práctica clínica; salud; implementación; planificaciónatención sanitaria.

Introduction

The aims of this systematic review are to document the key published evidence on identification, selection and evaluation of Clinical Practice Guidelines (CPGs) implementation; to critically appraise the quality of studies on CPGs implementation; to delimit the methodology and the main characteristics of implementation plans; to describe the implementation roadmaps and to extract the principles, advantages and limitations of the different implementation strategies encountered.

Current western health care systems are submerged in a complex context with exponentially increasing availability of scientific information, great variability in clinical practice, high economic and social impact of many health problems, side effects of certain treatments and new high-tech diagnostic tests among their determining factors.

Fortunately, progress in medical research reduces the level of uncertainty in clinical practice. CPGs enhance this development by establishing standards of care supported by solid scientific evidence, potentially reducing variability and improving clinical practice (1). According to the latest definition of the Institute of Medicine (IOM) (1), CPGs are “statements that include recommendations intended to optimize patient care: that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options" (2). Therefore, they are considered key instruments to transfer knowledge to clinical practice, aiming to improve the effectiveness, efficiency, safety and to simplify daily clinical decision making. They have been developed through the systematic review of evidence and the benefit-risk assessment of the alternatives (3). The definition of IOM provides a clear distinction between the term "CPGs" and other tools to improve clinical practice, such as expert consensus, appropriate use criteria, or quality measures, among others.

Protocols and clinical pathways are other instruments to enhance healthcare (4). Protocols are usually internal documents that describe the steps to follow in case of a care problem and that have been agreed upon by the concerned team. Clinical pathways offer guidelines to make proceedings operational in case of patients presenting pathologies with a predictable clinical course, establishing the time sequence of the actions to be undertaken by all professionals involved.

The usefulness of CPGs can be seen from different perspectives. On one hand, health professionals demand practical and transparent guidelines and they are increasingly accepting CPGs (5). On the other hand, health managers express a bigger need to incorporate instruments that favour efficiency and u optimise the use of resources (6). However, some aspects play against CPGs, such as their costly development, the availability of evidence or the difficulty to integrate them into local practice.

CPGs can address a wide range of health interventions. Therefore, to make them generally applicable in clinical practice, it is necessary to guarantee their quality and their methodological development (7). In many cases, however: the effort and resources invested in developing guidelines are not reflected in their use by health professionals and patients nor in real changes to the quality of care or health outcomes (8,9).

The term "implementation" is most widely accepted to refer to the introduction of scientific evidence into clinical practice. It refers to a planned process to systematically introduce innovations or changes of proven value in specific environments at different levels (clinical service, organization or health system) of decision making (10). Implementing a CPG requires a planning process with special attention for the institutional and social context, the barriers and facilitators for change in practice, and for the assessment of intervention strategies for the implementation to be effective and efficient.

Currently, in a variety of settings such as health care, science, politics or management, working lines 7have been promoted to assess the defining factors of CPGs use, from the appropriate clinical approach to their development methods, dissemination and implementation (1,11).

Some Spanish authors (12,13) have uttered that most manuals published in the last decade do not define the elaboration method, nor are they evidence-based. Furthermore, they would lack credibility because of not stating potential conflicts of interests. In recent times this has changed, however. GuíaSalud, for instance, provides professionals with methodological documents including the "CPGs Implementation Manual". This document offers support to implementers in selecting which guide to employ, in constituting the implementation team, and in planning the whole process. Moreover, it helps to acknowledge certain aspects that need to be considered to increase the possibilities of implementing the CPGs, taking into account that implementation strategies are only effective in specific circumstances.

Methods

A series of exhaustive and systematic generic profile searches were performed in online search engines Medline (via PubMed), EMBASE (Elsevier), CINAHL (EBSCO) and CUIDEN. To cover the entire scope of each database until May 2015, no start date was entered. Likewise, specific searches of methodological manuals and other documents of interest on implementation were carried out in the electronic databases of national and international guideline development and clearinghouses, such as the UK National Institute for Health and Clinical Excellence (NICE) http://www.nice.org.uk, the Scottish Intercollegiate Guidelines Network (SIGN) http://www.sign.ac.uk, the New Zealand Guidelines Group (NZGG) http://www.health.govt.nz/about-ministry/ministry-health-websites/new-zealand-guidelines-group, the US National Guideline Clearinghouse https://www.guideline.gov/, the US Agency for Health Research and Quality (AHRQ) http://www.ahrq.gov, Australia’s National Health and Medical Research Council (NHMRC) http://www.nhmrc.gov.au/, the US Institute for Clinical Systems Improvement (ICSI) http://www.icsi.org, The Guidelines International Network (G-I-N) http://www.gin.net/library/international-guidelines-library and the CPGs library of the Spanish National Health System (GuíaSalud), http://portal.guiasalud.es

The used search strategy in MEDLINE was "Health Plan Implementation" [Majr] AND "Practice Guidelines as Topic" [Majr]. Similar but adapted searches were performed in the other search engines. To start, “Clinical Practice Guide” and “Implementation” were established as conceptual sets, maximizing the outcomes by incorporating synonyms. This resulted in a union of free text terms preferably within the fields related to the title, abstract or keywords, and where possible, using thesaurus terms specific to each of the databases: MeSH, EMTREE or CINAHL headings. The final findings of the search records resulted from the intersection.

The following inclusion criteria were handled for the study: description of implementation design methodology; barriers and facilitators; strategies and interventions; and evaluation. Publications written in languages other than English or Spanish and those not available in full text were excluded.

The results of the search were transferred to the bibliographic management software Mendeley© version 1.13.8, eliminating duplicate records. Subsequently, screening of titles, summaries and key words was performed to determine their applicability and relevance according to the predefined criteria.

A similar selection process of the specific documents obtained from the CPGs development and clearinghouses was carried out. After this first selection, the quality criteria of the Prisma statement were applied (14) (Figure 1). Furthermore, an electronic file was elaborated to record and analyse all relevant articles and documents.

Results

A total of 284 records were obtained from the online search engines. After eliminating 31 duplicate records, 253 were submitted to screening. Twenty registers remained for analysis and qualitative synthesis (Figure 1).

Fifty percent of the selected articles (n= 10) was published in the last five years and three of them more than ten years ago. Concerning publication type, forty-five percent (n= 9) are research papers, twenty percent (n= 4) are opinion or reflection articles, twenty percent (n= 4) are reviews (literature or systematic) and the remaining three articles are of another kind. The article content ranges from implementation plan design, over the process of implementation, its strategies and interventions, the local adaptation, barriers and facilitators, challenges and opportunities, the opinion of local leaders; to its evaluation (Table 1).

From the websites of the CPGs development and clearinghouses, four CPGs development manuals with particular sections on implementation (Table 2) and one specific methodology implementation guide was retrieved from GuíaSalud, namely the “Manual metodológico. Implementación de CPGs en el Sistema Nacional de Salud”. Furthermore, access to two specific implementation documents and two supporting tools was obtained (Table 3).

A comprehensive search on implementation was only carried out on the website of GuíaSalud resulting in 207 hits in total (Table 4). The earlier cited methodological manual deserves special mention as it emphasizes the importance of local adaptation. In addition, the website incorporates valuable documents on implementation experiences presented on a national study day in 2013.

Health care decisions are usually influenced by cultural, organisational, educational, interpersonal and individual factors. The implementation process involves a complex mechanism of selecting a specific CPG, constituting the implementing team and evaluating its development.

Although a great amount of literature is available on CPGs elaboration, information on specific implementation methodology is limited. Furthermore, it is generally accepted that CPGs elaboration alone does not imply their use nor implementation.

Five manuals offered a specific chapter on implementation and one contained an extensive supplement on the subject. The authors concluded that most manuals lack advice on the use of recommendations considered important by health professionals. They stipulate that an effort should be made to integrate new approaches that include the use of different sources of evidence and the participation of different experts who synthesize data on clinical efficacy, plan the implementation, are responsible for it and who will provide the resources.

Ten barriers were incorporated in the first category, including the perceived validity, relevance and practical aspects of implementation for a specific adaptation, among others. Most of these barriers can be eliminated by the intervention of CPGs developers. The second category includes sixteen implementation barriers that relate to the characteristics of CPGs users. These were classified into barriers of knowledge, attitudes and beliefs and as well as skills and abilities. To address these barriers, interventions involving training, persuasion or education are required. However, some could be dealt with by changing aspects of service delivery or by introducing reminders. The remaining fifteen barriers are mainly related to the characteristics of the organisation or setting, including established practices and decision-making processes, the organisational culture and the means of evaluation. The responsibility to address second and third category barriers lies with managers and politicians.

Moreover, the role of patients to improve clinical practice is becoming increasingly important. To date, patients' knowledge and expectations have been identified in different studies as barriers to CPGs implementation. However, this is not always the case. Informed patients can either facilitate or inhibit the CPGs implementation according to the circumstances. Nevertheless, patient characteristics are not easily identified by means of the framework previously described. The last barrier identified in the review (“Conflicting patient knowledge, expectations and preference over choices in clinical management”), has been labelled as a patient factor while it is rather patient educators that have the potential to address not only patient information needs but also the ability of clinicians to negotiate treatment decisions with their patients or the management of conflicts.

Although some of the divisions between the defined barriers seem arbitrary, the framework is not presented as a theoretical taxonomy but as a checklist to guide those interested in changing clinical practice. The authors state that there may be additional barriers that have not been identified so far by means of research or practical experience. They conclude by declaring that multilevel and multifaceted strategies are needed to address barrier elimination.

The incorporation of a CPG has to be supported by active implementation strategies in each phase of the process. Different kinds of strategies have been developed; some focussing on health professionals or patients, others considering economic, organisational and even regulatory factors.

The evaluation of CPGs implementation does not only involve the assessment of the plan, but also the appraisal of the process and results.

Discussion and conclusions

To dispose of different strategies for dissemination and implementation of guidelines is important. The technical efficiency is examined to evaluate the process and, to measure the results, the degree of planning, the assigned budget and human resources are appraised (16).

Knowledge and application of the “Action Cycle” might help to systematise the implementation process (17). This action cycle promotes the use of recommendations through an interactive roadmap, including local context adaptation, identification of implementation barriers and specific interventions to promote the use of the recommendations. In addition, it includes supervision, evaluation of results and sustainability. This procedure has been identified in various implementation manuals, both from CPGs processing centres as clearinghouses.

Regardless of the authorship, resource provision or guidelines implementation responsibility, detailed instructions for implementation are required (18). The SGIGN proposes a specific methodology concerning the implementation planning process. Furthermore, it emphasizes on the previously planned dissemination of the CPGs through the distribution of written material, the free electronic downloads, news bulletins, publications in magazines, participation in conferences and congresses etc., in all involved settings and with the available economic and human resources.

In order to promote the development and use of high-quality recommendations, different CPGs assessment tools have been developed. The most widely used is the Appraisal of Guidelines Research and Evaluation (AGREE) instrument (19), and its updated version, AGREE II. This tool assesses, among other things, the scope and purpose of CPGs, stakeholder involvement, the clarity of its presentation, the editorial independence and the applicability. Another instrument, the GuideLine Implementability Appraisal (GLIA) (20), considers the impact of guideline recommendations on individuals and organisations, and whether they provide indicators for evaluating.

On the other hand, there is a hypothesis that relates the use of a recommendation with the limited instructions in the development manuals of the recommendations. To test this hypothesis, R Gagliardi and MC Brouwers conducted a systematic review examining six manuals to mainly enumerate the domains of applicability (21).

Context analysis, meaning, the evaluation of the physical or situational environment (political, historical, cultural, etc.) plays a decisive role in the implementation process as it affects health professionals, influences clinical practice and mediates the diffusion of change. Its characteristics have promoted different proposals and models to approach the analysis, for example, the “Promoting Action on Research Implementation in Health Services” (PARIHS) model (22). It claims that an effective implementation is based on three dimensions: the nature and type of evidence, the qualities of the context in which the evidence is introduced and the way in which this process is facilitated. This proposal supports that there is a dynamic relationship between the three dimensions and that each one of them is positioned along a continuum in between the extremes. The approach maintains that implementation will be more effective if the context is receptive to change, with comprehensive cultures, strong leadership, and effective information systems to monitor it. The authors of this model have developed a checklist for each of the three dimensions, which is a reference for researchers and implementers.

The importance of the organisational context also emphasizes the significant influence of the management structure as a whole in the implementation process. Managers and field professionals establish practical implications through negotiation. In this way, the authority, the available resources and the ability to influence on implementation are dispersed through multiple levels of hierarchy and in decision-making (23).

Adherence to the recommendations of a CPG, or the lack of it, are related to factors that totally or partially hinder (or facilitate) implementation; these factors are defined as barriers or facilitators. A Scottish review (5) identified forty-one barriers for CPGs implementation and classified them into three broad categories characteristics of the guideline to be introduced, characteristics of the individuals who need to change, and characteristics of the organisation or environment in which the change is to occur.

To investigate the effectiveness and costs of developing, disseminating and implementing clinical recommendations, a systematic review was conducted in the United Kingdom in 2004 (1). The reviewers concluded that, in general, most studies examining the implementation process showed an improvement in care. However, considerable variation existed in the observed effects on the interventions and the obtained results. Multifaceted interventions that showed improvement in care usually involved outreach actions through training. The conclusions of this study pointed out the need to advance in the investigation of a coherent theoretical framework on the behaviour of health professionals, organisations and health services. Furthermore, progress should be made in research to evaluate the effectiveness of dissemination and implementation strategies in the presence of different barriers and facilitators.

Another systematic review conducted in 2015 (18) also indicates that, despite the increasing awareness of the need for implementation tools, the recommendations usually lack such resources. The authors indicate that to improve health care and associated outcomes, more research is required to determine the most needed implementation tools and the most preferred ones by CPGs users, as well as the best-suited tools for different recommendations. Furthermore, they mention that CPGs developers may lack relevant experience to develop implementation tools. They also encourage all stakeholders to form alliances to achieve the objective.

According to G. Barosi, interventions that aim to implement and disseminate CPGs in an effective way fall into different categories: dissemination of educational material (including journal articles, audio-visual material and electronic publications); support systems and reminders; training sessions; audits and evaluation feedback (16).

In a study conducted by the University of Malmö published in 2010 (24), whose objective was to determine the experiences and factors related to the implementation of CPGs in a hospital setting, the nursing management and the nurses responsible for CPGs implementation showed a positive attitude towards CPGs and described them as supportive in care provision, due to the fact that they deliver clear instructions that imply a more homogenous practice. However, they were concerned about the potential risk that the critical thinking of professionals would stagnate by creating a false sense of security in case a large number of CPGs were to be implemented. For the study, interviews were conducted revealing several factors that motivated the use of CPGs and promoted their implementation. In general, professionals considered CPGs implementation as an opportunity to keep updated and to incorporate evidence-based knowledge into daily practice. A collaborative attitude of professionals to use CPGs had a positive impact on the implementation process. Changes were achieved when the team was enthusiastic and they had a common strategy even though there was no institutional support initially. Unfortunately, there were always professionals who decided not to participate actively. Intents to minimise these groups were done by describing a range of monitoring and feedback strategies to promote and guarantee the use of CPGs by means of reminders, feedback of evaluation and continuous updating of the CPGs among other measures. Similarly, uncertain situations regarding implementation occurred when there were discrepancies in the recommendations and when the recommendations were used for patients who initially hadn’t agreed on their use. These situations imposed the need for a functional and automated documentation system that would guarantee the implementation of the recommendations. Another identified need concerned the evaluation of CPGs use in relation to the balance between prioritized needs and costs. The conducted interviews revealed that most of the development and implementation of CPGs took place within existing economic frameworks.

The implementation plan design should emphasize on adapting the guidelines recommendations to local circumstances. This requires a multifocal strategy that goes beyond simple diffusion and passive education of professionals (25).

Nursing management should participate actively in the organisation of CPGs implementation and should have knowledge of the available tools for the design and evaluation of an implementation plan. Its contribution to the local adaptation of the implementation plan is essential. Nursing managers should be able to identify barriers and facilitators related to health professionals and to the social and organisational context. Active involvement and application of multifaceted strategies are needed to encourage the adoption of guidelines.

This work has allowed to study a set of possibilities related to CPGs and their implementation. Transferring evidence into clinical practice is a complex task, but we are aware that there is a great interest in uniting efforts in this line of work. Also this research is open to a more comprehensive exploration of the CPGs clearinghouses and other specific electronic resources on CPGs implementation.

We can conclude that:

The publication and dissemination of a CPG do not guarantee its use in practice. Not only is it necessary to design a holistic, dynamic and particular implementation plan that facilitates the transfer of recommendations to clinical practice, but also a multidisciplinary implementation team needs to be established.
Clinical Practice Guidelines are meant to influence the decision-making process of health professionals, patients, politicians and other stakeholders involved in policy development. Implementation of a CPG should be a holistic process that requires careful planning, dynamism and particularity.
The quality and number of studies on implementation obtained from the MEDLINE, EMBASE and CINAHL databases are similar. On the contrary, CUIDEN has a small number of studies and of low quality. The information on implementation from the CPGs clearinghouse SIGN and the CPGs compiling centres GIN and GuíaSalud deserve special highlighting.
The relationship between evidence, the specific context and facilitation need to be strengthened further.
The impact of CPGs and their recommendations on decision-making, expert clinical judgment and patient preference remains variable.
The implementation plan design should emphasize on adapting the guidelines recommendations to local circumstances.
The identification of barriers and facilitators of CPGs implementation are related to aspects of both health professionals and the context.
The key elements to evaluate the results of the CPGs implementation are structure, process and outcomes indicators.
The active implication of healthcare administration and management is necessary.

References

[1] Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess (Rockv). 2004; 8(6).

[2] OM. Clinical practice guidelines we can trust [internet]. 2011. p. 266. [cited 8 may 2019]. Available from: http://nap.edu/catalog.php?record_id=13058%5Cnhttp://books.google.com/books?hl=en&lr=&id=ppNG6FZNuSMC&oi=fnd&pg=PR1&dq=Clinical+Practice+Guidelines+We+Can+Trust&ots=B-s_-zzhtV&sig=Xy8aQABdmPNTj4RI1U8D3XB9cJY

[3] Graham R, Mancher M, Miller Wolman M, Greenfield S, Steinberg E (eds.). Clinical Practice Guidelines We CanTrust. Washington DC: NAP. Institute Of medicine (U.S). Committee on Standars for Developing Trustworty Clinica Practice Guidelines; 2011.

[4] Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Potential benefits, limitations, and harms of clinical guidelines. Bmj. 1999; 318:527-30.

[5] Foy R, Walker A, Penney G. Barriers to clinical guidelines: the need for concerted action. Br J Clin Gov. 2001; 6(3):166-74.

[6] Bonfill X. Algunos apuntes para impulsar una atención sanitaria basada en la evidencia. Med Clin. 2002; 118(3):2-6.

[7] Grupo de trabajo sobre GPC. Madrid: Plan, I+CS N para el S del MIA de C de la S-I 2007. G de PC en el S, No 2006/0I. Elaboración de Guías de Práctica Clínica en el Sistema Nacional de Salud. Manual Metodológico. 2007.

[8] NHS. Effective Health Care: Getting evidence into practice. R Soc Med Press. 1999; 5(1):1-16.

[9] Feder G, Eccles M, Grol R, Griffi ths C GJ. Getting evidence into practice. Eff Heal Care. 1999; 5(1).

[10] Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet [internet]. 2003 Oct 11 [cited 8 may 2019]; 362(9391):1225-30. Available from: http://www.sciencedirect.com/science/article/pii/S0140673603145461

[11] Mäkelä M. TTI. Changing Professional Practice. Theory and practice of clinical guidelines implementation. Copenhagen; 1999.

[12] Briones E, Marin I VE. Cambiando la Práctica Profesional teoría y práctica de la implementación de las Guías de Práctica Clínica. (Traducción al español del informe: Thorsen T, Makela M publicado por el DSI, Changing Professional Practice-methodology for guidelines implementati). Sevilla; 2002.

[13] Navarro Puerto MA, Ruiz Romero F, Reyes Domínguez A, Gutiérrez Ibarlucea I, Hermosilla Gago T, Alonso Ortiz del Río C, et al. ¿Las guías que nos guían son fiables? Evaluación de las guías de práctica clínica españolas. Rev Clínica Española [internet]. Elsevier; 2005 Nov 1 [citado 8 may 2019]; 205(11):533-40. Disponible en: http://www.revclinesp.es/es/las-guias-que-nos-guian/articulo/13081347/

[14] Hutton B, Catalá-López F, Moher D. La extensión de la declaración PRISMA para revisiones sistemáticas que incorporan metaanálisis en red: PRISMA-NMA. Med Clin (Barc) [internet]. 2016 Sep [citado 8 mayo 2019]; 147(6):262-6. Disponible en: http://linkinghub.elsevier.com/retrieve/pii/S0025775316001512

[15] Prisma. PRISMA. Transparent reporting of systematic reviews and meta-analyses. Ottawa Hospital Research Institute and University of Oxford. 2015.

[16] Barosi G. Strategies for dissemination and implementation of guidelines. Neurol Sci [internet]. 2006 [cited 8 may 2019]; 27 Suppl 3:S231-4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16752055

[17] RNAO. Toolkit: Implementation of Best Practice Guidelines. 2nd ed. Ontario; 2015.

[18] Gagliardi AR, Brouwers MC. Do guidelines offer implementation advice to target users? A systematic review of guideline applicability. BMJ Open [internet]. 2015 [cited 8 may 2019]; 5(2):e007047-e007047. Available from: http://bmjopen.bmj.com/cgi/doi/10.1136/bmjopen-2014-007047

[19] Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: Advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol. 2010; 63(12):1308-11.

[20] Shiffman RN, Dixon J, Brandt C, Essaihi A, Hsiao A, Michel G, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak [internet]. 2005 [cited 8 may 2019]; 5(1):1-8. Available from: http://dx.doi.org/10.1186/1472-6947-5-23

[21] Gagliardi AR, Brouwers MC. Integrating guideline development and implementation: analysis of guideline development manual instructions for generating implementation advice. Implement Sci [internet]. 2012 Jan [cited 8 may 2019]; 7:67. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3457906&tool=pmcentrez&rendertype=abstract

[22] Ward MM, Baloh J, Zhu X, Stewart GL. Promoting Action on Research Implementation in Health Services framework applied to TeamSTEPPS implementation in small rural hospitals. Health Care Manage Rev [internet]. 2016 [cited 8 may 2019]; 1. Available from: http://content.wkhealth.com/linkback/openurl?sid=WKPTLP:landingpage&an=00004010-900000000-99814

[23] Spyridonidis D, Calnan M. Opening the black box: A study of the process of NICE guidelines implementation. Health Policy (New York). 2011; 102(2-3):117-25.

[24] Bahtsevani C, Willman A, Stoltz P, Östman M. Experiences of the implementation of clinical practice guidelines - interviews with nurse managers and nurses in hospital care. Scand J Caring Sci [internet]. Faculty of Health and Society, Malmö University, Malmö, Sweden. 2010 Sep [cited 8 may 2019]; 24(3):514-22. Available from: http://search.ebscohost.com/login.aspx?direct=true&db=cin20&AN=2010763247&lang=es&site=ehost-live&scope=site

[25] Livesey EA, Noon JM. Implementing guidelines: what works. Arch Dis Child Educ Pract Ed. 2007; 92(5):ep129-p134.