Improving the use of research evidence in guideline development by Pan American Health Organization

Improving the Use of Research Evidence in Guideline Development

Washington DC 2010

Published also in Spanish with the title: Mejoramiento del uso de los datos de investigación para la formulación de directrices

© 2006 Schünemann et al. Licensee BioMed Central Ltd. Research publication series This publication is a reproduction of the collection of articles originally published in the journal Health Research Policy and Systems. It was also reproduced in the section “Online Extras” of the Pan American Journal of Public Health. The collection is offered in print format to facilitate its access by those who use it as reference material but have limited access to the Internet. This is an open access collection of articles distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The full collection is available in electronic format at http://www.health-policy-systems.com. and http://journal.paho.org You may also access other research publications at www.paho.org/researchportal Keywords Guidelines, Guideline development, scientific evidence. A product of the Research Promotion and Development team, HSS, of PAHO with the permission of Biomed Central. Printing was financed with funds provided by the Agencia Española de Cooperación Internacional para el Desarrollo, AECID, to PAHO

Contents

Introduction ................................................................................... 4

1. Guidelines for guidelines .............................................................. 7 2. Priority setting ............................................................................. 13 3. Group composition and consultation process ............................. 20 4. Managing conflicts of interests ................................................... 26 5. Group processes .......................................................................... 32 6. Determining which outcomes are important ............................... 36 7. Deciding what evidence to include ............................................. 42 8. Synthesis and presentation of evidence ....................................... 49 9. Grading evidence and recommendations .................................... 59 10. Integrating values and consumer involvement ............................ 66 11. Incorporating considerations of cost effectiveness, affordability and resource implications ....................................... 74 12. Incorporating considerations of equity ....................................... 80 13. Applicability, transferability and adaptation ............................... 86 14. Reporting guidelines ................................................................... 96 15. Disseminating and implementing guidelines ............................ 102 16. Evaluation ................................................................................. 106

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: introduction Andrew D Oxman*1, Atle Fretheim1, Holger J Schünemann2 and SURE3 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway, 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 3Subcommittee on the Use of Research Evidence (SURE) of the WHO Advisory Committee on Health Research (ACHR) Email: Andrew D Oxman* - oxman@online.no; Atle Fretheim - atle.fretheim@nokc.no; Holger J Schünemann - hjs@buffalo.edu * Corresponding author

Published: 20 November 2006 Health Research Policy and Systems 2006, 4:12

Received: 30 March 2006 Accepted: 20 November 2006

doi:10.1186/1478-4505-4-12

This article is available from: http://www.health-policy-systems.com/content/4/1/12 © 2006 Oxman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract In 2005 the World Health Organisation (WHO) asked its Advisory Committee on Health Research (ACHR) for advice on ways in which WHO can improve the use of research evidence in the development of recommendations, including guidelines and policies. The ACHR established the Subcommittee on the Use of Research Evidence (SURE) to collect background documentation and consult widely among WHO staff, international experts and end users of WHO recommendations to inform its advice to WHO. We have prepared a series of reviews of methods that are used in the development of guidelines as part of this background documentation. We describe here the background and methods of these reviews, which are being published in Health Research Policy and Systems together with this introduction.

Background

on processes to ensure that WHO's recommendations are well informed by the best available research evidence. These processes range from how WHO sets priorities for the development of recommendations to how its recommendations are disseminated and implemented, including recommendations developed at WHO headquarters in Geneva, at its regional offices and in countries.

In May of 2005 the 58th World Health Assembly passed a resolution requesting the Director-General "to undertake an assessment of WHO's internal resources, expertise and activities in the area of health research, with a view to developing a position paper on WHO's role and responsibilities in the area of health research, and to report through the Executive Board to the next World Health Assembly." Related to these resolutions, WHO has asked the Advisory Committee on Health Research (ACHR) for advice on ways in which WHO can improve the use of research evidence in the development of recommendations, guidelines and policies.

WHO from its inception has focused on research, which is mandated in its constitution, and has been a leading player in the global effort to strengthen ties between research and health development. Given WHO's position as the world's leading public health agency, it is essential that the organisation, its leaders and its governing body ensure that its recommendations and actions are as well informed as possible by the

The ACHR established a subcommittee to collect background documentation and consult widely among WHO staff, international experts and end users of WHO recommendations to inform this advice. The advice will focus

Page 1 of 4 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:12

http://www.health-policy-systems.com/content/4/1/12 http://www.health-policy-systems.com/content/4/1/14

and the best available Guidelines research International evidence. Network WHO has [8]) strived for existing to do this for overreviews systematic 50 years and with relevant much methodological success. research that address these questions. The answers to the questions However, are our conclusions major new based developments on the available have occurred evidence,since conWHO was established sideration of what WHO that have and led other governments organisations around are the world doing, and to logical reconsider arguments. the methods that they use to ensure that decisions about health care and public health are this For well review informed we searched by research PubMed evidence. usingThis (clinical reflection pracand guidelines tice subsequentorchanges public in health how guidelines) recommendations and (priority about health or setting aresetting developed priorities) have and beenrelated drivenarticles by recognition for selected of gaps between references [9,10]. available We searched research theevidence Cochrane and Methodolwhat is doneRegister ogy in practice, using variations priority in or practice priorities. andWe outcomes, reviewedconthe International Conference on Priorities cerns about website of the the5th quality of health care, and rising health in Health care costs.Care [11] and references that we had in our files [12-15]. Increasingly governments, professional and consumer organisations are demanding more rigorous processes to Findings What should be usedare to establish priorities? ensurecriteria that health decisions well informed by the best The US Institute Medicine's Committee on available research of evidence. The (IOM) processes, in contrast Methods for Setting Priorities Developwith traditional approaches thatfor relyGuidelines heavily on the opinment in its study of setting priorities clinical practice ions of experts, demand systematicforand transparent guidelines 1995 argued the priority setapproachespublished to access,insynthesise andthat interpret research ting process should be openthat and evidence defensiblewith [14]the Theyother recevidence; and to integrate ommended general criteria: prevalence, burden of information, six values and judgements to formulate recomillness, cost ofThe managing the more problem, variability in pracmendations. need for rigorous processes is tice, potentialby of evidence a guideline improve health outcomes, underscored of to inconsistencies between the and potential of a guideline to reduce costs. Because[1,2], data available evidence and expert recommendations used to make these judgements often lacking, theyother suginsufficient use of the available is evidence [3,4], and gested explicit in opportunities for important unmeasured insufficiencies how guidelines and recommendations factors to be considered. Theycriticisms further suggested separate are developed [5-12]. Similar have been raised "tracks" considering for specific and callsfor have been madeissues for better use of populations, research eviconditions or concerns. They arguedand thatpolicy the rationales dence for health care management making,for as designating well [13-15].special tracks should be defined clearly; i.e. why they warrant special consideration. They suggested that updating guidelines should be considered as a WHO has the of opportunity and thealso mandate to capitalise separate account ofmember issues such as the need on these "track", advancestaking and to assist its states to do so. for the availability of newabout evidence. Thiscorrections is essentialand to ensure that decisions health are well informed by research evidence, and that these deciOortwijn 25 criteria prioritise actions health sions leadidentified to effective, efficientused andtoequitable technology assessments categorised theseby into four towards achieving WHO'sand goal: the attainment all peobroad categories: burden of disease, potential effects, ples of the highest possible level of health. potential costs, and uncertainty regarding application of the technology As part of the[15]. background documentation to inform ACHR's advice to WHO we have prepared a series of In a more recent selective review for the New Zealand reviews on the following topics: Guidelines Group, the following criteria were identified as indicating that topic is suitable • Guidelines foraguidelines [16] for guideline development [16]: • Priority setting [17] 1. The topic is clinically important affecting large numbers of people with substantial morbidity or mortality (the [18] burden of • Group composition and consultation process illness). • Managing conflicts of interest [19] 2. The topic is complex enough to initiate debate about the recommendations. • Group processes [20]

3. There is evidence of variation between actual and appropri• Determining which outcomes are important [21] ate care. • Deciding what evidence to include [22] 4. There are no existing valid guidelines available to use. • Synthesis and presentation of evidence [23] 5. There is an adequate amount of existing evidence available. • Grading evidence and recommendations [24] 6. The recommendations will be acceptable to the potential users. • Integrating values and consumer involvement [25] 7. the guideline is feasible, will not exhaus• Implementation Incorporating ofconsiderations of cost-effectiveness, tively use the and communities' and [26] barriers to clinical affordability resource resources, implications change are not so high that they cannot be overcome. • Incorporating considerations of equity [27] While burden of disease is commonly used as a criterion for priority setting, it shouldand be noted that the use • Adaptation, applicability transferability [28]of summary burden of disease measures, such as disability adjusted life guidelines years (DALYs) • Reporting [29]has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent • Disseminating and implementing guidelines [30]in such measures, and because of the technical limitations of such measures (see[31] for example references [17] and [18]). • Evaluation What processes should used to agree on priorities? We have used the termbe'guidelines' broadly to include a Batistarange and Hodge in a review conducted 10 years ago wide of recommendations that WHO makes, found only three articles pertinent priority setting for including clinical, public health andto health policy recomclinical practice guidelines They suggested thehave folmendations. Although much[10]. of the literature that we lowing framework for priority setting: reviewed has focused on clinical practice guidelines, we have tried to incorporate corresponding literature for pub1. Consult with end and users and other before lic health guidelines health policystakeholders recommendations. selecting topics. The reviews are not full systematic reviews, although we 2. Consider during the consultation. have aimed feasibility to be reasonably systematic and transparent about the methods we have used and the basis for the rec3. Document thethat process used made. to select guideline topics. ommendations we have For each review we began with a series of key questions that were vetted The IOM the suggested following procedures [14]: on the amongst authorsthe and the ACHR Subcommittee Use of Research Evidence (SURE). The first author of each • the use of Delphi-like procedures forliterature obtaining expert review conducted searches for relevant and prejudgments or topic through correspondence pared the first draft.rankings The search strategies that were used are summarised in each review. We did not always con• theexhaustive use of questions specific, conduct reviews.that Weare tried first toexplicit identifyand existing sistent withreviews standard methods for questionnaire systematic that addressed the questionsconstructhat we tion and, secondarily, if we did not find a systematic asked review, relevant methodological research. When there was experimentation with formal procedures arrive a• paucity of research, wemore have also included someto descripat group judgments tive literature or, in some cases, evidence that was not directly related to guidelines development. They also suggested there is a need to define more narrowlyreview and precisely topicssummaries for guideline development. Each includes short of what WHO and They argued that thisare would result in key morefindings efficientinorganother organisations doing, our relaization of their work, some tionship to panels each of and the questions thatresolution we asked, of a discusapparent controversies, more responsive guidelines, and sion of those findings and some suggestions for further easierthat implementation. work is needed. Our answers to the key questions that we asked are summarised in the abstract of each review.

Page 4 2 of 7 4 (page number not for citation purposes)

the 2006, use of4:12 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/12

A draft of each review was first discussed and revised by the authors. The reviews were also circulated to the ACHR SURE members and discussed by the subcommittee. After peer review the articles were revised by the authors and updated if necessary. We are grateful to the editors of Health Research Policy and Systems for agreeing to publish these papers in their journal. In addition to benefiting from their editorial support, this has enabled us to take advantage of the BioMed Central's open peer review system to help ensure the quality of our reviews and advice. We also believe that these reviews are of wide interest to other organisations and individuals that are responsible for developing guidelines or health policy.

reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper and the others in this series. The members of SURE include: Andy Haines, Ana Langer, Andy Oxman (chair), Tikki Pang (WHO), Ulysses Panisset (WHO), Judith Whitworth (chair of the ACHR).

References 1. 2.

3. 4.

In addition to this series of reviews that is being published in Health Research Policy and Systems, we have conducted reviews of what WHO is currently doing, using both document analyses and interviews and we have conducted a survey of initiatives around the world that support the use of research evidence in developing guidelines or health policy. We have referred to these reports, which are being published separately, where relevant in the reviews in this series.

5. 6. 7. 8. 9. 10.

Preliminary advice from the ACHR has already been discussed with the leadership of WHO. It has been positively received and, to some extent, is being acted upon already. Before delivering our final report and advice to WHO, we will consult with a reference panel and others within and outside of WHO. We look forward to working with WHO to help implement this advice. We hope that it will assist WHO to better serve its member states by ensuring that its recommendations are well-informed by the best available research evidence, and by enabling those responsible for making decisions to make well-informed choices.

11. 12.

13. 14.

15.

Competing interests

ADO and AF work for the Norwegian Knowledge Centre forthe Health Services, an agency funded by the Norwegian government that produces systematic reviews and health technology assessments. All three authors are contributors to the Cochrane Collaboration. ADO and HJS are members of the GRADE Working Group. HJS is documents editor and chair of the documents development and implementation committee for the American Thoracic Society and senior editor of the American College of Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines.

16. 17. 18. 19. 20. 21.

Authors' contributions

ADO prepared the first draft of this introduction. AF and HJS contributed to drafting and revising it.

22. 23.

Acknowledgements We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these

Oxman AD, Guyatt GH: The science of reviewing research. Annals of the New York Academy of Science 1993, 703:125-34. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC: A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992, 268:240-248. Silagy CA, Stead LF, Lancaster T: Use of systematic reviews in clinical practice guidelines: case study of smoking cessation. BMJ 2001, 323:833-36. Vigna-Taglianti F, Vineis P, Liberati A, Faggiano F: Quality of systematic reviews used in guidelines for oncology practice. Annals of Oncology 2006, 17:691-701. Bradbury J: Storm over WHO-ISH hypertension guidelines. Lancet 1999, 353:563. Horton R: WHO: the casualties and compromises of renewal. Lancet 2002, 359:1605-11. Laing R, Waning B, Gray A, Ford N, t Hoen E: 25 years of the WHO essential medicines lists: progress and challenges. Lancet 2003, 36:1723-9. McCarthy M: Critics slam draft WHO report on homoeopathy. Lancet 2005, 366:705. Oxman AD, Lavis JN, Fretheim A: The use of evidence in WHO recommendations. Lancet in press. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999, 281:1900-5. Grilli R, Magrini N, Penna A, Mura G, Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000, 355:103-6. Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H: Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ 1998, 317:858-61. Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform policymaking. Lancet 2004, 364:1615-21. Lavis JN, Davies H, Oxman AD, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy 2005, 10(Suppl 1):35-48. Sheldon TA: Making evidence synthesis more useful for management and policy-making. J Health Serv Res Policy 2005, 10(Suppl 1):1-5. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 1. Guidelines for guidelines. Health Res Policy Syst 2006. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 2. Priority setting. Health Res Policy Syst 2006. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 3. Group composition. Health Res Policy Syst 2006. Boyd E, Bero L: Improving the Use of Research Evidence in Guideline Development: 4. Managing conflicts of interest. Health Res Policy Syst 2006. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 5. Group processes. Health Res Policy Syst 2006. Schünemann HJ, Oxman AD, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 6. Determining which outcomes are important. Health Res Policy Syst 2006. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include. Health Res Policy Syst 2006. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 8. Synthesis and presentation of evidence. Health Res Policy Syst 2006.

Page 3 of 4 (page number not for citation purposes)

Health Research Policy and Systems the 2006, use of4:14 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/14 BioMed Central

and the Guidelines International Network [8]) for existing 3. There is evidence of variation between actual and appropriOpen Access Review systematic reviews and relevant methodological research ate care. that address these questions. The answers to the questions are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. sideration of what WHO and other organisations are 1, Atle Fretheim25.and 2 Holger J Schünemann* Oxman doing, and logical arguments. ThereAndrew is an adequateDamount of existing evidence available.

Improving the use of research evidence in guideline development: 1. Guidelines for guidelines

For this1 review we searched PubMed using (clinical prac6. The recommendations will be acceptable to the potential Address: INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 2Norwegian Knowledge Centre for tice guidelines or public health guidelines) and (priority the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway users. setting or setting priorities) and related articles for selected Email: Holger J Schünemann* - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no; Andrew D Oxman - oxman@online.no references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhaus* Corresponding author ogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical website of the 5th International Conference on Priorities change are not so high that they cannot be overcome. in Health Care [11] and references that we had in our files [12-15]. 21 November 2006 While burden of2006 disease is commonly used as a criterion Published: Received: 07 April Accepted: 21setting, November 2006 for priority it should be noted that the use of sumHealth Research Policy and Systems 2006, 4:13 doi:10.1186/1478-4505-4-13 mary burden of disease measures, such as disability Findings This article is available from: http://www.health-policy-systems.com/content/4/1/13 What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing © 2006US Schünemann BioMed Central The Instituteet al; oflicensee Medicine's (IOM)Ltd.Committee on on disease rather than resource use and interventions, This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). ting process should be open and defensible [14] They recWhat processes should be used to agree on priorities? ommended six general criteria: prevalence, burden of Abstract Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracBackground: The World Health Organization (WHO), like many other organisations around the world, has found only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, recognised the need to use more rigorous processes to ensure that health care recommendations are informed clinical practice guidelines [10]. They suggested the foland potential of a guideline to reduce costs. Because data by the best available research evidence. This is the first of a series of 16 reviews that have been prepared as lowing framework for priority setting: used to make these judgements is often lacking, they sugbackground for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. gested explicit opportunities for important unmeasured We reviewed the literature on guidelines development of guidelines. Consult with end users and other stakeholders before factors Objectives: to be considered. They further suggested separate for1.the selecting topics. "tracks"Methods: for considering issues for specific populations, We searched PubMed and three databases of methodological studies for existing systematic reviews conditions or concerns. They argued that theWe rationales and relevant methodological research. did notfor conduct systematic reviews ourselves. Our conclusions are 2. Consider feasibility during the designating special tracks should be defined clearly; based on the available evidence, consideration ofi.e. what WHO and other organisations are consultation. doing and logical why they warrant special consideration. They suggested arguments. 3. Document the process used to select guideline topics. that updating of guidelines should also be considered as a Key questions answers: We found noneed experimental research that compared different formats of separate "track", taking and account of issues such as the guidelinesand for the guidelines or studies that compared different components of guidelines for guidelines. However, The IOM suggested the following procedures [14]: for corrections availability of new evidence. there are many examples, surveys and other observational studies that compared the impact of different guideline development guideline quality. health • the use of Delphi-like procedures for obtaining expert Oortwijn identifieddocuments 25 criteriaonused to prioritise judgmentsfor or topic rankings through technology categorised done these to intodevelop four guidelines Whatassessments have otherand organizations guidelines from whichcorrespondence WHO can broad learn?: categories: burden of disease, potential effects, • Establish a credible, independent committee that evaluates existing methods for developing guidelines • the use of questions that are specific, explicit and conpotential costs, and uncertainty regarding application of or that updates existing ones. sistent with standard methods for questionnaire constructhe technology [15]. • Obtain feedback and approval from various stakeholders during the development process of guidelines for tion guidelines. In a more recent selective review for the New Zealand • experimentation withasmore formal procedures to arrive Guidelines Group, the following were identified • Develop a detailed sourcecriteria document (manual) thatasguideline developers can use reference material. at group judgments indicating that a topic is suitable for guideline developWhat should be the key components of WHO guidelines for guidelines?: • Guidelines for guidelines ment [16]: should include information and instructions about the following components: 1) Priority setting; 2) Group They also suggested there is a need to define more narcomposition and consultations; 3) Declaration and avoidance of conflicts of interest; 4) Group processes; 5) rowly and precisely topics for guideline development. 1. The topic is clinically important affecting large numbers of Identification of important outcomes; 6) Explicit definition of the questions and eligibility criteria ; 7) Type of study They argued that this would result in more efficient organpeople with substantial morbidity or mortality (the burden of designs for different questions; 8) Identification of evidence; 9) Synthesis and presentation of evidence; 10) ization of panels and their work, resolution of some illness). Specification and integration of values; 11) Making judgments about desirable and undesirable effects; 12) Taking apparent controversies, more responsive guidelines, and account of equity; 13) Grading evidence and recommendations; 14) Taking account of costs; 15) Adaptation, easier implementation. 2. The topic is complex enough to initiate debate about the recapplicability, transferability of guidelines; 16) Structure of reports; 17) Methods of peer review; 18) Planned ommendations. methods of dissemination & implementation; 19) Evaluation of the guidelines.

Page 1 4 of 6 7 (page number not for citation purposes)

the 2006, use of4:13 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/13

What have other organizations done to implement guidelines for guidelines from which WHO can learn?: • Obtain buy-in from regions and country level representatives for guidelines for guidelines before dissemination of a revised version. • Disseminate the guidelines for guidelines widely and make them available (e.g. on the Internet). • Develop examples of guidelines that guideline developers can use as models when applying the guidelines for guidelines. • Ensure training sessions for those responsible for developing guidelines. • Continue to monitor the methodological literature on guideline development.

Background

What WHO is doing now An inter-cluster initiative (Guideline Development Group) led by the Evidence and Information for Policy (EIP) cluster produced the "Guidelines for WHO Guidelines" (GWG) as the recommended approach to development of WHO guidelines [3]. The process for developing the WHO document included drafting of the GWG by one group member before revision and approval by the committee. Following approval by the group, this document was reviewed and approved during a cabinet meeting before distribution as a technical cluster note to all WHO members.

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the first of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. The term guideline can be defined as "a rule or principle that provides guidance to appropriate behaviour" [1]. The Institutes of Medicine define clinical practice guidelines as "systematically developed statements to assist practitioner and patient decision about appropriate health care for specific clinical circumstances". The term "guidelines" in this document should be seen in the broad sense referring to any guideline or recommendation related to healthcare that is relevant to the mission of the WHO, including public health and health policy recommendations. A plethora of guidelines for clinical practice guidelines exist from various organizations, including national and governmental agencies and medical specialty societies. There are fewer guidelines for developing public health and health policy recommendations. We will use the term "guidelines for guidelines" as "guidelines for the development of guidelines and recommendations".

The GWG (version March 10, 2003) included the following general proposals for process (see GWG "WHO documents that guide the development, dissemination and implementation of recommendations by WHO" section 5): "b) choice of [guideline] topics; c) synthesis of the evidence; d) formulation of recommendations; e) dissemination of guidelines". The GWG makes special reference to the National Health and Medical Research Council (NHMRC) of Australia guidelines for guidelines. To accomplish proper guideline development the GWG recommends partnerships within and outside WHO according to a defined set of rules. Specific functions and composition of guideline groups are also described. The GWG also includes advice for the operationalisation of the process (section 6 GWG): a) Selection of partners; b) Organization of guideline groups; c) Process of developing guidelines; d) Guiding values. The committee also produced a self assessment checklist to ensure a consistent level of quality in the guidelines.

Guidelines for guidelines are important because of reports indicating that the lack of standardized guideline development leads to widely varying recommendations [2]. In this paper we addressed the following questions: • What have other organizations done to develop guidelines for guidelines from which WHO can learn?

Although comprehensive in the coverage of topics, due to brevity most sections of the GWG could not provide the same level of detailed instructions for guideline groups to follow that other organizations provide. Moreover, it is not entirely clear to what extent WHO guideline developers adhere to the GWG, but it appears that few departments have used the GWG [4]. In part this may be a result

• What should be the key components of WHO guidelines for guidelines? • What have other organizations done to implement guidelines for guidelines from which WHO can learn?

Page 2 of 6 (page number not for citation purposes)

Health HealthResearch ResearchPolicy Policyand andSystems Systems2006, 2006,4:14 4:13

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/13

and of athe lack Guidelines of a moreInternational detailed handbook Networkthat [8])WHO for existing guidesystematic lines development reviews and committees relevant methodological could follow, although research that there address are a number these questions. of other The possible answers explanations. to the questions are our conclusions based on the available evidence, conWhat otheroforganisations doing? sideration what WHOare and other organisations are The use quality of guidelines for guidelines varies doing, andand logical arguments. across organizations that develop guidelines. There is not an accepted international guideline developFor this review we searchedstandard PubMedfor using (clinical pracment. However, there are several specific and and(priority detailed tice guidelines or public health guidelines) examples of methods adopted by other organizations. setting or setting priorities) and related articles for selected Some of these areWe exemplary they give detailed references [9,10]. searched because the Cochrane Methodolguidance and resulted fromora thorough For examogy Register using priority priorities. process. We reviewed the Conference on Priorities ple, theof National Institute for Health and Clinical Excelwebsite the 5th International in Health Care [11] that we had inGuideline our files lence (NICE) and and the references Scottish Intercollegiate [12-15]. Network (SIGN), both large government agencies that develop guidelines, have produced comprehensive handbooks that provide guidance for its guideline developers Findings What should used to establish priorities? [5,6].criteria A number of be professional organizations have also The US Institute Medicine's (IOM) that Committee on developed detailedofguidance documents advise their Methods Setting Priorities for Guidelines guidelinefor developers about methods including Developthe develment in its of setting opment of study templates [7-10].priorities for clinical practice guidelines published in 1995 argued that the priority setting be open defensible They recWe process describeshould the single stepsand of what other[14] organizations ommended six to general criteria: prevalence,and burden of do in regards guideline development on what illness, cost of do managing thearticles problem, variability in pracgrounds they it in other in this series (see [11] tice, of a guideline togo improve healthpriorities outcomes, for apotential list of articles). The steps from setting for and potential of ato guideline to reduceof costs. Because data guideline topics implementation the guidelines. In used to make these judgements is oftenfrom lacking, they sugaddition, literature has emerged independent gested foronimportant groups,explicit such asopportunities the Conference Guidelineunmeasured Standardizafactors to be considered. They further suggested separate tion [12], that address the critical appraisal of guidelines "tracks" for considering for for specific and suggest the need forissues guidance each populations, of these steps conditions ordevelopment concerns. They argued that the rationales for of guideline [13]. For example, one tool (the designating special tracks should be defined clearly; i.e. AGREE instrument) has demonstrated its sensitivity to why they warrant suggested differentiate higherspecial qualityconsideration. guidelines thatThey followed techthat updating of guidelines be considered as a nical documentation from should those ofalso lower quality [14]. separate "track", taking account of issues such as the need for corrections and the availability of new evidence. Methods The methods used to prepare this review are described in the introduction to 25 thiscriteria series [11]. the keyhealth quesOortwijn identified used Briefly, to prioritise tions addressed in this and paper were vetted amongst the technology assessments categorised these into four authorscategories: and the burden ACHR Subcommittee on the effects, Use of broad of disease, potential Research costs, Evidence As a result of prior work in the potential and(SURE). uncertainty regarding application of area of guideline the technology [15].methodology we had knowledge of existing guidelines for guidelines by organisations such as NICE, SIGN, the US Preventive Services Task Force (USPIn a more recent selective review for the New Zealand STF), the New Zealand Guideline Group andidentified the AustralGuidelines Group, the following criteria were as ian NHRMC as the indicating thatasa well topicasisprofessional suitable for societies guidelinesuch developAmerican ment [16]: College of Chest Physicians (ACCP) and the American Thoracic Society (ATS). We attempted to search were unable to devise a search 1.PubMed, The topicbut is clinically important affecting largestrategy numbersthat of was both and reasonably specific. (the burden of people withsensitive substantial morbidity or mortality illness). Given time constraints we avoided duplication with work others focused on sources thatdebate had systematically 2.ofThe topicand is complex enough to initiate about the reccompiled relevant literature. We searched databases mainommendations.

the Agency for Healthcare andappropriQuality 3.tained Thereby is evidence of variation betweenResearch actual and (AHRQ ate care. [15]), the Guidelines International Network (GIN [16]), information obtained from prominent organizaown files. AHRQavailable databaseto(guideline 4.tions Thereand are our no existing validThe guidelines use. development methodology and guideline structure) is a database that of included references to 5.comprehensive There is an adequate amount existing 1205 evidence available. both journal but also non-journal sources. The GIN data104 references. there to was we 6.base Theincluded recommendations will beWhile acceptable theoverlap potential reviewed these citations in detail and evaluated each of users. these references for relevance. The answers to the quesare our conclusions based on the available 7.tions Implementation of the guideline is feasible, will notevidence, exhausconsideration of what WHO and other organisations are tively use the communities' resources, and barriers to clinical doing are andnot logical arguments. change so high that they cannot be overcome. Findings While burden of disease is commonly used as a criterion Wepriority found no experimental for setting, it should research be notedthat thatcompared the use of differsument formats for guidelines that mary burdenofofguidelines disease measures, suchorasstudies disability compared components ofcriticised guidelines guideadjusted lifedifferent years (DALYs) has been forfor focusing lines. However, many use examples, surveys and on disease ratherthere than are resource and interventions, other observational studiesabout of the impact of guideline because of the assumptions values inherent in such development on technical guidelinelimitations quality. of such measures, and documents because of the measures (see for example references [17] and [18]). What have other organizations done to develop guidelines for guidelines WHO from? What processesthat should be can usedlearn to agree on priorities? Batista Many large and Hodge organizations in a review that claim conducted to develop 10 years evidence ago found based only guidelines three articles have produced pertinent accessible, to priority transparent setting for clinical and detailed practice guidelines guidelines for[10]. guidelines. They suggested To make the a guidefollowing line for framework guidelinefor credible priorityand setting: acceptable, individuals with expertise in methodology, process and implementa1.tion Consult of guidelines with end were users involved and other in developing stakeholders a guideline before selecting for guideline topics. document. For example, NICE involved not only various internal groups (the national collaborating 2.centres Consider thatfeasibility develop during guidelines, the consultation. NICE patient involvement units, etc.), but also external advisors, including 3.individual Document the process to select guideline topics. academics andused governmental institutions (e.g. SIGN) [5]. SIGN and the RAND corporation published a The detailed IOM description suggested the offollowing the processes procedures involved [14]: in producing guidelines for guidelines involving various stakeholdâ&#x20AC;˘ers the[6,17]. use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence Other guideline developers carefully select the panels that â&#x20AC;˘produce the use guidelines of questions are specific, explicit conforthat guidelines ensuring thatand methodsistent ologists with and standard clinicians methods as well foras questionnaire representatives construcof the tion organization are involved. Most specialty societies have included experts and authorities in the relevant fields. â&#x20AC;˘While experimentation with formal procedures arrive this bears the riskmore of involving individualsto with less atmethodological group judgmentsand, therefore, relevant training for guideline development, it ensures that individuals who They are knowledgeable also suggested there aboutisthe a need relevant to define clinical more aspects, narrowly and ongoing preciselyresearch, topics for guideline development. including are represented and may supThey this would result in more organportsargued buy-inthat by users. Aspects focusing on efficient group processes ization of panelsincluding and their patient work, resolution of some and selection representation are apparent describedcontroversies, in other articles moreinresponsive this seriesguidelines, [18,19]. Most and easier implementation. organizations obtain approval of the final document by a board or other governing body. While no experimental

Page Page43ofof76 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:13 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/13

research indicates that providing a source document (e.g. a handbook) for guideline developers improves the quality of guidelines, observational studies suggest that organizations publishing their guidelines for guidelines in the form of reference material produce more methodologically sound guidelines [20].

12) Taking account of equity ([31], not addressed in GWG) 13) Grading evidence and recommendations ([30] and GWG Annex B) 14) Taking account of costs ([32] and GWG 5d "Making recommendations")

What should be the key components of WHO guidelines for guidelines? We have identified 19 components that are already or should be included in the GWG and that should be described in detail in a handbook or manual for WHO guideline developers. Other reviews in this series will describe these components in greater detail. We list in parenthesis the review that describes the component in more detail and the section of the GWG that has mentioned the component. The sections in the GWG cited below often consist of a single sentence. The components are:

15) Applicability, transferability and adaptation of guidelines ([33] and GWG 5a "A 3 stage process") 16) Structure of reports ([34] and GWG 6C) 17) Methods of peer review ([20] and [32]not addressed in GWG) 18) Planned methods of dissemination & implementation ([35] and GWG 5e "Dissemination of guidelines")

1) Priority setting ([21] and GWG 5b "Choice of topics for development of WHO guidelines")

19) Evaluation of the impact of the guideline ([36] and GWG 6C6)

2) Group composition (and consultations) ([22] and GWG 6A "Organization of guideline groups")

What have other organizations done to implement guidelines for guidelines from which WHO can learn? Other prominent guideline developers, such as NICE and SIGN, have ensured that those stakeholders who will become involved in guideline development also take part in the development of the guidelines for guideline. Similarly, obtaining buy-in from regions and country level representatives for GWG before agreeing on and disseminating a revised version is likely to be important. Once WHO reaches agreement on a revised version of the GWG, it should be widely disseminated and made easily available (e.g. on the Internet).

3) Declaration and avoidance of conflicts of interest ([23] and GWG 6B Note 1, annex A) 4) Group processes ([24] and GWG 6C3 "Process of developing guidelines") 5) Identification of important outcomes including cost ([25], not addressed in GWG) 6) Explicit definition of the question and eligibility criteria ([26,27], not addressed in GWG)

Examples and worksheets should be provided to facilitate implementation of the GWG. In addition, WHO should ensure training sessions for those responsible for developing guidelines. In their survey of 18 prominent international clinical guideline developers, Burgers and colleagues found that almost all guideline programs offer (in some organizations mandatory) training sessions to guideline developers [37]. SIGN, for example, offers a specific software program to guideline panel members and helps them with identifying specific learning needs [6]. SIGN also electronically records the amount of training of individuals who contribute to the guidelines.

7) Type of study designs for different types of questions ([27], not addressed in GWG) 8) Identification of evidence ([27], GWG 6C2 "Undertake a systematic review") 9) Synthesis and presentation of evidence ([28] and GWG 5C "Synthesizing the evidence") 10) Specification and integration of values ([29] and GWG 6D "Guiding values")

The GWG should not be a static document. NICE, for example, has outlined the process for updating its guidelines for guidelines. This specifies that the formal process for updating its manual will begin three years after publication of the original manual. Interim updates may be completed to accommodate small changes outside of the

11) Making judgments about desirable and undesirable effects ([29] and [30] and GWG 5d "Making recommendations")

Page 4 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:13

http://www.health-policy-systems.com/content/4/1/13 http://www.health-policy-systems.com/content/4/1/14

and the renewal regular Guidelines process. International NICE specifies Networkfour [8])criteria for existing that must all bereviews systematic fulfilledand to qualify relevantfor methodological a minor update: research a fundamental that address stage these inquestions. the guideline The answers for guideline to the process questions is neither are our added conclusions nor removed, based ona the fundamental available evidence, methods techconnique or step sideration of iswhat neither WHO added andnor other removed, organisations one or more are stakeholders doing, and logical will not arguments. obviously be disadvantaged, and the efficiency, clarity or fairness of the process or methodology this For will review be improved. we searched To develop PubMed revisions using (clinical of the GWG, pracWHO tice guidelines should monitor or publicthe health methodological guidelines) and literature (priority on guideline setting or setting development priorities)and andreview related articles updatesforofselected other organizations' references [9,10]. guidelines We searched for guidelines. the Cochrane Methodology Register using priority or priorities. We reviewed the website of the 5th International Conference on Priorities Discussion The studies we [11] identified and practical suggest in Health Care and references that experience we had in our files that guidelines for guidelines facilitate the development [12-15]. of guidelines. Our review is limited in that it is not a systematic review and is based on our own judgments. HowFindings What shouldhas be used to establish priorities? ever, criteria our review identified practical advice and The US Institute Medicine's (IOM) that Committee on components that aof handbook or manual accompany Methods Setting Priorities for Guidelines Developthe GWG for should include. ment in its study of setting priorities for clinical practice guidelines published in 1995 argued that thethe priority set-a Although WHO's leadership has endorsed GWG, ting process should be open defensible [14] They recdetailed handbook does notand exist. Moreover, implemenommended six general criteria: prevalence, of tation of the GWG in WHO guideline programsburden appears to illness, of managing the reasons problem, in pracbe very cost limited [4]. Potential forvariability this shortcoming tice, potential of a guideline improve health outcomes, are discussed elsewhere [4].toKey explanations include a and of a guideline reduce costs. Because data lack potential of resources, technicalto(methodological) capacity, used to make these judgements lacking, theynonsugknowledge about the GWG, andisa often tradition of using gested explicit opportunities for important systematic, expert opinion-based approachesunmeasured [4]; as well factors considered. further separate as a lacktoofbe training and a They lack of a moresuggested detailed manual. "tracks" for considering issues for specific populations, conditions or concerns. They argued that the rationales for Other organizations have invested substantial resources designating special tracks shouldincluding be defined clearly; i.e. into guideline development, resources to why they and warrant special consideration. Theyguidelines. suggested develop implement guidelines for that updating guidelines also and be considered as a Because WHOofhas limitedshould resources because well separate taking account of issues such as the need described"track", processes used by other organizations already for corrections theon availability of new evidence. exist, WHO canand build existing high quality guidelines for guidelines. As we discuss in another article in this Oortwijn identified criteria used adapting to prioritise health series, WHO should25also consider guidelines technology assessments and categorised intoguidefour developed by other organizations, if highthese quality broad categories: disease,WHO potential effects, lines already exist burden [33]. In of addition, should conpotential costs, andcollaborations uncertainty regarding application of sider establishing with other guideline the technology [15].unnecessary duplication of efforts and developers to avoid use of resources. In a more recent selective review for the New Zealand Further work Guidelines Group, the following criteria were identified as A systematic review of guidelines guidelines unlikely indicating that a topic is suitableforfor guidelineisdevelopto yield empirical information beyond what we have ment [16]: found in this review, but could provide useful information other organisations with 1. Theabout topic iswhat clinically important affecting are largedoing numbers of respectwith to key steps in the guideline development process. people substantial morbidity or mortality (the burden of We do not consider such a review to be a priority for illness). WHO. Similar information has been obtained through surveys [20,is complex 38, 39].enough More toinformation about 2. The topic initiate debate about specific the recquestions regarding, for example, processes that are used ommendations.

forThere updating, implementing and evaluating for 3. is evidence of variation between actualguidelines and appropriguidelines, is more likely to come from further surveys ate care. than from a systematic review. 4. There are no existing valid guidelines available to use. WHO should develop a handbook or manual that provides detailed information andofexamples for its guideline 5. There is an adequate amount existing evidence available. developers. This handbook should build on existing work, will require time resources.to Nonetheless, 6. The but recommendations will and be acceptable the potential this is likely necessary to improve the quality of WHO users. guideline development. 7. Implementation of the guideline is feasible, will not exhausEffortsuse arethe needed that ensure guideline developers begin tively communities' resources, and barriers to clinical speaking "guideline" language and improve the change arethe notsame so high that they cannot be overcome. standardisation of the guideline development processes used byburden WHO.ofThis standardisation would While disease is commonly usedhelp as a facilitate criterion the priority production of guidelines that can easily beuse adapted to for setting, it should be noted that the of sumdifferent contexts, thusmeasures, reduce global spent mary burden of and disease suchresources as disability on guideline development, particularly for low midadjusted life years (DALYs) has been criticised forand focusing dle-income countries WHOuse should participate in on disease rather than[33]. resource and interventions, international aimed about at improving guidelines for because of theefforts assumptions values inherent in such guidelines,and andbecause should of aim take a leading role in measures, thetotechnical limitations ofthese such efforts in the measures (seefuture. for example references [17] and [18]). What processes should be used to agree on priorities? Competing interests ADO and Batista andAFHodge work for in the a review Norwegian conducted Knowledge 10 years Centre ago forthe Health found only three Services, articles anpertinent agency funded to priority by the setting Norwefor gian government clinical practice guidelines that produces [10]. They systematic suggested reviews the and folhealth technology lowing framework assessments. for priority setting: All three authors are contributors to the Cochrane Collaboration. ADO and HJS areConsult 1. members with of the endGRADE users and Working other Group. stakeholders HJS isbefore documents editor selecting topics. and chair of the documents development and implementation committee for the American Thoracic 2. Consider Societyfeasibility and seniorduring editorthe of the consultation. American College of Chest Physicians' Antithrombotic and Thrombolytic Ther3. the process used to select guideline topics. apyDocument Guidelines.

The IOM suggested the following procedures [14]: Authors' contributions HJS prepared the first draft of this article. AF and ADO contributed drafting and revising for it. obtaining All authorsexpert have â&#x20AC;˘ the use of to Delphi-like procedures read and approved the final versioncorrespondence of this manuscript. judgments or topic of rankings through â&#x20AC;˘ the use of questions that are specific, explicit and conAcknowledgements We gratefully WHO and the Knowledge Centre sistent withacknowledge standard methods forNorwegian questionnaire construcforthe tion Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of â&#x20AC;˘ experimentation with more formal procedures to arrive this paper. The authors thank Mr. John Fox for administrative assistance.

at group judgments

References

They also suggested thereWord is a reference need to define more[http:// nar1. www.wordreference.com: dictionary. www.wordreference.com/definition]. rowly and precisely topics for guideline development. 2. Beck C, Cody M, Souder E, Zhang M, Small GW: Dementia diagThey argued that this would result in more efficient organnostic guidelines: methodologies, results, and implementaization panels and Soc their resolution of some tion of costs. J Am Geriatr 2000,work, 48(10):1195-1203. 3. Globalcontroversies, Programme onmore Evidence for Health Policy. Guideapparent responsive guidelines, and lines for WHO Guidelines. EIP/GPE/EQC/2003.1. Geneva , easier implementation. World Health Organization; 2003. 4.

Panisset U: A Review of WHO recommendations published in 2005. Health Res Policy Syst 2006.

Page 4 5 of 7 6 (page number not for citation purposes)

the 2006, use of4:13 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/13

5. 6.

7. 8.

10.

11. 12.

13. 14.

15. 16. 17. 18. 19. 20.

21. 22. 23. 24. 25. 26. 27. 28. 29.

National Instutites for Health and Clinical Excellence [http:/ /www.nice.org.uk/pdf/GDM_Allchapters_0305.pdf] Scottish Intercollegiate Guideline Network. SIGN 50: A guideline developers' handbook. Section 6. :(accessed on March 17, 2006) [http://www.sign.ac.uk/guidelines/fulltext/50/ section6.html]. Beyea SC: Clinical practice guidelines--an organizational effort. Aorn J 2000, 71(4):852, 855-8. Jones RH, Ritchie JL, Fleming BB, Hammermeister KE, Leape LL: 28th Bethesda Conference. Task Force 1: Clinical practice guideline development, dissemination and computerization. J Am Coll Cardiol 1997, 29(6):1133-1141. ATS: Attributes of ATS documents that guide clinical practice. Recommendations of the ATS Clinical Practice Committee. American Journal of Respiratory & Critical Care Medicine 1997, 156((6)):2015-2025. Schünemann HJ, Munger H, Brower S, O'Donnell M, Crowther M, Cook D, Guyatt G: Methodology for guideline development for the Seventh American College of Chest Physicians Conference on Antithrombotic and Thrombolytic Therapy: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004, 126:174S-178S. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: Introduction. Health Res Policy Syst 2006. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003, 139(6):493-498. Guideline development in Europe. An international comparison. Int J Technol Assess Health Care 2000, 16(4):1039-1049. Collaboration AGREE: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12(1):18-23. Agency for Healthcare Research and Quality [http:// www.guidelines.gov] Guidelines International Network [http://www.g-i-n.net] Woolf SH: Practice guidelines: a new reality in medicine. I. Recent developments. Arch Intern Med 1990, 150(9):1811-1818. Fretheim A Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 3. Group composition. Health Res Policy Syst 2006. Fretheim A Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 5. Group process. Health Res Policy Syst 2006. Burgers JS, Cluzeau FA, Hanna SE, Hunt C, Grol R: Characteristics of high-quality guidelines: evaluation of 86 clinical guidelines developed in ten European countries and Canada. Int J Technol Assess Health Care 2003, 19(1):148-157. Oxman AD SHJ Fretheim A.: Improving the Use of Research Evidence in Guideline Development: Priority setting. Health Res Policy Syst 2006. Fretheim A SHJ Oxman AD: Improving the Use of Research Evidence in Guideline Development: Group composition. Health Res Policy Syst 2006. Boyd E Bero L: Improving the Use of Research Evidence in Guideline Development: 4. Managing conflicts of interest. Health Res Policy Syst 2006. Fretheim A SHJ Oxman AD: Improving the Use of Research Evidence in Guideline Development: Group process. Health Res Policy Syst 2006. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 6. Determining which outcomes are important. Health Res Policy Syst 2006. Oxman AD Schünemann HJ, Fretheim A.: Improving the Use of Research Evidence in Guideline Development: 2. Priority setting. Health Res Policy Syst 2006. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include. Health Res Policy Syst 2006. Oxman AD Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 8. Synthesis and presentation of evidence. Health Res Policy Syst 2006. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 10. Integrat-

30. 31. 32.

33. 34. 35. 36. 37.

38. 39.

ing values and consumer involvement. Health Res Policy Syst 2006. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 9. Grading evidence and recommendations. Health Res Policy Syst 2006. Oxman AD Schünemann HJ, Fretheim A.: Improving the Use of Research Evidence in Guideline Development: 12. Incorporating considerations of equity. Health Res Policy Syst 2006. Edejer TTT: Improving the Use of Research Evidence in Guideline Development: Incorporating considerations of cost-effectiveness, affordability and resource implications. Health Res Policy Syst 2006. Schünemann HJ Oxman AD, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 13. Adaptation, applicability and transferability. Health Res Policy Syst 2006. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 14. Reporting guidelines. Health Res Policy Syst 2006. Fretheim A SHJ Oxman AD: Improving the Use of Research Evidence in Guideline Development: Disseminating and implementing guidelines. Health Res Policy Syst 2006. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 16. Evaluation. Health Res Policy Syst 2006. Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care 2003, 15(1):31-45. Oxman AD Lavis J, Fretheim A: The use of research evidence in WHO recommendations. . (submitted) Moynihan R Oxman AD, Lavis JN, Paulsen E: Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier. A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research. Oslo , Norwegian Knowledge Centre for the Health Services; 2006.

Publish with Bio Med Central and every scientist can read your work free of charge "BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours — you keep the copyright

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 6 of 6 (page number not for citation purposes)

Health Research Policy and Systems Health Research Policy and Systems 2006, 4:14

http://www.health-policy-systems.com/content/4/1/14 BioMed Central

Review and the Guidelines International Network [8]) for existing

3. There is evidence of variation between actual and appropriOpen Access systematic reviews and relevant methodological research ate care. that address these questions. The answers to the questions are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. sideration of what WHO and other organisations are 1 2 and Atle Fretheim3 Andrew D Oxman* , Holger J Schünemann doing, and logical arguments. 5. There is an adequate amount of existing evidence available.

Improving the use of research evidence in guideline development: 2. Priority setting

2INFORMA, Address: for the Health Services, pracP.O. Box 7004, St. Olavs plass, N-0130 Oslo, S.C.potential For this1Norwegian review weKnowledge searchedCentre PubMed using (clinical 6. The recommendations will Norway, be acceptable to the Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53,(priority 00144 Rome, users. Italy and 3Norwegian Knowledge Centre for the Health Services, P.O. tice guidelines or public health guidelines) and Box 7004, St. Olavs plass, N-0130 Oslo, Norway setting or setting priorities) and related articles for selected Email: Andrew D Oxman* oxman@online.no; Holger JMethodolSchünemann - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no references [9,10]. We -searched the Cochrane 7. Implementation of the guideline is feasible, will not exhaus* Corresponding authorpriority or priorities. We reviewed the ogy Register using tively use the communities' resources, and barriers to clinical website of the 5th International Conference on Priorities change are not so high that they cannot be overcome. in Health Care [11] and references that we had in our files [12-15]. While burden of disease is commonly used as a criterion Published: 29 November 2006 Received: 07 April 2006 for priority it should be noted that the use of sumAccepted: 29setting, November 2006 Health Research Policy and Systems 2006, 4:14 doi:10.1186/1478-4505-4-14 mary burden of disease measures, such as disability Findings This article is available from: What criteria should behttp://www.health-policy-systems.com/content/4/1/14 used to establish priorities? adjusted life years (DALYs) has been criticised for focusing The Institute of Medicine's (IOM) on disease rather than resource use and interventions, © 2006US Oxman et al; licensee BioMed Central Ltd. Committee on This is an Open article distributedfor under the terms of the Creative Commons Attribution (http://creativecommons.org/licenses/by/2.0), Methods forAccess Setting Priorities Guidelines Developbecause of the License assumptions about values inherent in such which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). ting process should be open and defensible [14] They recWhat processes should be used to agree on priorities? ommended six general criteria: prevalence, burden of Abstract Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracWorld to Health Organization (WHO), like many other organisations around the world, has recognised found only three articles pertinent to priority setting for tice,Background: potential of aThe guideline improve health outcomes, need toof usea more rigorous processes to Because ensure that healthclinical care recommendations are informed by the best available practice guidelines [10]. They suggested the folandthe potential guideline to reduce costs. data research evidence. This is the second a seriesthey of 16sugreviewslowing that have been prepared as background framework for priority setting: for advice from used to make these judgements is oftenoflacking, the WHO Committee on Health Research to WHO on how to achieve this. gested explicitAdvisory opportunities for important unmeasured Consult end users and other stakeholders before factors to be considered. They suggested separate Objectives: We reviewed thefurther literature on priority setting for1.health carewith guidelines, recommendations and technology selecting topics. "tracks" for considering issues for specific populations, assessments. conditions or concerns. They argued that the rationales for Methods:special We searched PubMedbeand three clearly; databasesi.e.of methodological studies forduring existing reviews and 2. Consider feasibility thesystematic consultation. designating tracks should defined relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the why they warrant special consideration. They suggested available evidence, consideration of what WHO and other organisations are doing and logical arguments. 3. Document the process used to select guideline topics. that updating of guidelines should also be considered as a separate "track", taking of issues as the need evidence to guide the choice of criteria and processes for Key questions andaccount answers: There such is little empirical The IOM the following procedures for establishing corrections priorities, and the availability of broad new evidence. but there are similarities in the criteria thatsuggested are used by various organisations and[14]: practical arguments for setting priorities explicitly rather than implicitly, • the use of Delphi-like procedures for obtaining expert Oortwijn identified 25 criteria used to prioritise health What criteria should be used to establish priorities?: • WHO has limited resources and capacity to develop judgments or topic rankings through correspondence technology assessments and categorised these into four recommendations. It should use these resources where it has the greatest chance of improving health, equity, and broad categories: burden of disease, potential effects, efficient use of healthcare resources. • the use of questions that are specific, explicit and conpotential costs, and uncertainty regarding application of We suggest[15]. the following criteria for establishing priorities sistent for developing recommendations on WHO'sconstrucaims with standard methods forbased questionnaire the •technology and strategic advantages: tion In a• Problems more recent selective for the ofNew Zealand associated withreview a high burden illness in low and middle-income countries, or new and emerging diseases. • experimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as • No existing good quality. developat group judgments indicating that arecommendations topic is suitableoffor guideline ment [16]:feasibility of developing recommendations that will improve health outcomes, reduce inequities or reduce • The They also suggested there is a need to define more narunnecessary costs if they are implemented. rowly and precisely topics for guideline development. 1. The topic is clinically important affecting large numbers of • Implementation is feasible, will not exhaustively use available resources, and barriers to change are not likely to be so They argued that this would result in more efficient organpeople with substantial morbidity or mortality (the burden of high that they cannot be overcome. ization of panels and their work, resolution of some illness). apparent controversies, more responsive guidelines, • Additional priorities for WHO include interventions that will likely require system changes and interventions whereand easierperspectives. implementation. 2. The topic is complex enough initiatebetween debate about the recthere might be a conflict in to choices individual and societal ommendations.

Page 1 4 of 7 (page number not for citation purposes)

the 2006, use of4:14 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/14

What processes should be used to agree on priorities?: • The allocation of resources to the development of recommendations should be part of the routine budgeting process rather than a separate exercise. • Criteria for establishing priorities should be applied using a systematic and transparent process. • Because data to inform judgements are often lacking, unmeasured factors should also be considered – explicitly and transparently. • The process should include consultation with potential end users and other stakeholders, including the public, using well-constructed questions, and possibly using Delphi-like procedures. • Groups that include stakeholders and people with relevant types of expertise should make decisions. Group processes should ensure full participation by all members of the group. • The process used to select topics should be documented and open to inspection. Should WHO have a centralised or decentralised process?: • Both centralised and decentralised processes should be used. Decentralised processes can be considered as separate "tracks". • Separate tracks should be used for considering issues for specific areas, populations, conditions or concerns. The rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration. • Updating of guidelines could also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.

Background

shift of resources to those areas. With regard to regional and country health issues, the team received diverging views. Some staff expressed concern about the little room for specific regional or country priorities, which would not relate directly to the global priorities. Others expressed the need for flexibility during the operational planning phase, which should contribute to the achievement of global priorities in terms of reducing a health problem or improving the health status of the population. It is important to create a monitoring and evaluation system, in which such flexibility can be taken into account.

• What criteria should be used to establish priorities?

The report offered these two recommendations regarding priority setting in general:

• What processes should be used to agree on priorities?

¾Criteria and parameters for rationalizing the setting of programme priorities should be re-examined with the view to achieving more objectivity.

• Should WHO have a centralised or decentralised process? Questions related to group processes for committees developing guidelines and recommendations and priority setting for systematic reviews are addressed in other papers in this series [1,2].

¾Exercises to shift resources to priority areas should be an integral part of the programme budgeting process, and not taken up as a separate exercise.

What WHO is doing now

The Guidelines for WHO Guidelines recommends the following [3]:

WHO does not have a centralised process specifically for setting priorities for the development of recommendations. A report of the Director-General to the Executive Board on WHO's strategic budgeting and planning process had this to say about priority setting in general (without specific reference to priorities for recommendations):

Guideline development is a process which consumes resources (see Sec VII). They could be developed on almost every health topic or intervention so it is necessary for WHO to decide which topics should be given priority. It is suggested that the following areas be given priority:

Specific global priorities were included in the procedural guidance for 2002–2003, and measures were provided to ensure a

Page 2 of 7 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:14

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing ¾interventions that will require system changes (feasibility systematic reviews and relevant methodological research concerns) as opposed to those dealing solely with provider/ that address these questions. The answers to the questions patient comparative advantage are our interactions. conclusionsWHO based has on greater the available evidence, conin dealing with governments, for interventions which require sideration of what WHO and other organisations are inputs and at different levels of the system. It has doing, and coordination logical arguments. less comparative advantage on purely provider/patient interacFor this review we searched PubMed using (clinical practions. tice guidelines or public health guidelines) and (priority setting or setting priorities) and related articles for selected ¾cost-effective interventions that address a disease burden references [9,10]. We searched the Cochrane Methodolwhich is still causing major health losses, implying under-utiliogy Register using priority or priorities. We reviewed the zation of the technology (population perspective). website of the 5th International Conference on Priorities in Health Care [11] and references that we had in our files [12-15]. ¾interventions that are of limited or questionable effectiveness but are being used widely (opportunity costs).

Once collected, all this information is translated by approprithe HEN 3. There is evidence of variation between actual and team into answerable questions. The list of questions is then ate care. presented to the Steering Committee once a year for its prioritization to policy relevance, feasibility, timelines, 4. Thereaccording are no existing valid guidelines available to use. controversy, existing evidence. 5. There is an adequate amount of existing evidence available.

What other organisations are doing

About of respondents international of 6. The 30% recommendations will to be an acceptable to the survey potential producers of clinical practice guidelines or health technolusers. ogy assessments reported using explicit methods of setting priorities, including theguideline use of explicit criteria, con7. Implementation of the is feasible, will formal not exhaussensus and burden of disease [4]. ThetoUSclinical Comtively useprocesses, the communities' resources, and barriers munityare Preventive Services Task Force, example, change not so high that they cannot be for overcome. choosesburden broad topics (e.g.,istobacco use, cancer, the While of disease commonly used asdiabetes, a criterion social environment) forshould review be on noted the basis of the the use public for priority setting, it that of health sumburdenburden of the problem; how preventable is; how relates to mary of disease measures, itsuch as itdisability other public initiatives;has andbeen the current level research adjusted lifehealth years (DALYs) criticised foroffocusing anddisease practicerather activitythan in public health, and other seton resource useclinical, and interventions, tings. Theofagenda-setting process incorporates input from because the assumptions about values inherent in intersuch ested others. measures, and because of the technical limitations of such measures (see for example references [17] and [18]). The process of selecting specific interventions for review within What should be used atocandidate agree on list priorities? those processes topics involves developing of intervenBatista and Hodge in a review conducted 10 years tions, and setting priorities using a voting procedure amongago the found only articlesThe pertinent to priority for team and thethree consultants. Task Force approvessetting or modifies clinical practice guidelines [10]. They suggested folthe resulting priorities. Priority-setting criteria that arethe adapted lowing framework forperceived priority potential setting: to reduce the burden for the reviews include of disease and injury; potential to increase healthy behaviors 1. Consult end users and other stakeholders before and reducewith unhealthy behaviors; potential to increase the selecting topics.of effective but not widely used interventions; implementation potential to phase out widely used, less effective interventions in 2. Consider the consultation. favor of morefeasibility effective orduring more cost-effective options; and current level of interest among providers and decision makers. 3. Document the process used to select guideline topics. Other priority-setting criteria may be added as relevant and appropriate. Occasionally, review teams have engaged in forThe suggested the following procedures mal IOM scoring and weighting of the criteria. One or[14]: more rounds of this process results in a prioritized list of interventions. • the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence Other respondents to the survey reported selecting topics based on consultations with their constituencies, requests •from the use of questions that aremade specific, explicitpanels and conend-users, or decisions by expert or a sistent with standard methods questionnaire construcsteering group without explicitfor methods. Seventeen of 67 tion respondents (25%) reported involving target users in the groups that set priorities. • experimentation with more formal procedures to arrive at group judgments Methods The methods used to prepare this review are described in the introduction to this [5]. Briefly, themore key quesThey also suggested thereseries is a need to define nartions addressed in this paper vetteddevelopment. amongst the rowly and precisely topics for were guideline authors andthat thethis ACHR the Use of They argued wouldSubcommittee result in more on efficient organResearchofEvidence (SURE). did not conduct aoffull sysization panels and theirWework, resolution some tematic review. We searched andguidelines, three databases apparent controversies, morePubMed responsive and of methodological studies (the Cochrane Methodology easier implementation. Register [6], the US National Guideline Clearinghouse [7],

Findings

What criteria should be used to establish priorities? ¾Interventions for diseases which have a high burden in develThe US Institute of Medicine's (IOM) Committee on oping countries, or newPriorities and emerging for which there Methods for Setting for diseases Guidelines Developare no existing guidelines. ment in its study of setting priorities for clinical practice guidelines published in 1995 argued that the priority setting process should openmight and defensible They rec¾interventions wherebethere possibly be[14] a conflict in ommended six general criteria: prevalence, burden of choices between individual and societal perspectives (political illness, cost of managing the problem, variability in pracconcerns: when countries will need WHO's normative support tice, potential of a guideline to improve health outcomes, to make recommendations based the population perspective and potential of a guideline to on reduce costs. Because data especially in the context of other influential organizations used to make these judgements is often lacking, they sugespousing guidelines adopting anfor individual perspective). gested explicit opportunities important unmeasured factors to be considered. They further suggested separate The Health Network basedpopulations, at the Euro"tracks" for Evidence considering issues (HEN), for specific pean Regional Office ofThey WHO [4], that collects concerns conditions or concerns. argued the policy rationales for and questions from several sources and through both a proactive designating special tracks should be defined clearly; i.e. and athey reactive approach: why warrant special consideration. They suggested that updating of guidelines should also be considered as a Proactive:"track", taking account of issues such as the need separate for corrections and the availability of new evidence. 1. Call for topics once a year, through a simple and userfriendly questionnaire Ministries of Members Oortwijn identified 25to criteria usedof toHealth prioritise health States, WHO technical and unitscategorised (TUs) including European technology assessments these into four observatory for Health care systems broad categories: burden of disease, potential effects, potential costs, and uncertainty regarding application of 2. Reviewing the[15]. work already done by HEN Members as well the technology as their work in progress In a more recent selective review for the New Zealand 3. Review ofGroup, minutesthe of following EU' Parliament Guidelines criteria were identified as indicating that a topic is suitable for guideline developReactive: ment [16]:

1. The The topic Health Evidenceimportant Network affecting To the HEN or 1. is clinically largee-mailbox numbers of direct requests from policymakers people with substantial morbidity or mortality (the burden of illness). 2. Specific questions or policy concerns identified by the Observatory WHO Technical Unitstoininitiate their processes of production 2. Theortopic is complex enough debate about the recof papers. ommendations.

Page Page 43 of of 77 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:14 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing systematic reviews and relevant methodological research that address these questions. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

3. There is evidence of variation between actual and appropriate care.

For this review we searched PubMed using (clinical practice guidelines or public health guidelines) and (priority setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodology Register using priority or priorities. We reviewed the website of the 5th International Conference on Priorities in Health Care [11] and references that we had in our files [12-15].

6. The recommendations will be acceptable to the potential users.

4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available.

7. Implementation of the guideline is feasible, will not exhaustively use the communities' resources, and barriers to clinical change are not so high that they cannot be overcome. While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]).

Findings

What processes should be used to agree on priorities? Batista and Hodge in a review conducted 10 years ago found only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the following framework for priority setting:

Oortwijn identified 25 criteria used to prioritise health technology assessments and categorised these into four broad categories: burden of disease, potential effects, potential costs, and uncertainty regarding application of the technology [15].

â&#x20AC;˘ the use of questions that are specific, explicit and consistent with standard methods for questionnaire construction

In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]:

â&#x20AC;˘ experimentation with more formal procedures to arrive at group judgments They also suggested there is a need to define more narrowly and precisely topics for guideline development. They argued that this would result in more efficient organization of panels and their work, resolution of some apparent controversies, more responsive guidelines, and easier implementation.

1. The topic is clinically important affecting large numbers of people with substantial morbidity or mortality (the burden of illness). 2. The topic is complex enough to initiate debate about the recommendations.

Page 4 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14

http://www.health-policy-systems.com/content/4/1/14

and the Guidelines Oortwijn identifiedInternational six steps in the Network development [8]) for of existing practical procedures systematic reviews forand setting relevant priorities methodological [15]: research that address these questions. The answers to the questions 1. Clarifying are our conclusions goals andbased responsibilities: on the available evidence, consideration of what WHO and other organisations are 2. Choosing doing, and logical a general arguments. approach, method, and criteria for prioritisation; For this review we searched PubMed using (clinical prac3. Establishing tice guidelinesadvisory or public mechanisms health guidelines) and relations andwith (priority external bodies; setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodol4. Establishing arrangements and We manage the proceogy Register using priority to orsupport priorities. reviewed the dure; website of the 5th International Conference on Priorities in Health Care [11] and references that we had in our files [12-15]. 5. Defining a time table and cycle of activity; and

level or isduring implementation run actual the risk being 3. There evidence of variation between and of appropriignored, ate care. misused, and modified in ways detrimental to patients." This is even more so for internationally developed guidelines. Priority setting at each level should 4. There are no existing valid guidelines available to use. draw on the strengths and minimize the limitations of international, and local organizations. Thus, both cen5. Therenational is an adequate amount of existing evidence available. tralised and decentralised processes that take account of these strengths and needs, 6. Thedifferent recommendations will limitations, be acceptableastowell the aspotential are necessary. users. 7. Implementation of the guideline is feasible, will not exhausDiscussion WHOuse hasthe limited resourcesresources, and limited tively communities' and technical barriers tocapacity clinical for developing essential that it change are not so recommendations. high that they cannotItbeisovercome. should set priorities for how best to use the resources and capacity it has.ofWe did not find an empirical for While burden disease is commonly used as a basis criterion deciding how best ittoshould set priorities. theofuse of for priority setting, be notedHowever, that the use sumexplicitburden criteria of anddisease systematic processes areasmore likely mary measures, such disability than implicit criteria andhas non-systematic to adjusted life years (DALYs) been criticisedprocesses for focusing ensure openrather and defensible priority Based on the on disease than resource usesetting. and interventions, experience of other organisations, logic and the aims and because of the assumptions about values inherent in such strategic advantages of of WHO we suggest that the followmeasures, and because the technical limitations of such ing criteria(see should be used references to set priorities: measures for example [17] and [18]).

6. Evaluating and developing the procedure. Findings

What criteria should be used to establish priorities? The Institute of Medicine's (IOM) ways Committee on She US further identified the following in which Methods fortoSetting forvary: Guidelines Developapproaches priorityPriorities setting can ment in its study of setting priorities for clinical practice guidelines inprocedure 1995 argued that the set• the extent published to which the is explicit and priority systematic ting process should be open and defensible [14] They recommended general criteria: prevalence, • the extent tosix which external input and advice is burden accepted of or illness, cost of managing the problem, variability in pracactively sought tice, potential of a guideline to improve health outcomes, and of a guideline Because data • thepotential relative weight given to tothereduce views costs. of decision-makers, used to make judgements is often lacking, they sugresearchers, andthese others gested explicit opportunities for important unmeasured factors to be They further suggested separate • the extent to considered. which the procedure is transparent "tracks" for considering issues for specific populations, conditions or concerns. argued the rationales for • the effort and resources They devoted to thethat procedure designating special tracks should be defined clearly; i.e. why they conclusion warrant special consideration. They suggested Her main was that explicit and transparent prithat of health guidelines should also be considered as a orityupdating setting for technology assessment is feasible, separate taking account of issues such as the need but that "track", some important methodological issues need to for correctionstoand the availability of new evidence. be addressed ensure that the procedure used is valid, reliable, consistent and useful for policy making. Oortwijn identified 25 criteria used to prioritise health technology assessments and categorised these four There is some debate, variation in practice, andinto limited broad categories: burden ofof disease, potential effects, data regarding involvement the public in priority setpotential and uncertainty regarding of ting. Therecosts, is limited evidence from a small application survey in Austhe technology tralia that the[15]. public overwhelmingly want their preferences to inform priority-setting decisions [19]. In a more recent selective review for the New Zealand Should WHO have the a centralised decentralised process? Guidelines Group, following or criteria were identified as There are two in which priority setting is currently indicating thatways a topic is suitable for guideline developdecentralised: geographically (across headquarters, ment [16]: regional offices and countries), and across technical departments. are important limited findings in large the literature 1. The topic is There clinically affecting numbers to of informwith decisions about how this best(the be burden handled. people substantial morbidity or might mortality of The IOM noted, "that it is unreasonable – indeed imposillness). sible – to expect nationally developed guidelines to cover every issue for to every kind of setting". 2. The operational topic is complex enough initiate debate about the"Yet recguidelines that leave too much to be decided at the local ommendations.

What processes shouldwith be used to agree onofpriorities? • Problems associated a high burden illness in low Batista and Hodge in a revieworconducted 10 years disago and middle-income countries, new and emerging found eases. only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the following framework for priority setting: • No existing guidelines or recommendations of good quality. 1. Consult with end users and other stakeholders before selecting topics. of developing recommendations that will • The feasibility improve health outcomes, reduce inequities or reduce 2. Consider feasibility during the consultation. unnecessary costs if they are implemented.

3. Document the process used to select topics. • Implementation is feasible, will notguideline exhaustively use available resources, and barriers to change are not likely to The following procedures [14]: be soIOM highsuggested that theythe cannot be overcome. • the use of Delphi-like procedures for obtaining expert Additional priorities for WHO include interventions judgments or topic rankings through correspondence that will likely require system changes and interventions where there might be a conflict in choices between indi• the use questions that are specific, explicit and convidual andofsocietal perspectives. sistent with standard methods for questionnaire construction The application of these criteria requires judgements. Appropriate processes are needed, in addition to explicit • experimentation with more formal procedures arrive criteria, to ensure that these judgements are madeto openly, at group that theyjudgments are taken account of in how WHO uses its resources, and that they reflect the priorities of WHO's They alsostates, suggested there is athose needofto low define narmember particularly andmore middlerowly precisely development. incomeand countries. We topics suggestfor thatguideline the following processes They argued that this would result in more efficient organbe used for these reasons: ization of panels and their work, resolution of some apparent controversies, more to responsive guidelines, • The allocation of resources the development of and receasier implementation. ommendations should be part of the routine budgeting process rather than a separate exercise.

Page 4 5 of 7 (page number not for citation purposes)

the 2006, use of4:14 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/14

Competing interests

• Criteria for establishing priorities should be applied using a systematic and transparent process.

• Because data to inform judgements are often lacking, unmeasured factors should also be considered – explicitly and transparently. • The process should include consultation with potential end users and other stakeholders, including the public, using well-constructed questions, and possibly using Delphi-like procedures. • Groups that include stakeholders and people with relevant types of expertise should make decisions. Group processes should ensure full participation by all members of the group.

Authors' contributions

ADO prepared the first draft of this review. HJS and AF contributed to drafting and revising it.

Acknowledgements

• The process used to select topics should be documented and open to inspection.

We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper and the others in this series.

Both centralised and decentralised processes should be used to take account of different strengths, limitations and needs within WHO across headquarters, regions and countries; and across different technical areas. Drawing on the suggestion of the IOM for having different tracks for considering issues for specific populations, conditions or concerns [14], we suggest:

References 1. 2.

• Both centralised and decentralised processes should be used. Decentralised processes can be considered as separate "tracks".

3. 4.

• Separate tracks should be used for considering issues for specific areas, populations, conditions or concerns. The rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration.

• Updating of guidelines could also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.

6. 7. 8.

Further work

Many organisations are now using explicit and systematic priority setting processes for practice guidelines and health technology assessments. A more comprehensive and systematic survey of this experience could inform decisions about processes WHO should use to set priorities for recommendations. Because there is uncertainty about the best ways to set priorities, the processes that are used should be evaluated. When feasible and relevant, alternative processes should be directly compared with respect to the priorities that are generated and the resources that are used.

10. 11. 12.

13.

14.

Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 5. Group processes. Health Res Policy Syst 2006. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 8. Synthesis and presentation of evidence. Health Res Policy Syst 2006. Global Programme on Evidence for Health Policy. In Guidelines for WHO Guidelines Geneva: World Health Organization, EIP/ GPE/EQC/2003.1; 2003. Moynihan R, Oxman AD, Lavis JN, Paulsen E: Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier. In A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research Oslo: Norwegian Knowledge Centre for the Health Services; 2006. Oxman AD, Fretheim A, Schünemann HJ: Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst 2006. Cochrane Methodology Register [http://www.cochrane.org/ access_data/cmr/accessDB_cmr.asp] National Guideline Clearinghouse [http://www.guidelines.gov] Guidelines International Network (GIN) [http://www.g-in.net] Helou A, Perleth M, Schwartz FW: [Determining priorities in the development of medical guidelines. 1: Criteria, procedures and actors: a methodological review of international experiences] [Article in German]. Z Arztl Fortbild Qualitatssich 2000, 94:53-60. Battista RN, Hodge MJ: Setting priorities and selecting topics for clinical practice guidelines. CMAJ 1995, 153:1233-6. 5th International Conference on Priorities in Health Care [http://www.healthpriorities.org/] Priority-Setting Group, Council on Health Care Technology, Institute of Medicine: National Priorities for the Assessment of Clinical Conditions and Medical Technologies. Report of a Pilot Study. Washington DC: National Academy Press; 1990. Committee on Priorities for Assessment and Reassessment of Health Care Technologies, Institute of Medicine: Setting Priorities for Health Technology Assessment. A Model Process. Washington DC: National Academy Press; 1992. Committee on Methods for Setting Priorities for Guidelines Development, Institute of Medicine: Setting Priorities for Clinical Practice Guidelines. Washington DC: National Academy Press; 1995.

Page 6 of 7 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:14

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/14

15. the Oortwijn WJ: First Things First. Network Priority Setting Health and Guidelines International [8]) forfor existing Technology Assessment. In (PhD thesis) Leiden: The Netherlands systematic reviews and relevant methodological research Organisation for Applied Scientific Research (TNO) Prevention and that address these questions. The answers to the questions Health; 2000. 16. our McKinlay E, McLeodbased D, Dowell T, available Howden-Chapman P: Clinical are conclusions on the evidence, conPractice Guidelines. A selective Literature Review. Presideration of the what WHO andGuidelines other organisations are pared for New Zealand Group. Wellington: Wellington Schoolarguments. of Medicine, University of Otago; 2001. doing, and logical

3. There is evidence of variation between actual and appropriate care. 4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available.

Mooney G: Discussion of the Paper: Uses of Summary Measures for Population Health by Murray, Salomon and MathFor this searched PubMed using (clinical pracers. review Presentedwe at WHO Conference on Summary Measures of Populatoin Health 1999 [http://www.who.int/whosis/smph/]. tice guidelines or public health guidelines) and (priority 18. Arnesen T, Kapiriri L: Can the value choices in DALYs influence setting or setting andPolicy related for selected global prioritypriorities) setting? Health 2004,articles 70:137-49. references [9,10]. WeG,searched the KC: Cochrane 19. Wiseman V, Mooney Berry G, Tang InvolvingMethodolthe general public in priority setting: experiences from Sci ogy Register using priority or priorities. WeAustralia. reviewedSocthe Med 2004, 58:1459. th 17.

website of the 5 International Conference on Priorities in Health Care [11] and references that we had in our files [12-15].

While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]).

Findings

• the use of questions that are specific, explicit and consistent with standard methods for questionnaire construction Publish with Bio Med Central and every scientist can read your work free of charge • experimentation with more formal procedures to arrive "BioMed Central will be the most significant development for at group judgments

In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]:

disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your and research precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organreviewed and immediately upon acceptance ization peer of panels andpublished their work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation.

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page Page 47 of of 77 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Review

Improving the use of research evidence in guideline development: 3. Group composition and consultation process

Open Access

Atle Fretheim*1, Holger J Schünemann2 and Andrew D Oxman1 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway and 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy Email: Atle Fretheim* - atle.fretheim@nokc.no; Holger J Schünemann - hjs@buffalo.edu; Andrew D Oxman - oxman@online.no * Corresponding author

Published: 29 November 2006 Health Research Policy and Systems 2006, 4:15

Received: 07 April 2006 Accepted: 29 November 2006

doi:10.1186/1478-4505-4-15

This article is available from: http://www.health-policy-systems.com/content/4/1/15 © 2006 Fretheim et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract Background: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the third of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. Objective: In this review we address the composition of guideline development groups and consultation processes during guideline development. Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key questions and answers: What should be the composition of a WHO-panel that is set up to develop recommendations? The existing empirical evidence suggests that panel composition has an impact on the content of the recommendations that are made. There is limited research evidence to guide the exact composition of a panel. Based on logical arguments and the experience of other organisations we recommend the following: • Groups that develop guidelines or recommendations should be broadly composed and include important stakeholders such as consumers, health professionals that work within the relevant area, and managers or policy makers. • Groups should include or have access to individuals with the necessary technical skills, including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing. • Groups should include or have access to content experts. • To work well a group needs an effective leader, capable of guiding the group in terms of the task and process, and capable of facilitating collaboration and balanced contribution from all of the group members. • Because many group members will not be familiar with the methods and processes that are used in developing recommendations, groups should be offered training and support to help ensure understanding and facilitate active participation. What groups should be consulted when a panel is being set up? We did not identify methodological research that addressed this question, but based on logical arguments and the experience of other organisations we recommend that as many relevant stakeholder groups as practical should be consulted to identify suitable candidates with an appropriate mix of perspectives, technical skills and expertise, as well as to obtain a balanced representation with respect to regions and gender. What methods should WHO use to ensure appropriate consultations? We did not find any references that addressed issues related to this question. Based on logical arguments and the experience of other organisations we believe that consultations may be desirable at several stages in the process of developing guidelines or recommendations, including: • Identifying and setting priorities for guidelines and recommendations • Commenting on the scope of the guidelines or recommendations • Commenting on the evidence that is used to inform guidelines or recommendations • Commenting on drafts of the guidelines or recommendations • Commenting on plans for disseminating and supporting the adaptation and implementation of the guidelines or recommendations. • Key stakeholder organisations should be contacted directly whenever possible. • Consultation processes should be transparent and should encourage feedback from interested parties.

Page 1 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:15

http://www.health-policy-systems.com/content/4/1/15 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing Background The World reviews Health Organization like many other systematic and relevant (WHO), methodological research organisations around the world, has recognised the need that address these questions. The answers to the questions to use rigorousbased processes toavailable ensure that health concare are ourmore conclusions on the evidence, recommendations informed by the best available sideration of whatare WHO and other organisations are researchand evidence. is the third of a series of 16 reviews doing, logical This arguments. that have been prepared as background for advice from the this WHO Advisory Committee on using Health(clinical Research to For review we searched PubMed pracWHO on how to this. guidelines) and (priority tice guidelines or achieve public health setting or setting priorities) and related articles for selected Health care[9,10]. recommendations systematically and references We searchedthat the are Cochrane Methodoltransparently developed well informed by the best ogy Register using priorityand or priorities. We reviewed the Conference on Priorities availableofevidence require several types of evidence and website the 5th International in Health Care [11] and references that we had in our files judgements. Judgements must be made about: [12-15]. • The expected effects of the options that are being considered, Findings What criteria should be used to establish priorities? The US Institute Medicine's (IOM)effects Committee on • Factors that mightofmodify the expected in specific Methods settings, for Setting Priorities for Guidelines Development in its study of setting priorities for clinical practice guidelines published in 1995inargued the priority set• Needs, risks and resources specificthat settings, ting process should be open and defensible [14] They recommended sixand general criteria: prevalence, • Ethical, legal political constraints, and burden of illness, cost of managing the problem, variability in practice, a guideline to improve healthharms outcomes, • Thepotential balance of between the expected benefits and and of a guidelineisto reduce costs. Because data costspotential if a recommendation implemented. used to make these judgements is often lacking, they suggested opportunities for important unmeasured A groupexplicit developing recommendations must be capable of factors to the be considered. They suggested separate assessing evidence that is further available to inform these "tracks" for and considering specific populations, judgements to makeissues all offor these different types of conditions concerns. They rationales for judgements.orIn this paper weargued addressthat thethe following quesdesignating special tracks should be defined clearly; i.e. tions: why they warrant special consideration. They suggested that updating guidelines should also considered that as a • What shouldofbe the composition of abe WHO-panel separate taking account of issues such as the need is set up "track", to develop recommendations? for corrections and the availability of new evidence. • What groups should be consulted when a panel is being Oortwijn set up? identified 25 criteria used to prioritise health technology assessments and categorised these into four broad burden of disease, potential effects, • Whatcategories: methods should WHO use to ensure appropriate potential costs, and uncertainty regarding application of consultations? the technology [15]. Questions related to group processes or integrating values In more recent selective review for the New Zealand anda consumer involvement are addressed in two other Guidelines Group, following criteria were identified as papers in this seriesthe [1,2]. indicating that a topic is suitable for guideline development [16]: What WHO is doing now Expert committees are sometimes used by WHO to provide The Director General selects 1. Theguidance. topic is clinically important affecting large committee numbers of members WHO'smorbidity expert advisory panels. mempeople withfrom substantial or mortality (theThe burden of bers of these panels are primarily included based on "their illness). technical ability and experience". When an expert committee is assembled shouldtohave: "equitable geographi2. The topic is complexitenough initiate debate about the reccal representation, gender balance, a balance of experts ommendations.

from developed and countries, 3. There is evidence of developing variation between actualrepresentation and appropriof different trends of thought, approached and practical ate care. experience in various parts of the world, and an appropriateThere interdisciplinary balance" [3]. 4. are no existing valid guidelines available to use. Establishing expert amount committee is a formal process with 5. There is anan adequate of existing evidence available. regulations established by the World Health Assembly. Many recommendations are not to developed by 6. The WHO recommendations will be acceptable the potential expert committees. Less formal procedures that are not users. subject to the Regulations for Expert Advisory Panels and Committees are frequently usedistofeasible, convene that 7. Implementation of the guideline willgroups not exhausdevelop or recommendations. "Consultations" tively useguidelines the communities' resources, and barriers to clinical or "proceedings" alsothat frequently provide the basis for recchange are not so high they cannot be overcome. ommendations. Consumers or representatives of the general public areofrarely included in groups develop While burden disease is commonly used that as a criterion recommendations. for priority setting, it should be noted that the use of summary burden of disease measures, such as disability The Guidelines for(DALYs) WHO Guidelines state thatfor the "Techadjusted life years has been criticised focusing nicaldisease Guidelines should be multion ratherDevelopment than resourcegroup" use and interventions, disciplinary with around 8–12 representing because of the assumptions aboutindividuals values inherent in such stakeholders experts, primary measures, and(professionals, because of thedisease technical limitations of care/ such public health, users, and patients)[17] as well methodmeasures (see end for example references andas [18]). ologists [4]. However, up to now WHO has published few What processes should used to agree priorities? recommendations that be have adhered to on these guidelines. Batista and Hodge in a review conducted 10 years ago found consultations only three articles pertinent to be priority settingused for Broad do not appear to commonly clinical practice guidelines [10].ofThey suggested the folto identify potential members expert committees or lowinggroups framework priority setting: or recommendaother that for develop guidelines tions. Consultations at other stages during the process of 1. Consult with end users andare other developing recommendations alsostakeholders uncommon,before apart selecting from peertopics. review of draft reports. Occasionally, draft recommendations may be circulated more widely. 2. Consider feasibility during the consultation.

What other organisations are doing

In Document a recent international 152 units that topics. support 3. the process survey used toofselect guideline the use of research in developing guidelines and health policy, most respondents reported that their[14]: guidelines The IOM suggested the following procedures development panels consisted of several stakeholders or expert groups – often including end-users and consumers • the use of Delphi-like procedures for obtaining expert [5]. Another of guidelines oncorrespondence hypertension and judgments or review topic rankings through hyperlipidaemia, found that stakeholder involvement was much lower guidelines sponsored by specialty societies • the use of in questions that are specific, explicit and conthan inwith guidelines sponsored groups [6]. A third sistent standard methodsby forother questionnaire construcreview of 18 prominent guideline organisations in Austion tralia, Canada, Europe, New Zealand and the U.S. found that guideline development groups consist of 10 • experimentation with more formaltypically procedures to arrive to group 20 members and the number of disciplines per group is at judgments often three to five [7]. Most of these programs invite methodological experts (epidemiologists, scientists and They also suggested there is a need library to define more narothers)and and precisely patient representatives. Most have permanent rowly topics for guideline development. staff providing editorial support. They argued that this would result in more efficient organization of panels and their work, resolution of some Many agencies have issued for developing apparent controversies, more guidelines responsive guidelines, and guidelines [8-11] (see also our review of Guidelines for easier implementation. Guidelines [12]), and all such documents that we are

Page 4 2 of 7 6 (page number not for citation purposes)

the use of research evidence in guideline development Health Research PolicyImproving and Systems 2006, 4:15 http://www.health-policy-systems.com/content/4/1/15

aware of recommend convening multi-disciplinary groups. This includes consumers, professionals working in the field, and individuals with the necessary methodological skills (e.g. epidemiologists, economists). Procedures for recruitment of panel members are usually prescribed or suggested, with emphasis on extensive consultation with relevant stakeholder-groups.

• When the draft scope of the guideline has been prepared

In contrast to approaches that rely heavily on clinical experts or research experts, as exemplified by many specialty societies [6] and the WHO, the National Institute for Health and Clinical Excellence (NICE) in the UK does not necessarily include experts in their guideline development groups (GDGs): "Experts attending a GDG are present because of their knowledge in a particular area. Therefore, it is important that they sit within the group and enter fully into any discussion. However, they are not full members of the group; they do not have voting rights and should not be involved in the final wording of recommendations." The role of the professional members in the GDG is to "represent the perspective(s) of the health care workers involved" [10]. In an external evaluation carried out by WHO, it was concluded that NICE is "internationally a leading agency", and that the organisation "has developed a well-deserved reputation for innovation and methodological development" [13].

The methods used to prepare this review are described in the introduction to this series [15]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We searched PubMed and three databases of methodological literature (the Cochrane Methodology Register [16], the US National Guideline Clearinghouse [17], and the Guidelines International Network [18]) for existing systematic reviews and relevant methodological research that address these questions. We did not conduct systematic reviews ourselves. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

• During the selection of panel members • When the full draft version of the guidelines is completed

Methods

In our literature search on panel composition we used the terms "group composition" or "panel composition" or "'consumer involvement' and guidelines". We also checked the reference lists of key papers and contacted key researchers in the field. In our search on consultation processes, we used the term "guidelines and consultation and process".

Most guidelines for guidelines highlight the importance of having an effective leader. This person has a key role in facilitating "the interpersonal aspects of the group processes" and ensuring "that the group works in a spirit of collaboration, with a balanced contribution from all members" [10].

Findings

What should be the composition of a WHO-panel that is set up to develop recommendations? We identified relatively few articles on group composition. A key paper was a comprehensive report by Murphy and colleagues from 1998, who reviewed the research literature on group composition and clinical guideline development [19]. This systematic review identified several studies that compared recommendations by groups with different compositions, and several comparisons of judgements made by homogenous subgroups of mixed groups [19]. The authors found that "these studies, although few in number, show that differences in group composition may lead to different judgements. More specifically, members of a specialty are more likely to advocate techniques that involve their specialty." Their conclusion was that "these studies confirm that the composition of groups is important in determining the decision reached."

Some agencies arrange training for members of GDGs – particularly, but not solely, aimed at facilitating the active participation of consumer representatives [14]. Another suggested approach to ensure consumer involvement is to establish separate focus groups for this purpose [10]. Wide consultation in the course of developing recommendations may be done in various ways, for instance by hosting open meetings to discuss guideline drafts [8], or by posting guideline drafts on the web [10]. Peer-review is also commonly used. In the survey of 152 units that support the use of research in development of guidelines and health policy, most reported involvement of target-users in the selection of topics, e.g. in priority-setting groups, through surveys or by reviewing draft lists of priority topics [5]. Most respondents also reported having consumers involved at various stages of the development process, often by review of draft guidelines or reports [5]. In its manual for guideline developers, NICE specifies several stages during guideline development where consultations with stakeholders should take place:

Knowing that groups with different compositions produce different recommendations does not necessarily tell us what group composition will provide the most appro-

Page 3 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:15

http://www.health-policy-systems.com/content/4/1/15 http://www.health-policy-systems.com/content/4/1/14

and the priate recommendations. Guidelines International Arguments Network for using [8]) for multidisciexisting plinary groups systematic reviews are and largely relevant basedmethodological on logic. For example, research "Individuals' that address these biases questions. may be better The answers balanced to in themultidisciquestions plinary are our groups, conclusions and such basedbalance on the may available produce evidence, more valid conguidelines"of sideration [20]. what A report WHOfrom andthe other U.S. organisations Institute of Mediare cine (IOM) doing, and logical put forward arguments. three arguments for multidisciplinary groups: 1) Multidisciplinary participation increases the probability For this review we thatsearched all relevant PubMed scientific using evidence (clinicalwill pracbe located tice guidelines and critically or publicevaluated; health guidelines) 2) Such and participation (priority increases setting or setting the likelihood priorities)that and related practical articles problems for selected with using guidelines references [9,10].will Webe searched identified theand Cochrane addressed; Methodol3) Participation ogy Register helps using build priority a sense or priorities. of involvement We reviewed or "ownerthe on Priorities ship" among website of thedifferent 5th International audiencesConference for the guidelines [21]. in Health Care [11] and references that we had in our files [12-15]. against having narrowly focused expert groups Arguments are based in part on research that have compared expert recommendations to systematic reviews and which have Findings What criteriathe should be used tobetween establishexpertise priorities? investigated relationship and sysThe US reviews. InstituteInofone Medicine's (IOM) Committee on tematic study comparisons were made Methods for Setting Priorities for Guidelines Developbetween recommendations of clinical experts in textbooks ment in its medical study ofjournals setting priorities forofclinical practice and major and results meta-analyses guidelines published in 1995 argued that the for priority setof randomized controlled trials of treatments myocarting should open and defensible recdial process infarction [22].be The investigators found[14] thatThey clinical ommended general criteria: prevalence, burden of experts oftensix made recommendations that were not conillness, managing the problem, variability pracsistent cost withofavailable research findings. Anotherin study tice, potential of a guideline to improve health outcomes, found strong correlations between the expertise of authors and potential a guideline to reduce costs.inBecause data of reviews andofthe methods that were used the reviews used make these oftenstronger lacking, opinions they sug[23]. to Expertise wasjudgements associated iswith gested opportunities for time important unmeasureda prior toexplicit conducting a review, less spent conducting factors be the considered. Theysystematic further suggested separate review, to and use of less and transparent "tracks" methods.for considering issues for specific populations, conditions or concerns. They argued that the rationales for designating special tracks shouldand be colleagues defined clearly; i.e. The systematic review by Murphy included why they special suggested studies of warrant the effects thatconsideration. heterogeneity They has on group that updating of guidelines alsoweight be considered as a judgement, and concluded should that "The of evidence separate "track", taking account of issues such asgroup the need suggest that heterogeneity in a decision-making can for the availability of new evidence. leadcorrections to a betterand performance than homogeneity. There is, however, also some evidence that heterogeneity may have Oortwijn 25 criteria used may to prioritise health an adverseidentified effect because conflict arise between technology assessments diverse participants" [19].and categorised these into four broad categories: burden of disease, potential effects, potential costs, uncertainty In addition, the and review found fewregarding studies ofapplication the extent of to the technology [15]. individuals that participate in a which the particular group affect the groups' decisions. The authors concluded In more recent of selective review the but New thata "the selection individuals hasfor some, notZealand a great Guidelines Group, the as deal, of influence on following outcome",criteria basedwere on identified this limited indicating research. that a topic is suitable for guideline development [16]: Finally, the review included research related to the opti1. The topic is clinically affecting large numbers of mal size of groups. The important authors remark that "having more people substantial mortality (the of burden of group with members will morbidity increase or the reliability group illness). judgement", while "large groups may cause coordination problems". They base their conclusion mainly on research 2. The topic is complex enough to initiate debate about recwithin social and organisational psychology: "It isthe likely ommendations. that below about six participants, reliability will decline

3. There is evidence of variation between and appropriquite rapidly, while above about 12,actual improvements in ate care. will be subject to diminishing returns." reliability 4. no existing valid guidelines to use.in the WeThere haveare identified one recent study available not included review by Murphy et al. where recommendations made by 5. Thereof is an adequate amount of existing evidence available. groups different composition were compared [24], as well as four studies comparing judgements made by dif6. The subgroups recommendations willgroups be acceptable the findings, potential ferent in mixed [25-28].toThe users. which are consistent with the conclusions of the review, indicate that clinical experts have a lower threshold for 7. Implementation of thethey guideline is feasible, willappropriate. not exhausrating the procedures perform as being tively use study, the communities' resources, and to clinical Another in which clinicians werebarriers surveyed about change are not so highof that they cannot be overcome. the appropriateness coronary angiography for various indications, found a similar relationship [29]. While burden of disease is commonly used as a criterion for should on be noted that the use of comsumWe priority did not setting, find anyit studies the impact of group mary burden of disease such recommendaas disability position on public health ormeasures, health systems adjusted life yearsthere (DALYs) has evidence been criticised for focusing tions. However, is some that suggests that on disease rather thanbetween resourceexpertise use andand interventions, the same relationships recommenbecause thehave assumptions about inherent in such dations of that been found forvalues clinical recommendameasures, andfound because the technical limitations of such tions are also forofpublic health recommendations; measures (see for example references and [18]). i.e. that expert recommendations are [17] frequently not consistent with the available research evidence [30]. MoreoWhatthose processes shouldpublic be usedhealth to agree on priorities? ver, making and health policy Batista and Hodgemay in afrequently review conducted 10 years conago recommendations not systematically foundpotential only three articles pertinent to priority setting for sider adverse effects of public health and health clinicalinterventions. practice guidelines [10]. They suggested the folpolicy lowing framework for priority setting: We found several systematic reviews of consumer involve1. Consult withThere end are users and other before ment [31-35]. a number of stakeholders relevant arguments selecting topics. for including consumers in groups that develop recommendations and descriptions of practical experience. 2. Consider feasibility during the consultation. However, a review of comparative studies of interventions to promote consumer involvement [31], found a "lack of 3. Document process usedstudies to select topics. evidence fromthe comparative to guideline inform decisions about desirable and adverse effects of consumer involveThe IOM suggesteddecisions the following ment in collective aboutprocedures health care[14]: or how to achieve effective consumer involvement". The authors of â&#x20AC;˘ the usesystematic of Delphi-like procedures for obtaining expert another review, which included non-comparajudgments through correspondence tive studiesoroftopic howrankings to involve consumers in setting the research and development agenda for health systems [32], â&#x20AC;˘ the use of collaborating questions thatwith are "well-networked specific, explicit and conrecommend consumsistent standard methods for questionnaire construcers andwith providing them with information, resources and tion support to empower them in key roles for consulting with their peers." They also recommended "consultations â&#x20AC;˘ experimentation with more formal procedures to arrive should engage consumer groups directly and repeatedly in at group judgments facilitated debate." See also our review on how to integrate values and involve consumers in guideline developThey also ment [2]. suggested there is a need to define more narrowly and precisely topics for guideline development. Whatargued groupsthat should consulted a panel is being set They thisbe would resultwhen in more efficient organup? ization of panels and their work, resolution of some We did not identify papers addressing thisguidelines, question. and apparent controversies, more responsive easier implementation.

Page 4 of 7 6 (page number not for citation purposes)

the use of 4:15 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/15

What methods should WHO use to ensure appropriate consultations? We did not identify any research findings that could inform the answer to this question

The process should also include wide consultation that encourages feedback at subsequent steps in the process, which may include: • Identifying and setting priorities for guidelines and recommendations

Discussion

Based on the findings from the reports we have identified, there is sufficient evidence to conclude that how a panel is composed can have an important impact on conclusions drawn by a group when making recommendations for health care. In particular, clinical experts are more likely to recommend procedures linked to their own specialty than others. Furthermore, experts in a field frequently do not employ systematic methods when reviewing evidence and developing recommendations. These findings support the current recommendation in the Guidelines for WHO Guidelines: Panels should be multidisciplinary, including a broad representation of stakeholders, as well as methodologists [4].

• Commenting on the scope of the guidelines or recommendations • Commenting on the evidence that is used to inform guidelines or recommendations • Commenting on drafts of the guidelines or recommendations • Commenting on plans for disseminating and supporting the adaptation and implementation of the guidelines or recommendations.

The research evidence to guide panel composition and consultation processes is limited. However logical arguments and the experience of other organisations suggest that

• Commenting on what research should be conducted based on the guidelines

Further work

We have not conducted an exhaustive systematic review, but have based much of this paper on a systematic review from 1998. We have not found subsequent studies that provide conflicting evidence. There is, however, limited research for the questions addressed in this report. We do not believe that a more exhaustive review would yield a great deal of additional evidence at this time. However, it would be valuable for WHO or others to undertake and keep up-to-date systematic methodology reviews that address specific issues of group composition, including the selection of a group leader, methods for effective consultations, and methods for effective consumer involvement.

• Groups that develop guidelines or recommendations should be broadly composed and include important stakeholders such as consumers, health professionals that work within the relevant area, and managers or policy makers. • Special attention should be paid to the selection of a group leader who has a crucial role in ensuring a positive group process and that all voices within the group can be heard. • Wide consultations should be done when selecting members of a group to develop WHO recommendations, for example by direct contact with stakeholder groups.

Competing interests

AF and ADO work for the Norwegian Knowledge Centre forthe Health Services, an agency funded by the Norwegian government that produces systematic reviews and health technology assessments. All three authors are contributors to the Cochrane Collaboration. ADO and HJS are members of the GRADE Working Group. HJS is documents editor and chair of the documents development and implementation committee for the American Thoracic Society and senior editor of the American College of Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines.

• Groups should include or have access to individuals with the necessary technical skills, including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing. • Groups should include or have access to content experts. • Many group members will not be familiar with the methods and processes that are used in developing recommendations, and should be offered training and support to help ensure understanding and facilitate active participation.

Authors' contributions

AF prepared the first draft of this review. HJS and ADO contributed to drafting and revising it.

The process of developing recommendations, including the selection of group members, should be transparent.

Page 5 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:15

http://www.health-policy-systems.com/content/4/1/15 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing Acknowledgements We gratefully acknowledge WHO and themethodological Norwegian Knowledge Centre systematic reviews and relevant research forthe Health Services financial support, the otherto authors of these that address these for questions. The answers the questions reviews their contributions, other of the ACHR SURE, are ourforconclusions basedtheon themembers available evidence, conand all of those who have provided us with feedback on earlier versions of sideration of what WHO and other organisations are this paper and the others in this series. doing, and logical arguments.

23. There Oxman Guyattof GH: The science of reviewing research. Ann 3. is AD, evidence variation between actual and appropriN Y Acad Sci 1993, 703:125-133. ate care. 24. Raine R, Sanderson C, Hutchings A, Carter S, Larkin K, Black N: An experimental study of determinants of group judgments in

clinical development. Lancetavailable 2004, 364:429-437. 4. There areguideline no existing valid guidelines to use.

Herrin J, Etchason JA, Kahan JP, Brook RH, Ballard DJ: Effect of panel composition on physician ratings of appropriateness of abdominal aortic aneurysm elucidating 5. There is an adequate amount surgery: of existing evidencedifferences available. between multispecialty panel results and specialty society recommendations. Health Policy 1997, 42:67-81. 6. recommendations will be acceptable the Burnand potential 26. The Vader JP, Porchet F, Larequi-Lauber T, DuboistoRW, B: users.Appropriateness of surgery for sciatica: reliability of guidelines from expert panels. Spine 2000, 25:1831-1836. 27. Bernstein SJ, Lazaro P, Fitch K, Aguilar MD, Kahan JP: Effect of spe7. Implementation of theon guideline is feasible, not exhauscialty and nationality panel judgments of will the appropriaterevascularization: a pilot study.to Med Care tivelyness use ofthecoronary communities' resources, and barriers clinical 2001, 39:513-520. change not so P,high thatMD, they cannot be overcome. 28. Fitchare K, Lazaro Aguilar Martin Y, Bernstein SJ: Physician recommendations for coronary revascularization. Variations by clinical speciality. Eur J Public Health 1999, 9:181-187. While burden of disease is commonly used as a criterion 29. Ayanian JZ, Landrum MB, Normand SL, Guadagnoli E, McNeil BJ: Ratfor priority setting, it should noted angiography--do that the use of suming the appropriateness of be coronary pracphysicians agree with an expert such panel as and disability with each maryticing burden of disease measures, other? N Engl J Med 1998, 338:1896-1904. adjusted life I:years (DALYs) hasGood been than criticised focusing 30. Chalmers Trying to do more Harmfor in Policy and Practice:rather The Role Rigorous, use Transparent, Up-to-Date on disease thanof resource and interventions, Evaluations. The Annals of theabout American Academy of Political in andsuch Social because of the assumptions values inherent Science 2003, 589:22-40. measures, and because of the technical of such 31. Nilsen ES, Myrhaug HT, Johansen M, Oliver S,limitations Oxman AD: Interventions for consumer involvement developing measures (seepromoting for example references [17] andin[18]). healthcare policy and research, clinical practice guidelines and patient information material. The Cochrane Database of SysWhat processes be used to agree on priorities? tematic Reviews,should Issue 3 2006. 32. Oliver S, Clarke-Jones R, Milne R, Buchanan P, Batista and Hodge inL, aRees review conducted 10Gabbay yearsJ, Gyte ago G, Oakley A, Stein K: Involving consumers in research and found only three articles pertinent to priority setting for development agenda setting for the NHS: developing an evidence-based Health Technol Assess 2004, 8:1-IV. clinical practiceapproach. guidelines [10]. They suggested the fol33. Boote J, Telford R, for Cooper C: Consumer lowing framework priority setting: involvement in health research: a review and research agenda. Health Policy 2002, 61:213-236. 34. Consult Crawfordwith MJ, Rutter Manley C, Weaver T, Bhui K, Fulop N, Tyrer 1. end D, users and other stakeholders before P: Systematic review of involving patients in the planning and selecting topics. development of health care. BMJ 2002, 325:1263. 35. Coney S: Effective Consumer Voice and Participation for New Zealand. feasibility New Zealandduring Guidelines 2004. 2. Consider theGroup; consultation. 25.

References

For review we searched PubMed (clinical prac1. this Fretheim A, Schünemann H, Oxman AD: using Improving the use of research evidence in guideline Group Proctice guidelines or public healthdevelopment: guidelines) 5. and (priority esses. Health Res Policy Syst 2006. setting or setting priorities) and related articles for selected 2. Schünemann H, Fretheim A, Oxman AD: Improving the use of references [9,10]. We searched the Cochrane research evidence in guideline development: 10.MethodolIntegrating and Consumer Involvement. Health Resreviewed Policy Syst 2006. ogy Values Register using priority or priorities. We the 3. Regulations for Advisory Panels and Committees. In th Expert International Conference on Priorities website of the 5 Resolution WHA 35 10 WHO; 1982. in Carefor [11] andguidelines. referencesGeneva, that we hadProgramme in our files 4. Health Guidelines WHO Global on Evidence for Health Policy, WHO; 2003. [12-15].

Moynihan R, Oxman AD, Lavis J, Paulsen E: A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for Findings the WHO Advisory Committee on Health Research. Oslo, What criteria should be used to establish priorities? Norwegian Knowledge Centre for the Health Services; 2006. The Fretheim US Institute of JW Medicine's Committee on 6. A, Williams Jr., Oxman (IOM) AD, Herrin J: The relation between recommendations in clinical practice Methods formethods Settingand Priorities for Guidelines Developguidelines for hypertension and hyperlipidemia. J Fam Pract ment2002, in its study of setting priorities for clinical practice 51:963-968. 7. Burgers published JS, Grol R, Klazinga NS,argued Makela M, Zaat J: Towards eviguidelines in 1995 that the priority setdence-based clinical practice: an international survey of 18 ting process should be open and defensible [14] They recclinical guideline programs. Int J Qual Health Care 2003, 15:31-45. ommended six general criteria: prevalence, burden of 8. SIGN Guideline Development Handbook. Edinburgh, Scottish Intercollegiate Guidelines Network.; 2004. variability in pracillness, cost of managing the problem, 9. Handbook for the Preparation of Explicit Evidence Based tice, Clinical potential of a guideline to Wellington, improve health outcomes, Practice Guidelines. New Zealand Guideand lines potential a guideline to reduce costs. Because data Group.;of 2001. 10. National Institute Health and Clinical Excellence: The they guidelines used to make theseforjudgements is often lacking, sugmanual. London, National Institute for Health and Clinical Excelgested explicit opportunities for important unmeasured lence; 2006. 11. National and Medical Research Council: A guide to the factors to beHealth considered. They further suggested separate development, implementation and evaluation of clinical "tracks" for considering issues for specific populations, practice guidelines. Canberra, National Health and Medical conditions concerns. Researchor Council; 1998. They argued that the rationales for 12. Schünemann: H, Fretheim Oxman AD: Improving the usei.e. of designating special tracksA,should be defined clearly; research evidence in guideline development: 1. Guidelines why for they warrant Health special consideration. They suggested Guidelines. Res Policy Syst 2006. 13. Hill S, Garattini van Loenhout J, O'Brien BJ, be de Joncheere K: Techthat updating of S, guidelines should also considered as a nology Appraisal Programme National Institute for separate "track", taking account of of the issues such as the need Clinical Excellence. Copenhagen, WHO Regional Office for for corrections Europe; 2003. and the availability of new evidence. 14. Kelsom M: The NICE Patient Involvement Unit. Evidence-Based Healthcare & Public Health 2005, 9:304-307. Oortwijn used prioritise 15. Oxman identified AD, Fretheim25A, criteria Schünemann H: to Improving thehealth use of technology and categorised these into four research assessments evidence in guideline development: introduction. Health Res Policy Systburden 2006. broad categories: of disease, potential effects, 16. Cochrane Methodology Register [http://www.cochrane.org/ potential costs, and uncertainty access_data/cmr/accessDB_cmr.asp] regarding application of 17. technology National Guideline the [15]. Clearinghouse Agency for Healthcare Research and Quality 2006 [http://www.guidelines.gov]. 18. Guidelines International Network (GIN) Guidelines International In aNetwork more 2006 recent selective review for the New Zealand [http://www.g-i-n.net]. 19. Murphy MK, Blackthe NA,following Lamping DL, McKeewere CM, identified Sanderson CF, Guidelines Group, criteria as Askham J, Marteau T: Consensus development methods, and indicating that a topic is suitable for guideline developtheir use in clinical guideline development. Health Technol mentAssess [16]: 1998, 2:i-88. 20. Shekelle PG, Woolf SH, Eccles M, Grimshaw J: Clinical guidelines: developing guidelines. BMJ 1999, 318:593-596. 1. topic Practice is clinically important affecting numbers of 21. The Clinical Guidelines. Directions forlarge a New Program. Edited Field MJ and Lohr KN. Washington DC, National Academy people withby:substantial morbidity or mortality (the burden of Press; 1990. illness). 22. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC: A comparison of results of meta-analyses of randomized control trials and clinicaldebate experts. Treatments 2. The topic is recommendations complex enough toofinitiate about the recfor myocardial infarction. JAMA 1992, 268:240-248. 5.

3. Document the process used to select guideline topics. The IOM suggested the following procedures [14]: • the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence • the use of questions that are specific, explicit and consistent with standard methods for questionnaire construction Publish with Bio Med Central and every scientist can read your work free of charge • experimentation with more formal procedures to arrive "BioMed Central will be the most significant development for at group judgments disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation.

ommendations.

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 4 6 of 7 6 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 4. Managing conflicts of interests Elizabeth A Boyd*1 and Lisa A Bero2 Address: 1Department of Clinical Pharmacy, University of California, San Francisco, 3333 California Street, Suite 420, San Francisco, CA 941430613, USA and 2Department of Clinical Pharmacy and the Institute for Health Policy Studies, University of California, San Francisco, Suite 420, Box 0613, 3333 California Street, San Francisco, CA 94118, USA Email: Elizabeth A Boyd* - boyde@pharmacy.ucsf.edu; Lisa A Bero - berol@pharmacy.ucsf.edu * Corresponding author

Published: 01 December 2006 Health Research Policy and Systems 2006, 4:16

doi:10.1186/1478-4505-4-16

Received: 07 April 2006 Accepted: 01 December 2006

This article is available from: http://www.health-policy-systems.com/content/4/1/16 © 2006 Boyd and Bero; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Page 1 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:16

http://www.health-policy-systems.com/content/4/1/16 http://www.health-policy-systems.com/content/4/1/14

andHow the Guidelines could conflict International of interest Network policies [8]) befor enforced? existing 3. There is evidence of variation between actual and approprisystematic reviews and relevant methodological research ate care. • Although there are no empirical studies of the enforcement of conflict if interest policies, descriptive studies of other that address these questions. The answers to the questions organizations and institutions suggest that WHO convene a standing committee to review all financial disclosure are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. statements prior to the commencement of committee meetings/hearings and to make management recommendations sideration of what WHO and other organisations are when necessary. A standard policy requiring all financial ties to be made public (i.e., recorded into the meeting minutes) doing, and logical arguments. 5. There is an adequate amount of existing evidence available. should reduce the number of problematic cases. In instances where the conflicts seem intractable, a recommendation of recusal may be necessary to protect the greater interests of WHO and its constituents. For this review we searched PubMed using (clinical prac6. The recommendations will be acceptable to the potential tice guidelines or public health guidelines) and (priority users. setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhausIn thisuse paper we address the following Background ogy Register using priority or priorities. We reviewed the tively the communities' resources, andquestions: barriers to clinical The World Health Organization (WHO), like many other website of the 5th International Conference on Priorities change are not so high that they cannot be overcome. organisations the world, has recognised • What is the best way to obtain complete and accurate in Health Carearound [11] and references that we had inthe ourneed files to use more rigorous processes to ensure that health care disclosures onof financial tiescommonly and otherused competing inter[12-15]. While burden disease is as a criterion recommendations are informed by the best available ests? for priority setting, it should be noted that the use of sumresearch evidence. This is the fourth of a series of 16 mary burden of disease measures, such as disability Findings reviews that have been as background for advice • How to whenhasa been disclosed financial tie or What criteria should beprepared used to establish priorities? adjusted lifedetermine years (DALYs) criticised for focusing from the WHO Advisory Committee on Health Research other competing interest constitutes a conflict of interest? The US Institute of Medicine's (IOM) Committee on on disease rather than resource use and interventions, to WHO on to achieve this.for Guidelines DevelopMethods forhow Setting Priorities because of the assumptions about values inherent in such • When a conflict of interest identified, how should the ment in its study of setting priorities for clinical practice measures, and because of theistechnical limitations of such A conflict of interest exists when an individual's secondary conflict be managed? guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). interests (e.g.should personal financial) interfere with or influting process be open and defensible [14] They recence judgments the individual's primary inter•What Howprocesses could conflict interest be priorities? enforced? shouldofbe used topolicies agree on ommended six regarding general criteria: prevalence, burden of ests (e.g. patient welfare, education, research integrity). Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracThere is evidence the association of finanRelated questions about group composition, consultation found only three articles pertinent to priority setting for tice, potential of ademonstrating guideline to improve health outcomes, cial ties with a breakdown in research integrity. Recent and group processes are addressed in another paperthe in this clinical practice guidelines [10]. They suggested foland potential of a guideline to reduce costs. Because data studies and reviews have found that industry funding for series [17,18]. lowing framework for priority setting: used to make these judgements is often lacking, they sugresearch is associated with favourable outcomes for the gested explicit opportunities for important unmeasured sponsor [1-5]. Financial ties of investigators with their What WHO doing 1. Consult withisend usersnow and other stakeholders before factors to be considered. They further suggested separate Expert Advisory Panel members are currently required to sponsors (stock ownership, consulting income, etc.) are selecting topics. "tracks" for considering issues for specific populations, disclose "all circumstances that could give rise to a potenalso associated with favourable research for the conditions or concerns. They argued that outcomes the rationales for tial conflict of interest during as a result their membership on sponsor [5]. This scholarly evidence has been accentuated 2. Consider feasibility the of consultation. designating special tracks should be defined clearly; i.e. an expert committee." [19] by lay media stories documenting how financial conflicts why they warrant special consideration. They suggested of interest have to biasedshould and even research 3. Document the process used to select guideline topics. that updating ofled guidelines alsodangerous be considered as a According to the WHO Declaration of Interests for WHO (e.g., [6,7]). Biased research may be intentional or separate "track", taking account of issues such as theuninneed Experts, conflict ofthe interest occurs when "the expert or tentional [8] and may result from damaged objectivity at The IOMasuggested following procedures [14]: for corrections and the availability of new evidence. his/her partner (a spouse or other person with whom s/he multiple stages in the research process, including concephas a similar close personal relationship), or the administualization of the question, or prioritise conduct of the • the use of Delphi-like procedures for obtaining expert Oortwijn identified 25 criteriadesign used to health trative unit with which the expert has an employment research, interpretation of the results, and publication (or judgments or topic rankings through correspondence technology assessments and categorised these into four relationships, has a financial or other interest that could not) the research [9,10].ofRegardless of its source, the broadofcategories: burden disease, potential effects, unduly influence the expert's respect the bias associated with financial and other conflicts of inter• the use of questions that areposition specific,with explicit andtoconpotential costs, and uncertainty regarding application of subject matter being considered." An apparent conflict of est may damage both the public's and other researcher's sistent with standard methods for questionnaire constructhe technology [15]. interest exists when the existence of an interest could trust in science [11]. The type of conflict most likely to tion result in the expert's objectivity being questioned by othaffect the public's is a financial conflict whereZealand the sciIn a more recent trust selective review for the New ers, and a "potential of interest exists to with an entist tends to gain financially from a particular research • experimentation withconflict more formal procedures arrive Guidelines Group, the following criteria were identified as interest which any reasonable person could be uncertain outcome [11-16], although other competing interests, at group judgments indicating that a topic is suitable for guideline developwhether or not should be reported" [20]. such professional advancement, are important. Conment as [16]: flict of interest policies are designed to protect the integThey also suggested there is a need to define more narThe Declaration identifies types of financial and other rity of topic research and decision-making processes through rowly and precisely topics5 for guideline development. 1. The is clinically important affecting large numbers of interests that must be disclosed by all experts, including disclosure and transparency. They argued that this would result in more efficient organpeople with substantial morbidity or mortality (the burden of proprietary interests and patents, shares or bonds in a ization of panels and their work, resolution of some illness). related commercial entity, employment or consultancies, The following report relies heavily on published research apparent controversies, more responsive guidelines, and paid or research, and grants or fellowships from a related to conflicts ofenough interest the context of U.S. easierwork implementation. 2. The topic is complex to in initiate debate about the acareccommercial entity that has an interest in the subject-matdemic research and U.S. and U.K. biomedical journals ommendations. ter or work of the committee. [20] because there is little empirical research from other areas.

Page 4 2 of 7 6 (page number not for citation purposes)

the use of research evidence in guideline development Health Research PolicyImproving and Systems 2006, 4:16 http://www.health-policy-systems.com/content/4/1/16

We are not aware of specific WHO documents providing guidance on how to avoid or manage conflicts of interest, and we know of no processes required to ensure that the committees discuss potential conflicts of interest on a case-by-case basis and handle them appropriately. There may be some variability in how departments collect and manage the disclosed information.

solicit information; they employ different standards for determining conflicts of interest and for managing them (see below).

Methods

The methods used to prepare this review are described in the introduction to this series [24] Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We searched PubMed and the Cochrane Methodology Register [25] for existing systematic reviews and relevant methodological research that address these questions. We did not conduct systematic reviews ourselves. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

In October 2005, the WHO Office of Legal Counsel recommended a set of proposed revisions to the existing conflict of interest procedures that are similar to the recommendations in our report. These revisions would clarify the definition of a conflict of interest, include recommendations for avoiding situations that might result in conflicts of interest, and expand the relationships and affiliations that must be disclosed. The draft guidelines also recommend that a determination be made as to whether the expert's declared interest is insignificant, clearly significant, or potentially significant (para. 26). Suggestions for making this determination include weighing the nature and extent of the interest, the context of the work, and the importance of the expert's contribution (para. 29). The draft Guidelines also suggest three options for managing a conflict: 1) continue with public disclosure of the interest; 2) limit the expert's involvement; or 3) exclude the expert from the meeting or work altogether (para. 30)[21]

For this review, we searched PubMed for original qualitative and quantitative research using the terms "conflicts of interest" and "disclosure" and the Cochrane Methodology Register using "conflict of interest". We searched the reference lists of all relevant publications, consulted references from the Council of Medical Editors meeting on disclosure (Sept 2004) and selectively searched for the published policies of several organizations, including the Cochrane Collaboration, NICE, FDA, and National Academies of Science [26-30].

The draft guidelines also include a requirement that WHO experts disclose ties to the tobacco industry. This recommendation is in response to a 2000 commissioned report investigating the influence of the tobacco industry on WHO's global tobacco control policies. That report recommended that WHO formally vet prospective experts, consultants, and advisers for possible conflicts of interest related to the tobacco industry and that staff should be barred from having links with the tobacco industry [22]. In 2003, WHO's hypertension guidelines were revised in response to criticism about possible conflicts of interest among expert members [23].

Findings

Our database searches did not yield any systematic reviews of conflict of interest or financial disclosure policies. We found several systematic reviews of literature examining the association between commercial sponsorship and outcomes favorable to the sponsor and the financial ties of investigators and favorable outcomes. We also found a number of empirical studies of particular aspects of industry involvement in science and medicine, case studies and commentaries. What is the best way to obtain complete and accurate disclosures on financial ties and other competing interests? We were unable to identify any randomized, controlled trials or other rigorous studies evaluating different methods for obtaining conflict of interest disclosures. Biomedical journals gather financial interest statements from authors of submitted manuscripts in three ways: 1) minimal requests about authors' professional and financial affiliations that may be perceived to have biased the presentation of results; 2) detailed instructions that request authors to describe all involvements with organizations or entities with direct financial interest in the subject matter of the study; and 3) detailed, structured checklists that require authors to declare specific interests [31]. Krimsky and others are critical of the utility of minimal and open-

What other organizations are doing

Many organizations recognize the importance of protecting against actual and potential conflicts of interest and require special employees, advisory committee members, and participants to disclose their financial ties to the organization. For example, the US Food and Drug Administration (FDA), the Cochrane Collaboration, the UK National Institute for Health and Clinical Excellence (NICE), and the US National Academies of Science all require advisory committee members and other special participants to disclose financial relationships, including research sponsorship, equity ownership, consulting fees, honoraria, related to the work or topic of the committee. These organizations use a structured disclosure form to

Page 3 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:16

http://www.health-policy-systems.com/content/4/1/16 http://www.health-policy-systems.com/content/4/1/14

and therequests ended Guidelines [31].International Bero et al. caution Network that[8]) simple for existing disclosure requests systematic reviews may and not relevant reveal the methodological nature and extent research to which that address commercial these questions. interests exert The answers influencetoover the questions the scientificour are process conclusions [32]. based on the available evidence, consideration of what WHO and other organisations are The Cochrane doing, and logical Collaboration arguments. Steering Group members, the Food and Drug Administration advisory committee members, For this review NICE, weand searched the National PubMedAcademies using (clinical use strucpracturedguidelines tice disclosure or forms public and health request guidelines) information and (priority on a range of setting or financial setting priorities) ties, including and related research articles funding, for selected paid consultancies, references [9,10]. honoraria, We searched equitythe holdings, Cochrane gifts, Methodolpatents, and Register ogy royalties.using The Cochrane priority orCollaboration priorities. We also reviewed requests the International Conference on information website of theon5thpositions of management in aPriorities related in Health Care [11] and as references that we had in our trusfiles entity, including service a director, officer, partner, [12-15]. tee or employee, and information on outstanding loans from the entity. The National Academies of Science request disclosure of any position that would give the Findings What criteria should used to establish priorities? individual access to be confidential information, including The US records, Instituteclassified of Medicine's (IOM) Committee on patient and proprietary information. Methods for Setting Priorities for Guidelines DevelopNICE requests information regarding an individual's priment in its study of setting priorities for outcome clinical practice vate practice that could be affected by the or disguidelines 1995 or argued that the priority setcussion of apublished particularinmatter product. ting process should be open and defensible [14] They recommended six general criteria: prevalence, burden of There is considerable variation along other dimensions illness, costasofwell. managing the problem, variability in pracdisclosure These include: tice, potential of a guideline to improve health outcomes, disclosures should be and potential of a guideline to made: reduceUpon costs. appointment Because data • When to thetocommittee? to the start of committee work? used make these Prior judgements is often lacking, they sugUnder agenda item? At the of eachunmeasured committee gested each explicit opportunities for start important meeting? factors to be considered. They further suggested separate "tracks" for considering issues for specific populations, financialThey interest should •conditions What levelorofconcerns. argued thatbe thedisclosed: rationalesAny for amount (>US$0)? Over should US$250/per year clearly; in annual designating special tracks be defined i.e. income? US$10,000 in equity holdings? Exact why they Over warrant special consideration. They suggested amounts or ranges (i.e., US$1000–$5000)? that updating of guidelines should also be considered as a separate "track", taking account of issues such as the need should be covered by the disclo• What period and of time for corrections the availability of new evidence. sure: The current calendar year? Past 12 months? Past 5 years? Pastidentified 5 years and 2 years? Oortwijn 25future criteria used to prioritise health technology assessments and categorised these into four the disclosure Individual only? Indi•broad Who should categories: burden ofcover: disease, potential effects, vidual's Individual's potentialspouse costs, and and children? uncertainty regarding institution? application of the technology [15]. How to determine when a disclosed financial tie or other competing constitutes a conflict of interest? In a more interest recent selective review for the New Zealand Few organizations or institutions provide explicit guideGuidelines Group, the following criteria were identified as lines for determining when a particular financial developrelationindicating that a topic is suitable for guideline ship constitutes a conflict of interest. The Association of ment [16]: American Medical Colleges prohibits financial relationships principal investigators 1. The between topic is clinically important affectingand largecommercial numbers of sponsors ofsubstantial clinical trials, but uses "rebuttable people with morbidity or amortality (the presumpburden of tion" clause to allow the prohibition to be waived when illness). the benefits of the research outweigh the risks of the conflict oftopic interest [33]. The US National of the Health 2. The is complex enough to initiate Institutes debate about recand National Science Foundation establish financial ommendations.

thresholds for disclosure – $10,000 inactual annual income or 3. There is evidence of variation between and appropri5%care. equity ownership in a commercial entity related to the ate scientific work [34]. 4. There are no existing valid guidelines available to use. The US FDA does not prohibit financial relationships among Advisory members and available. regularly 5. There its is an adequateCommittee amount of existing evidence issues waivers for disclosed conflicts of interest when 1) "the disqualifying financial is nottosothe substantial 6. The recommendations will interest be acceptable potential that it is likely to affect the integrity of an employee's servusers. ices to the government;" and 2) the "need for the employee's services outweighs potential of 7. Implementation of the guidelinethe is feasible, willconflicts not exhausinterest" In making resources, these determinations, FDA tively use [29]. the communities' and barriers tothe clinical evaluates interest creating the disqualificachange are "the not sotype highof that they cannot be overcome. tion; the identity of the person whose financial interest is at issue; the dollar valueis of the disqualifying While burden of disease commonly used as a financial criterion interest including value inberelationship to the for priority setting,its it should noted that the use individof sumual's overall assets; the nature and importance the indimary burden of disease measures, such asofdisability vidual's role in the(DALYs) matter, has including the extent which adjusted life years been criticised forto focusing the employee is called upon to exercise discretion; the senon disease rather than resource use and interventions, sitivity ofofthe and theabout need for the employee's because thematter; assumptions values inherent in servsuch ices in the and particular matter" measures, because of the [29]. technical limitations of such measures (see for example references [17] and [18]). These criteria are in line with the criteria used by UniverWhatofprocesses should be used to agree on priorities? sity California conflict of interest committees. In the Batista and Hodge review 10 of years ago only empirical studyintoa date of conducted how conflict interest found only three pertinent to priority settingrelafor committees definearticles and manage disclosed financial clinical practice guidelines [10].[35], Theyfound suggested the foltionships of faculty investigators that commitlowing framework for priority tees typically examined the setting: nature of the proposed scientific work (basic or applied), the overlap between 1. Consult with end andtopic, otherthe stakeholders paid activities and theusers research length andbefore dollar selectingof topics. amount the relationship between the investigator and the commercial entity, and the degree to which the inves2. Consider the consultation. tigator couldfeasibility be seen during as independent of the company's interests. 3. Document the process used to select guideline topics. The overall lack of explicit criteria for determining which The IOM suggested the following procedures [14]: a comrelationships constitute conflicts of interest reflect mon perception that these decisions should be made on • the use basis of Delphi-like procedures for obtaining expert an ad hoc and that the organization must always baljudgments orneeds topic for rankings throughexpertise correspondence ance its own the particular of the individual with the needs of the public (in terms of advancing • the use discovery of questions that as aretrust specific, explicit andprocconscientific as well in the scientific sistent withisstandard methods about for questionnaire construcess). Little known, however, the needs and undertion standings of the public in this regard. The few studies we have identified to date provide evidence of both favorable • experimentation with more procedures arrive and unfavorable reactions of formal the public [36,37].toProfesat groupwith judgments sionals industry ties are more supportive of financial ties than those without industry ties. Investigators recogTheygeneral also suggested there is of a need to define narnize risks of conflicts interest, but notmore for themrowly Investigators and preciselytend topics for guideline development. selves. to support disclosure of financial They although argued that this is would resultthat in more efficient organties, there evidence disclosure leads to izationcritical of panels and work, resolution some more review of their research findings [38]. of Schroter apparent controversies, morethe responsive guidelines, releand and colleagues showed that overall importance, easier implementation. vance, validity, and believability of studies disclosing

Page 4 of 7 6 (page number not for citation purposes)

the use of 4:16 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/16

competing interests were rated lower by readers than those without competing interests. [39]

essary and can taint the reputation of "good" researchers [49,50]. Others believe that "the key to avoiding conflict of interest is public disclosure" [51]. Studies that disclose industry sponsorship have a systematic bias towards outcomes that favor the sponsor [3,5,52,53], so, therefore, disclosure does not eliminate bias. Although disclosure does not eliminate the association of research funding with outcomes favorable to the sponsor, many argue that it can minimize perceived conflicts of interest.

When a conflict of interest is identified, how should the conflict be managed? The only empirical studies of management decisions in conflicts of interest detail a number of management strategies that are commonly used by university conflict of interest committees [35,40]. These possible management strategies include: disclosure of the financial tie(s) in publications and public presentations; reducing equity holdings to below 5%; altering consulting agreements to ensure separation between consulting and research work; eliminating the financial tie; appointing oversight committees to review the scientific process and resulting research; and recusal. Government and professional society guidelines recommend that institutions "manage" the financial conflicts of interest of their researchers. Disclosure of financial ties in all publications and presentations is the most frequently used management strategy [40-43].

Additional research is necessary to be able to evaluate different methods for defining conflicts of interest and to determine their relative impact on the decision-making capabilities of the organization. How should conflict of interest policies be enforced? There is no empirical evidence evaluating the enforcement of conflict of interest policies. Most organizations and academic institutions convene a standing or ad hoc committee to review financial interest disclosures and, where deemed necessary, recommend management strategies. The US FDA vets all financial disclosure statements through a multi-stage process, beginning with initial review, followed by consultation with the individual and an FDA official, review by the FDA Ethics staff, and final approval by the appointing official. The FDA operates under federal regulations and thus has the power to enforce its decisions [28,29]. The Cochrane Collaboration directs unclear cases of financial disclosure for reviews to a "Funding Arbiter" who convenes a standing panel of four to give guidance [26].

Scientific journals are also encouraging disclosure as a way of dealing with financial conflicts of interest [44], however, the adequacy of disclosures in scientific articles has been questioned [32,45]. Even when financial sponsorship is disclosed, few studies describe the role of the sponsor [46]. A study of the relationships between authors of clinical practice guidelines and the pharmaceutical industry found considerable interaction between guideline authors and the pharmaceutical industry [47]; another study found that clinical practice guidelines published in journals almost never published conflict of interest statements along with the guidelines [48].

Further work

There is currently a lack of empirical evidence regarding the most effective ways to determine the existence of conflicts of interest, manage conflicts of interest, and enforce conflict of interest policies. Additional research is necessary to evaluate different methods for defining conflicts of interest and to determine their relative impact on the decision-making capabilities of the organization. WHO's proposed draft recommendations (October 7, 2005) represent a more rigourous evaluation of conflict of interest because it requires more complete disclosure, clearer standards for evaluating conflicts of interest, and explicit management strategies.

The FDA in 2002 issued draft guidance amending their disclosure regulations related to Advisory Committee members. The draft guidance now requires that Advisory Committee members granted waivers of their conflicts of interest will have the nature and magnitude of their conflicts of interest disclosed and read into the public record at the start of the committee hearings. [28]. NICE, which is currently reviewing its policies on disclosure and conflicts of interests, recommends that "members should declare all interests at the beginning of all appraisals" and that those declarations of interests be kept in files available for public scrutiny or are recorded in the minutes of the meeting [27]. The Cochrane Collaboration publishes the declarations of interests of its Steering Group members [26].

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

Although disclosure of financial ties is becoming more accepted within the scientific and policy communities, there are widely varying opinions about the adequacy of disclosure as a management strategy for financial conflicts of interest. Some critics of disclosure feel that it is unnec-

EAB prepared the first draft and revisions of this paper. LAB contributed to drafting and revising it.

Page 5 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:16

http://www.health-policy-systems.com/content/4/1/16 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing Acknowledgements We gratefully acknowledge therelevant members of the ACHR SURE, and all of systematic reviews and methodological research those who have provided us with feedback earlier versions this paper. that address these questions. The on answers to the of questions are our conclusions based on the available evidence, conReferences sideration of what WHO and other organisations are 1. Barnes DE, Bero LA: Why review articles on the health effects doing, and logical arguments. of passive smoking reach different conclusions. Jama 1998,

29. There Policies and procedures for between handling actual conflicts interest 3. is evidence of variation andofappropriwith FDA advisory committee members, consultants, and ate care. experts. U.S. Food and Drug Administration; 2005. 30.

Policy on committee composition and balance and conflcits

of interest for committees used inavailable the development of 4. There are no existing valid guidelines to use.

reports. The National Academies; 2003. Krimsky S: Science in the Private Interest. Lanham, Maryland , Rowan 2003:247. 5. There is &anLittlefield; adequate amount of existing evidence available. 32. Bero LA, Glantz S, Hong MK: The limits of competing interest disclosures. Tobacco Control 2005, 14:118-126. 6. recommendations be Colleges: acceptable to the potential 33. The Association of American will Medical Protecting subjects, users.preserving trust, promoting progress: Guidelines for developing and implementing a policy concerning individual financial interests in human subjects research. Philadelphia ; 2001. 7. guideline is feasible, willObjectivity not exhaus34. Implementation U.S. Departmentofofthe Health and Human Services: in Volume 35810 (codified at 42 CFR 60 Federal tivelyresearch. use the communities' resources, and50.barriers to Register; clinical 1995. change are not soS,high they cannot beofovercome. 35. Boyd E, Lipton Berothat L: Implementation financial disclosure policies to manage conflicts of interest. Health Affairs 2004, 23(2):206-214. While burden of disease is commonly used as a criterion 36. Sprecker K: How involvement, citation style, and funding for priority setting, should beofnoted that scientists. the use of Science sumsource affect theitcredibility university 2002, 24(1):72-97. maryCommunication burden of disease measures, such as disability 37. Kim SYH, et al.: Potential research participants' views regardadjusted life yearsand (DALYs) has been criticised foroffocusing ing researcher institutional financial conflicts interest. Journal of Medical Online Electronic p. 1-18 on disease ratherEthics than2003. resource use andVersion: interventions, 38. Glaser Beroassumptions L: Attitudes ofabout Academic andinherent Clinical Researchbecause ofB,the values in such ers toward Financial Ties in Research. Science and Engineermeasures, and 2005, because of the technical limitations of such ing Ethics. 11:553-573. 39. Schroter S, Morris J, Chaudhry S, Smith RD, Barratt Does the measures (see for example references [17] and H: [18]). type of competing interest statement affect readers' perceptions of the credibility of research? A randomized trial. BMJ What processes should be used to agree on priorities? 2004, 328:742-743. 40. Boydand EA Ph.D., Beroin LA aPh.D.: Assessing Faculty 10 Financial Batista Hodge review conducted yearsRelaago tionships with Industry. Journal of the American Medical Association found only three articles pertinent to priority setting for 2000, 284(November 1, 2000):2209-2238. 41. Boydpractice EA, Cho MK, Bero LA: Financial conflict-of-interest clinical guidelines [10]. They suggested the polifolciesframework in clinical research: issuessetting: for clinical investigators. Acalowing for priority demic Medicine 2003, 78(8):769-774. 42. Krimsky S Rothenberg, LS., Stott, P., Kyle, G.: Financial interests on authors in end scientific a principal investigator lot 1. Consult with usersjournals: and other stakeholders before study of 14 publications. Science and Engineering Ethics 1996, selecting topics. 2(4):395-410. 43. Shamoo AE, Resnik DB: Responsible Conduct of Research. New York , Oxford University Press; 2003:345. 2. Consider feasibility during the consultation. 44. Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication. International Committee Medicalguideline Journal Editors; 2005. 3. Document the process used toof select topics. Available at: http://www.icmje.org 45. Krimsky S, Rothenberg LS: Conflict of interest policies in science The IOM suggested the following procedures [14]: discloand medical journals: Editorial practices and author sures. 2001, 7:205-218. 46. Gross CP, Gupta AR, Krumholz HM: Disclosure of financial com• thepeting use of Delphi-like procedures for obtaining expert interests in randomised controlled trials: cross sectional review. BMJrankings 2003, 326:526-527. judgments or topic through correspondence 47. Choudhry NK, Stelfox HT, Detsky AS: Relationships between authors of clinical practice guidelines and the pharmaceuti• thecaluse of questions that are specific, explicit and conindustry. JAMA 2002, 287(5):612-617. 48. Papanikolaou GN, Baltogianni MS, Contopoulos-Ioannidis DS, Haidich sistent with standard methods for questionnaire construcAB, Giannakakis IA, Ioannidis JPA: Reporting of conflicts of intertion est in guidelines of preventive and therapeutic interventions. BMC Medical Research Methodology 2001, 1(3):. Jansen LA, Sulmasywith DP: Bioethics, conflicts of interest, the •49.experimentation more formal procedures toand arrive limits of transparency. Hastings Center Report 2003, 33:40-43. at judgments 50. group Rothman K: Conflict of interest: The new McCarthyism in science. JAMA 1993, 269:2782-2784. 51. Duderstadt J: A university for the 21st century. Ann Arbor , UniThey also suggested there is a need to define more narversity of Michigan Press; 2000. rowly andD,precisely topics guideline development. 52. Barnes Bero L: Why reviewfor articles on the health effects of different conclusions. Journal of the Theypassive arguedsmoking that thisreach would result in more efficient organAmerican Medical Association 1998, 279:1566-1570. ization ofD,panels and their work, resolution of articles some 53. Barnes Bero L: Scientific quality of original research on environmental tobacco smoke. Tobacco Control 1997, apparent controversies, more responsive guidelines, and 6:19-26. easier implementation. 31.

279(19):1566-1570.

2. this Barnes DE, Bero ScientificPubMed quality ofusing original researchpracartiFor review weLA: searched (clinical cles on environmental tobacco smoke. Tob Control 1997, tice guidelines or public health guidelines) and (priority 6(1):19-26. 3. Bekelman JE, Li Y, Gross CP: Scope and impact of financial consetting or setting priorities) and related articles for selected flicts of interest in biomedical research: A systematic review. references [9,10]. We searched the Cochrane MethodolJAMA 2003, 289(454-65):. ogy Register usingLA:priority or priorities. We reviewed 4. Cho MK, Bero The quality of drug studies publishedthe in symposium Ann Intern Med 1996, 124(5):485-489. Conference on Priorities website of the proceedings. 5th International 5. Lexchin J, Bero LA, Djulbegovic B, Clark O: Pharmaceutical indusin Health Care [11]and andresearch references that we in our files try sponsorship outcome andhad quality: systematic review. Bmj 2003, 326(7400):1167-1170. [12-15].

Wilson D, Heath D: He saw the tests as a violation of 'trusting desperate human beings'. In The Seattle Times Seattle ; 2001. Findings 7. Wilson D, Heath D: Uninformed consent: they call the place Mother Hutch. In The Times Seattle ; 2001. What criteria should be Seattle used to establish priorities? 8. Dana J: Harm avoidance and financial conflict of interest. JourThe nal USof Medical Institute of Medicine's (IOM) Committee on Ethics 2003, Online Electronic Version:1-18. Methods forTobacco Settingindustry Priorities for Guidelines Develop9. Bero LA: manipulation of research. Public Repstudy in press. mentHealth in its of setting priorities for clinical practice 10. Montori VM, Jaeschke R, Schunemann HJ, Bhandari M, Brozek JL, guidelines published in 1995 that the priority setDevereaux PJ, Guyatt GH: Users'argued guide to detecting misleading in should clinical research BMJ 2004, [14] 329:1093-1096. ting claims process be openreports. and defensible They rec11. Friedman PJ: The impact of conflict of interest on trust in sciommended six general criteria: prevalence, burden of ence. Science and Engineering Ethics 2002, 8(3):413-420. illness, cost of managing the problem, variability in prac12. Drazen JM, Koski G: To protect those who serve [editorial]. New England of Medicine 2000, 343:1643-1645. tice, The potential of aJournal guideline to improve health outcomes, 13. Baltimore D: The worsening climate for biological research; is and the potential a guideline reduce Technology costs. Because public of losing faith in to science? Review data 1989, used92(4):22. to make these judgements is often lacking, they sugB, D. P: How to restore trust in science [com14. Haerlin gested explicit opportunities forpublic important unmeasured mentary]. Nature 1999, 400:499. factors to JJ: beTrust considered. They further suggested separate 15. Cohen us to make a difference: Ensuring public confidence the integrity of clinical Medicine "tracks" forinconsidering issues forresearch. specificAcademic populations, 2001, 76(2):209-214. conditions orCD: concerns. They argued the rationales for 16. DeAngelis Conflict of interest andthat the public interest. Jourdesignating specialMedical tracksAssociation should2000, be defined clearly; i.e. nal of the American 284(17):2237-2238. 17. A, Schünemann Oxman AD: Improving the Use of why Fretheim they warrant specialHJ,consideration. They suggested Research Evidence in Guideline Development: 3. Group that composition updating ofand guidelines should also be considered as a consultation process. Health Research Policy and Systems 2006. separate "track", taking account of issues such as the need 18. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of for corrections and the availability of new evidence. Research Evidence in Guideline Development: 5. Group processes. Health Research Policy and Systems 2006. 19. World identified Health Organisation: Regulations Expert Advisory Oortwijn 25 criteria used toforprioritise health Panels and Committees. 2003:101-109. technology assessments and categorised these into four 20. World Health Organisation: Global Programme on Evidence for Health Policy, Guidelines Guidelines. 2003:15. broad categories: burden for of WHO disease, potential effects, 21. World costs, Health Organisation: Draft Guidelines Declaration of of potential and uncertainty regardingforapplication Interest (WHO Experts). 2005. the technology [15].industry's smoke screen blown. The Scientist 22. Godlee F: Tobacco 2000. 23. World Health Organisation: WHO/ISH Hypertension GuideIn alines. more2003. recent selective review for the New Zealand Guidelines Group, theA,following criteria were identified 24. Oxman AD Fretheim Schünemann HJ: Improving the Use as of Research Evidence in Guideline Development: Introduction. indicating that a topic is suitable for guideline developHealth Res Policy Syst 2006. mentCochrane [16]: Methodology Register [http://www.cochrane.org/ 25. access_data/cmr/accessDB_cmr.asp] 26. Cochrane Collaboration: Cochrane Collaboration policy on 1. The topic is clinically important affecting large numbers of commercial sponsorship. 2004. people with Institute substantial morbidity or mortality (the burden of 27. National for Clinical Excellence: Review of code of practice on declaration of conflicts of interest. . illness). 28. Draft guidance on disclosure of conflicts of interest for special government employees participating in FDA product 2. The topic advisory is complex enough to initiate debate about the recspecial committees. U.S. Food and Drug Administration; 2002. ommendations. 6.

Page 4 6 of 7 6 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 5. Group processes Atle Fretheim*1, Holger J Schünemann2 and Andrew D Oxman3 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway, 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 3Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway Email: Atle Fretheim* - atle.fretheim@nokc.no; Holger J Schünemann - hjs@buffalo.edu; Andrew D Oxman - oxman@online.no * Corresponding author

Published: 01 December 2006 Health Research Policy and Systems 2006, 4:17

Received: 07 April 2006 Accepted: 01 December 2006

doi:10.1186/1478-4505-4-17

This article is available from: http://www.health-policy-systems.com/content/4/1/17 © 2006 Fretheim et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Objective: In this review we address approaches to facilitate sound processes within groups that develop recommendations for health care. Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key question and answer: What should WHO do to ensure appropriate group processes? Various strategies can be adopted to ensure that the group processes in play when panels are developing recommendations are inclusive, so that all voices can be heard and all arguments given fair weight, including • the use of formal consensus development methods, such at the Nominal Group Technique or the Delphi method • the selection of a group leader who is qualified and responsible for facilitating an appropriate group process.

Page 1 of 4 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:17

http://www.health-policy-systems.com/content/4/1/17 http://www.health-policy-systems.com/content/4/1/14

Background

clinical practice guidelines. Three of the most common methods reaching consensus are the Nominal Group 3. There isfor evidence of variation between actual and appropriTechnique (NGT), the Delphi Method, and Consensus ate care. Conferences. A brief description of commonly used consensus is found in Table 1. How4. Theredevelopment are no existingmethods valid guidelines available to use. ever, there is considerable variation in how these techniques areadequate implemented 5. There is an amountinofpractice. existing evidence available.

The World Health Organization other and the Guidelines International (WHO), Networklike [8])many for existing organisations around world, methodological has recognised the need systematic reviews andthe relevant research to use more these rigorous processes ensure to that care that address questions. Thetoanswers thehealth questions recommendations informed by the evidence, best available are our conclusionsare based on the available conresearch evidence. is theand fifthother of a series of 16 reviews sideration of whatThis WHO organisations are that have been prepared as background for advice from doing, and logical arguments. the WHO Advisory Committee on Health Research to WHO onreview how to this. For this weachieve searched PubMed using (clinical practice guidelines or public health guidelines) and (priority A grouporthat is convened formulate recommendations setting setting priorities)toand related articles for selected will necessarily enter sort ofthe consensus development references [9,10]. Wesome searched Cochrane Methodolprocess. The using panel-members pass judgements on the ogy Register priority orwill priorities. We reviewed availableofresearch evidence, consider various Conference on trade-offs Priorities website the 5th International (between harmsthat and we costs), finally in Health expected Care [11]benefits, and references had and in our files try to reach a consensus on what recommendation to [12-15]. make. The consensus may be reached through informal processes or more formal methods. Findings What criteria should be used to establish priorities? In this we address the following What The USpaper Institute of Medicine's (IOM) question: Committee on should do to ensure appropriate group processes? MethodsWHO for Setting Priorities for Guidelines DevelopQuestions to setting group composition, integrating valment in itsrelated study of priorities for clinical practice ues and consumer involvement are addressed in other guidelines published in 1995 argued that the priority setpapers in this series be [1,2]. ting process should open and defensible [14] They recommended six general criteria: prevalence, burden of What is WHO doing the now? illness, cost of managing problem, variability in pracWe are not aware of any examples of the use outcomes, of formal tice, potential of a guideline to improve health consensus development methods by costs. groups that have and potential of a guideline to reduce Because data developed recommendations onisbehalf of WHO.they In docused to make these judgements often lacking, sugumentsexplicit describing procedures for for Expert Committees, it is gested opportunities important unmeasured stated that meetings They "shallfurther normally be ofseparate private factors to bethe considered. suggested character" "tracks" for[3]. considering issues for specific populations, conditions or concerns. They argued that the rationales for In our literature search weshould identified a paper clearly; describing designating special tracks be defined i.e. the use formal special consensus development methods in why theyofwarrant consideration. They suggested modifying WHO's "Guidelines foralso thebemanagement that updating of guidelines should considered asofa HIV/AIDS in adults andaccount children" for usesuch in Malawi and separate "track", taking of issues as the need Barbados [4]. The method employed was the Nominal for corrections and the availability of new evidence. Group Technique. Oortwijn identified 25 criteria used to prioritise health What are assessments other organisations doing? technology and categorised these into four Several categories: formal approaches for disease, reachingpotential consensuseffects, exist, broad burden of and somecosts, organisations use theseregarding in the development potential and uncertainty application of the technology [15].

In The a recent international of organisations that 6. recommendations willsurvey be acceptable to the potential develop clinical practice guidelines or health technology users. assessments, approximately 42 % of the respondents reported using formal 7. Implementation of the consensus guideline is development feasible, will notmethods exhaus[5]. Ause smaller survey of prominent guideline tively the communities' resources, and barriersdevelopers to clinical reported 7 high of 18that programs used consensus change arethat not so they cannot be formal overcome. methods to formulate recommendations [6]. While burden of disease is commonly used as a criterion With informal consensus development, a use strategy is for priority setting, it should be noted that the of sumneededburden to ensure group processes. Typically mary of appropriate disease measures, such as disability responsibility for this is given the criticised group leader. Conseadjusted life years (DALYs) hasto been for focusing quently, much weight put on selecting right person on disease rather thanis resource use andtheinterventions, for this of position. The National for Health and because the assumptions aboutInstitute values inherent in such Clinical Excellence (NICE) in the UK notes that the group measures, and because of the technical limitations of such leader "needs sufficient time[17] for all measures (see to forallow example references andmembers [18]). to express their views without feeling intimidated or threatened shouldshould check be thatused all the members in the groups Whatand processes to agree on priorities? agree endorse [7].years FurtherBatistatoand Hodgeany in arecommendations" review conducted 10 ago more, should pertinent be selected someone whofor is found "The only Chair three articles toas priority setting neutral and whoguidelines has enough in coordinating clinical practice [10].expertise They suggested the folgroups health professionals and patients/carers so that lowing of framework for priority setting: the appointment is acceptable to all." [7]. 1. Consult with end users and other stakeholders before Methods selecting topics. The methods used to prepare this review are described in theConsider introduction to this seriesthe [8]. Briefly, the key ques2. feasibility during consultation. tions addressed in this paper were vetted amongst the authors and the process ACHR used Subcommittee on the topics. Use of 3. Document to select guideline Research Evidence (SURE). We searched PubMed and three databases methodological literature The IOM suggested of the following procedures [14]: (the Cochrane Methodology Register [9], the US National Guideline Clearinghouse [10] and thefor Guidelines â&#x20AC;˘ the use of Delphi-like procedures obtainingInternaexpert tional Network [11]) for existing systematic reviews and judgments or topic rankings through correspondence relevant methodological research that address these questions. We of did not conduct systematic â&#x20AC;˘ the use questions that are specific,reviews explicitourselves. and consistent with standard methods for questionnaire construction Murphy et al. [12]) Table 1: Characteristics of various consensus development methods (from In a more recent selective review for the New Zealand Consensus development method Mailed Private decisions elicited Formal feedback Face-to-face contact Interaction Aggregation method â&#x20AC;˘ofexperimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as questionnaires group choices structured at group judgments indicating that a topic is suitable for guideline development [16]: Informal No No No Yes No Implicit Delphi method Yes Yes Yes to define Explicit They Yes also suggested No there is a need more narNo Yes Yes Explicit rowlyYesand preciselyYes topics for guideline development. 1.NGT The topic is clinically important affecting large numbers of RAND version Yes Yes Yes Yes Yes Explicit They No argued that thisYes would result in people with substantial morbidity orNo mortality (the burden of Consensus development conference No No more efficient Implicitorganization of panels and their work, resolution of some illness). Other methods Staticised group No Yes No controversies, No more responsive Explicit and apparent guidelines, Social judgement analysis No Yes Yes Yes No Implicit easier implementation. 2. The topic is complex enough to initiate debate about the recStructured discussion No No No Yes Yes Implicit ommendations.

Page 4 2 of 7 4 (page number not for citation purposes)

the use of 4:17 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/17

The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

Concerning characteristics of a group facilitator, the research base is difficult to interpret as "the models of leadership used are often not directly transferable to facilitation". Although there is "very little work that looks at the effects of facilitation on group decision-making", the reviewers believe that "it is likely that this key role will influence group decision-making."

In the literature search we used the term "consensus and process and method". We also checked the reference lists of key papers and contacted researchers in the field.

Findings

We identified one additional study that compared informal consensus with a formal consensus method ("the appropriateness method") for developing clinical practice guidelines on the management of low-back pain [14]. The investigators found that guideline statements resulting from the two approaches were "qualitatively similar", however the formal method produced statements that in some instances were "more clinically specific".

What should WHO do to ensure appropriate group processes? One comprehensive review on consensus development methods was identified, and it provided most of the key findings for this report [12], together with an updated review that largely confirms the findings [13]. The review addresses three questions related to interaction within guideline development groups: 1) Does the choice of consensus development method influence the group's decision? 2) Does the setting for the group meetings affect the consensus decision? 3) Do the characteristics of a group facilitator affect the consensus decision?

Discussion

The idea of bringing people together to develop recommendations is based on the understanding that they all have something to contribute. Thus, it is essential to secure that all participants can be heard and have the opportunity of influencing the outcome of the process. This is a common understanding among groups that develop guidelines, and many have therefore adopted specific strategies to ensure appropriate group processes.

Various measurements of decision quality were used as outcome measures in the comparative studies that were included in the review. For instance, comparison with "gold standard", such as asking the groups to reach an agreement on "questions that have correct answers which the participants do not know with any precision," e.g. "What is the diameter of Jupiter?". For ranking tasks (e.g. "to rank items in terms of their value for survival on the moon"), the group decision "can be compared with rankings by experts." The applicability of these types of studies for processes taking place within guideline development groups is not obvious.

Given the costs of group meetings, different languages and cultural differences, it is especially important for WHO to ensure that all of the invited members contribute fully to the development of recommendations. Transparency is important to ensure that groups know and adhere to the methods that they are supposed to be use. For instance, the group may report that they base their recommendations on research evidence, while they in reality reach their conclusions on a different basis. A qualitative study of decision-making processes within drug-selection committees in hospitals in the UK, for example, found that many decisions were not based on research findings, despite being reported as if they were: "reports of decisions...are written so as to account for the decision in terms of scientific rationality...rather than the local rationality that was actually employed" [15].

For choice of consensus development methods, the reviewers identified 16 studies comparing NGT with informal methods, 11 comparing the Delphi method with informal methods, and seven studies comparing NGT and Delphi. Interpreting the results is not straight-forward since "the studies also differ in the particular way they operationalise the method used". The reviewers did not find any comparative studies involving consensus development conferences. Their summary conclusion was that "Formal methods generally perform as well or better than informal methods, but it is difficult to tell which of the formal methods is best."

The research base to inform the choice of strategy to ensure appropriate group processes is limited, however in addition to logical arguments there is also some empirical evidence in support of using formal consensus development methods rather than relying only on informal processes. Having a group leader that facilitates the group process is likely essential. Conflicts may arise within groups and the leader of the group will have an important role in trying to manage these. Dealing with conflict is usually a difficult task, and WHO should consider establishing routines to support groups in managing these.

With regards to the settings for group meetings, the reviewers concluded that "There is little research which actually looks at this question. However, of the many factors which can influence decision-making, except for extreme environments, the environment is likely to have only a marginal impact."

Page 3 of 4 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:17

http://www.health-policy-systems.com/content/4/1/17 http://www.health-policy-systems.com/content/4/1/14

and A weakness the Guidelines of ourInternational review was the Network literature [8]) for search existing was limited to PubMed systematic reviews and three relevant databases methodological of methodological research literature that address and these did questions. not include The additional answers to searches the questions in the social are ourscience conclusions literature. based on the available evidence, consideration of what WHO and other organisations are doing, andwork logical arguments. Further In general, there is need for research to learn more about the this relative merits of various methods facilitating For review we searched PubMed usingfor (clinical pracsound group processes. to guidelines) head comparisons of diftice guidelines or publicHead health and (priority ferent consensus development methods within groups setting or setting priorities) and related articles for selected that develop recommendations for health careMethodolshould be references [9,10]. We searched the Cochrane done, since most far has taken in very ogy Register usingresearch priority so or priorities. We place reviewed the International Priorities differentofsettings. research Conference is needed toon identify the website the 5th Also, in Health Care [11] and references that we had our filesa most critical selection criteria and processes for in selecting [12-15]. chairperson for groups developing recommendations.

7. There National Institute for and Clinical Excellence: 3. is evidence of Health variation between actual The and guidelines approprimanual. London, National Institute for Health and Clinical Excelate care. lence; 2006. 8.

Oxman AD, Fretheim A, Schünemann S: Improving the use of

research evidence guideline development: 4. There are no existinginvalid guidelines available introduction. to use.

Health Res Policy Syst 2006. Cochrane Methodology Register [http://www.cochrane.org/ access_data/cmr/accessDB_cmr.asp] 5. There is an adequate amount of existing evidence available. 10. National Guideline Clearinghouse Agency for Healthcare Research and Quality 2006 [http://www.guidelines.gov]. 6. recommendations will be acceptable to the International potential 11. The Guidelines International Network (GIN) Guidelines users.Network 2006 [http://www.g-i-n.net]. 12. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, Marteau T: Consensus development methods, and 7. Implementation of the guideline guidelinedevelopment. is feasible, willHealth not exhaustheir use in clinical Technol 2:i-88. tivelyAssess use 1998, the communities' resources, and barriers to clinical 13. Hutchings A, Raine R: A systematic review of factors affecting change not so high that they cannotconsensus be overcome. theare judgments produced by formal development methods in health care (In press). J Health Serv Res Policy 2006. 14. Shekelle PG, Schriger DL: Evaluating the use of the appropriateWhile burden of disease is commonly used as a criterion ness method in the Agency for Health Care Policy and for priority setting, should Guideline be noted Development that the use of sumResearch ClinicalitPractice process. Serv Resof 1996, 31:453-468. maryHealth burden disease measures, such as disability 15. Jenkings KN, Barber N: What constitutes evidence in hospital adjusted life years (DALYs) has criticised for focusing new drug decision making? Socbeen Sci Med 2004, 58:1757-1766. 9.

Competing interests Findings

What shouldfor bethe used to establish priorities? Centre AF andcriteria ADO work Norwegian Knowledge The Institute of Medicine's on fortheUSHealth Services, an agency (IOM) funded Committee by the NorweMethods for Setting Guidelines Developgian government thatPriorities producesfor systematic reviews and ment its study of setting priorities for authors clinical are practice healthintechnology assessments. All three conguidelines in 1995 argued that the priority settributors topublished the Cochrane Collaboration. ADO and HJS ting process should open Working and defensible recare members of the be GRADE Group.[14] HJSThey is docuommended criteria: prevalence, burden of ments editorsix andgeneral chair of the documents development illness, cost of managing the problem, variability in pracand implementation committee for the American Thotice, of asenior guideline toof improve health outcomes, racicpotential Society and editor the American College of and potential of aAntithrombotic guideline to reduce costs. BecauseTherdata Chest Physicians' and Thrombolytic used to make these judgements is often lacking, they sugapy Guidelines. gested explicit opportunities for important unmeasured factors to be considered. They further suggested separate Authors' contributions AF prepared the first draft of this and ADO "tracks" for considering issues forreview. specificHJS populations, contributedortoconcerns. drafting and it. the rationales for conditions Theyrevising argued that designating special tracks should be defined clearly; i.e. why they warrant special consideration. They suggested Acknowledgements We acknowledge WHO and the Norwegian Centre thatgratefully updating of guidelines should also be Knowledge considered as a forthe Health Services for financial support, the other authors of these separate "track", taking account of issues such as the need reviews for their contributions, the other members of evidence. the ACHR SURE, for corrections and the availability of new

on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]).

and all of those who have provided us with feedback on earlier versions of this paper and the others in this series.

• the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence

Oortwijn identified 25 criteria used to prioritise health technology Referencesassessments and categorised these into four broad categories: burden H,ofADdisease, potential 1. Fretheim A, Schünemann O: Improving the effects, use of research evidence guideline development: 3. Group Compotential costs, and in uncertainty regarding application of position and [15]. Consultation Process. Health Res Policy Syst 2006. the technology 2. Schünemann H, Fretheim A, AD O: Improving the use of

research evidence in guideline development: 10. Integrating

Consumer Involvement. Health PolicyZealand Syst 2006. In aValues more and recent selective review for theRes New 3. Regulations for Expert Advisory Panels and Committees. In Guidelines Group, the following criteria were identified as Resolution WHA 35 10 WHO; 1982. indicating a topic suitable for guideline develop4. Wabitschthat R, Margolis CZ,isMalkin JE, Abu-Saad K, Waksman J: Evaluthe process of tailoring the global HIV/AIDS practice mentating [16]:

disseminating the results of biomedical researc h in our lifetime."

guidelines for use in developing countries. Int J Qual Health Care 1998, 10:147-154. 5. The Moynihan Oxman AD,important Lavis J, Paulsen E: A review organiza1. topic isR, clinically affecting large of numbers of tions that support the use of research evidence in developing people with substantial morbidity or mortality (the burden of guidelines, technology assessments, and health policy, for illness). the WHO Advisory Committee on Health Research. Oslo, Norwegian Knowledge Centre for the Health Services; 2006. 6. Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evi2. The topic is complex to initiate debate about theofrecdence-based clinicalenough practice: an international survey 18 clinical guideline programs. Int J Qual Health Care 2003, 15:31-45. ommendations.

Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation. BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 4 of 7 4 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 6. Determining which outcomes are important Holger J Schünemann*1, Andrew D Oxman2 and Atle Fretheim2 Address: 1INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 2Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway Email: Holger J Schünemann* - hjs@buffalo.edu; Andrew D Oxman - oxman@online.no; Atle Fretheim - atle.fretheim@nokc.no * Corresponding author

Published: 01 December 2006 Health Research Policy and Systems 2006, 4:18

Received: 07 April 2006 Accepted: 01 December 2006

doi:10.1186/1478-4505-4-18

This article is available from: http://www.health-policy-systems.com/content/4/1/18 © 2006 Schünemann et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Page 1 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:18

http://www.health-policy-systems.com/content/4/1/18 http://www.health-policy-systems.com/content/4/1/14

and• the If the Guidelines importance International of outcomesNetwork is likely to[8]) vary foracross existing cultures, fromofdiverse cultures should be and consulted 3. stakeholders There is evidence variation between actual approprisystematic and involved reviews in the and selection relevantofmethodological outcomes. research ate care. that address these questions. The answers to the questions How should the importance of outcomes be ranked? are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. • Outcomes shouldWHO be ranked relative importance, are separated into benefits and downsides. sideration of what and by other organisations doing, and logical arguments. There the is an adequate amount ofwhenever existing evidence • Information from research on values and preferences should5.inform ranking of outcomes possible.available. If the importance of outcomes is likely to (clinical vary across cultures, of outcomes should in specific settings. For•this review we searched PubMed using prac6. ranking The recommendations will be be done acceptable to the potential tice•guidelines or public health guidelines) and (priority users. If evidence is lacking for an important outcome, this should be acknowledged, rather than ignoring the outcome. setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhausogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical th International Conference on Priorities website of the 5 change are notwe so high that they be overcome. In this paper addressed thecannot following questions: Background in Health Care [11] and references that we had in our files The World Health Organization (WHO), like many other [12-15]. While of disease commonly used as aimportant criterion organisations around the world, has recognised the need • Whatburden methods should is WHO use to identify for priority setting, it should be noted that the use of sumto use more rigorous processes to ensure that health care outcomes? mary burden of disease measures, such as disability Findings recommendations are informed by the best available What criteria should beisused to establish priorities? adjusted life of years (DALYs)should has been criticised for and focusing research evidence. This the sixth of a series of 16 reviews • What type outcomes WHO consider how The US Institute of Medicine's (IOM) Committee on on disease rather than resource use and interventions, that have been prepared as background for advice from should cultural diversity be taken account of in the selecMethods Setting Committee Priorities for Developbecause of the assumptions about values inherent in such the WHOfor Advisory onGuidelines Health Research to tion of outcomes? ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such WHO on how to achieve this. guidelines published in 1995 argued that the priority setmeasures (see references [17] and [18]). • How should for theexample importance of outcomes be ranked? tingoutcome process should open and They recAn can be be defined as a defensible measure of[14] an intervenWhat processes should be used to agree priorities? ommended six general prevalence, burden or of tion's desirable (benefits,criteria: less burden and savings) Questions related to integrating valuesonand consumer Batista and Hodge in a review conducted 10 years illness, cost of managing the problem, variability in pracundesirable effects (including harms, greater burdens and involvement are specifically addressed in another paperago in found only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, cost). Those making health care recommendations always this series [2]. clinical practice guidelines [10]. They suggested the foland potential of athe guideline reduce costs. Because data should consider benefits,topotential harms, including lowing for priority used to make these judgements is often lacking, they sugthe potential for unanticipated adverse effects, burdens What framework WHO is doing now setting: gested explicit opportunities for important unmeasured The Guidelines for WHO Guidelines suggests the follow(e.g. having to take a pill), and costs, including opportu1. factors to be considered. They further suggestedoutcomes separate ing:Consult with end users and other stakeholders before nity costs. Identifying all known and plausible selecting topics. "tracks" for considering issues for specific populations, that are important to those affected and associated with conditions or concerns. They argued that the rationales for • "To identify the issues to be addressed, it is helpful to an intervention is a key step in formulating questions for 2. Consider feasibility during the consultation. designating special tracksUnfortunately, should be defined clearly; i.e. develop a logic and analytical frameworks guide (Woolf, guideline development. guideline develwhy they warrant special consideration. They suggested 1994)" [3]. (GWG 6C1 Process of developing guidelines) opers sometimes select outcomes based on what has been 3. Document the process used to select guideline topics. that updating of guidelines should considered as a assessed in studies rather than basedalso on be what is important separate "track", taking account of issues such as the need • "Spell out any tradeoffs between the cost of applying to those affected. The IOMrecommendations suggested the following proceduresbasis, [14]:and the for corrections and the availability of new evidence. possible on a population population health impacts" in the second stage of guide• thedevelopment. use of Delphi-like procedures obtaining expert line (section 5d Makingfor recommendations) judgments or topic rankings through correspondence

Since interventions affect several outcomes (e.g. some Oortwijn identified 25 criteria usedon to mortality, prioritise stroke, health hypertensive treatments have effects technology assessments anddevelopers categorisedneed these four diabetes, libido), guideline to into consider broad categories: burden of disease, potential effects, their relative importance. This is also true for public potential costs, and systems uncertainty regarding application of health and health interventions. For example, the technology [15]. media campaigns might cause anxiety as well as promot-

• "All evidence, including that on safety, should be clearly • theout useinofanquestions are(GWG specific, explicit and conlaid evidence that table" section 6C2). sistent with standard methods for questionnaire construction Despite these guidelines, a review of several WHO guide-

ing a desired health behaviour, and there are always assoIn a more for opportunity the New Zealand ciated costs.recent At the selective very least,review there are costs. Guidelines Group, the following criteria were identified as indicating that a topic is suitable Patients may assign different valuesfor to guideline outcomes developthan climent [16]: nicians and clinical experts involved in guideline develop-

lines (e.g., contraceptive use, hypertension, air pollution, • experimentation more formal procedures to arrive inpatient treatmentwith of malnourished children, treatment at group judgments of non-breastfed children) revealed that the process of outcome identification is usually not described. They also suggested there is a need to define more narrowly precisely topics forare guideline Whatand other organisations doing development. They argued that this would result in more The UK National Institute for Health andefficient ClinicalorganExceization of panels and their work, resolution ofidentisome lence (NICE) defines a very explicit process for the apparent more guidelines, and fication ofcontroversies, outcomes using theresponsive Population, Intervention, easier implementation. Comparison and Outcome (PICO) format for the devel-

ment [1]. In addition, surrogate outcomes such as 1. The topicmeasures is clinically important large numbers of laboratory that are partaffecting of the clinician's reperpeople with substantial morbidity or mortality (the burden of toire often do not correlate with patient important outillness). comes and guideline developers should scrutinize surrogate outcomes about how directly they relate to 2. The topic is complex enough to initiate debate about the recpatient important outcomes. ommendations.

Page 4 2 of 7 6 (page number not for citation purposes)

the use of 4:18 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/18

opment of questions [4]. The NICE handbook asks guideline panels to consider:

Quality (AHRQ, [10]) and the Guidelines International Network (GIN, [11]). These searches were supplemented with information obtained directly from guideline development organizations and our own files. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

â&#x20AC;˘ What outcome is really important for the patient? â&#x20AC;˘ Which outcomes should be considered: intermediate or short-term measures (e.g., mortality, morbidity and treatment complications, quality of life, cost, etc)?

Findings

We did not find a systematic review that addresses any of the key questions and we found very little relevant research evidence.

Similarly, the National Health and Medical Research Council of Australia bases its approach on the NICE handbook and defines the appropriateness of the outcomes by asking "Are they relevant to the patient?" [5,6].

What methods should WHO use to identify important outcomes?

SIGN underlines (section 5.1. of the SIGN handbook for guideline developers) that patients' perspectives should be included early in the guideline development process [7]. Therefore, SIGN prescribes to conduct a specific literature search designed to cover both quantitative and qualitative evidence about outcomes without limitations of study design, but this is not (yet) done consistently (Robin Habour, personal communication). In theory, the results of this search inform the development of key questions. SIGN uses the PICO format for question development.

Few guideline developers have included descriptions of methods for the identification of important outcomes. SIGN uses an approach that begins with conducting a search for evidence using the patient perspective before finalizing the formation of the question. Most other guideline developers have not described formal processes of identifying important outcomes when formulating guideline questions. To be reproducible and understandable, the methods of outcome identification should be transparent and explicit. NICE suggests facilitating the process of formulating questions, "it may be helpful to construct a diagram listing outcomes and other key criteria the [guideline] group has considered important. Once the question has been framed, key words can be identified as potential search terms" [12]. NICE involves patient organistions in developing guideline scopes and routinely includes at least two patient or caregiver members who provide a patient perspective on all guideline development activities including the formulation of clinical questions and defining of relevant outcomes [13]. Owens and Nease suggest the use of influence diagrams to identify important outcomes and focus guideline questions [14]. They argue this helps to delineate an explicit link between interventions and outcomes, shifts the focus from broad questions to more sharply delineated questions to be addressed, and highlight the importance of a clear, unambiguous statement of whose benefit, downsides and costs are under consideration. Thus, this limited evidence suggests that a consultation process should start with identification of all relevant outcomes associated with an intervention.

The United States Preventive Services Taskforce (USPSTF) [8] describes that value judgments are involved in using the information in an outcomes table to rate either benefits or harms. USPSTF uses a 4-point scale to rate importance. Value judgments are also needed to weigh benefits against harms and to arrive at a rating of net benefit. The USPSTF does not use formal processes for identifying outcomes as part of the question formulation. Specialty societies do not consistently acknowledge a formal process for question development and the processes are often not transparent.

Methods

The methods used to prepare this review are described in the introduction to this series [9]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct a full systematic review. We reviewed existing guidelines for guidelines to identify processes for outcome identification and ranking. We also searched PubMed using (guideline OR policy making) and (identification) and (outcomes) as search terms (MESH headings/keywords) for systematic reviews and studies of methods for identifying outcomes for guideline development (69 citations). We also searched the Cochrane Methodology Register and Database of Methodology Reviews using the keywords "outcome" and "identification". We also searched databases maintained by the Agency for Healthcare Research and

Bravata and colleagues. conducted an overview of reviews to identify innovative methods for question formulation related to challenging topics in health care (organization, delivery and financing of health care) [15]. They found that the use of decision analytical frameworks for defining a question and systematic methods such as influence diagrams influenced how questions were formulated. Fur-

Page 3 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:18

http://www.health-policy-systems.com/content/4/1/18 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines thermore, systematic International methods appeared Networkto[8]) have foran existing effect on search strategies systematic reviews and to identify relevantunderlying methodological evidence. research that address these questions. The answers to the questions There are ourisconclusions little empirical based evidence on the available to inform evidence, decisions conabout whatofmethods sideration what WHO to useand to other identify organisations important outare comes. and doing, Given logical the paucity arguments. of data on patients' and the public's values WHO should consider using other evidence suchthis For as review systematic we searched summaries PubMed and original using (clinical research pracon people's tice guidelines perspectives or public andhealth experiences guidelines) ("views" and (priority studies) alongside setting or setting trials ofpriorities) effectiveness. and related Summarizing articles for views selected studies in a systematic references [9,10]. We waysearched could lead thetoCochrane a greater Methodolbreadth of perspectives ogy Register and using a deeper priorityunderstanding or priorities. We of public reviewed health the th International on Priorities issues from website of the the5point of view ofConference those targeted by interin Health Care [11]etand that wemethodology had in our files ventions. Harden al. references suggest that this is [12-15]. likely to create greater opportunities for people's own perspectives and experiences to inform policies to promote their health [16]. Findings What criteria should be used to establish priorities? The US Institute of Medicine's (IOM)methods Committee on Compared to the limited evidence about to idenMethods for Setting Priorities for Guidelines tify important outcomes in guidelines, there is aDeveloplarge litment its study of setting for clinical practice eratureinthat documents that priorities the importance of outcomes guidelines published in 1995 argued and that between the priority setcan vary within and across cultures, health ting shouldand be open and[1,17]. defensible [14] Theyboth reccare process professionals patients In addition, ommended six general prevalence, burden of clinical and public healthcriteria: interventions can have effects illness, of managing the problem, variability in practhat arecost important to consumers, but are not considered tice, potential a guideline improve health outcomes, important by of researchers or to health professionals in part and potential a guideline to decision reduce costs. Because because valuesofdiffer between makers. This data sugused make judgements often lacking, gests to two key these elements of any is approach that isthey usedsugto gested opportunities for important unmeasured identifyexplicit important outcomes. First, all relevant stakeholdfactors to be considered. further suggested ers (including consumers)They should be consulted at separate an early "tracks" for process. considering issues the for formulation specific populations, stage in the Secondly, of quesconditions or search concerns. argued that the tions and the for They evidence should thenrationales consider for all designating special tracks should be defined clearly; i.e. relevant outcomes. why they warrant special consideration. They suggested that updating of outcomes guidelines should also be considered as a What type of should WHO consider separate "track", taking account of issuesbe such as the need and how should cultural diversity taken for corrections availability new evidence. account of inand thethe selection of of outcomes? The AGREE Collaboration states that the guidelines development process "involves takingused intotoaccount the health beneOortwijn identified 25 criteria prioritise fits, harms and costs of the well as technology assessments andrecommendations, categorised these as into four the practical issuesburden attachedoftodisease, them" [18]. The AGREE broad categories: potential effects, instrument suggests guidelines "should health potential costs, and uncertainty regardingconsider application of benefits, side effects, the technology [15]. and risks of the recommendations. For example, a guideline on the management of breast cancer may recent includeselective a discussion onfor thethe overall on In a more review Neweffects Zealand various finalGroup, outcomes. These may include: survival, qualGuidelines the following criteria were identified as ity of life, adverse effects, symptom management or a indicating that a topic is and suitable for guideline developdiscussion ment [16]: comparing one treatment option to another. There should be evidence that these issues have been addressed." suggests that the potential cost implica1. The topic It is also clinically important affecting large numbers of tions ofwith applying the recommendations should been people substantial morbidity or mortality (thehave burden of considered. In general, desirable and undesirable effects illness). should be considered in all guidelines. Undesirable effects include harms (including thetopossibility of unanticipated 2. The topic is complex enough initiate debate about the recommendations.

adverse burdens (e.g. having go toand theappropridoctor) 3. There effects), is evidence of variation betweento actual andcare. costs (including opportunity costs). ate The GRADE Working suggestsavailable that explicit 4. There are no existing Group valid guidelines to use.judgements should be made about which outcomes are critical, which important but not critical,evidence and which ones 5. Thereones is anare adequate amount of existing available. are unimportant and can be ignored. The group emphasizes that all important outcomes should betoconsidered in 6. The recommendations will be acceptable the potential making a recommendation, but only critical ones should users. be considered when making judgements about the overall quality of the evidence underlying a recommendation 7. Implementation of the guideline is feasible, will not exhaus[19]. use They it is important to toconsider tively therecommend communities'that resources, and barriers clinical costs (resource before making a recommendachange are not soutilisation) high that they cannot be overcome. tion. They also suggest that studies using surrogate outcomes burden generally evidence than using While ofprovide diseaseweaker is commonly used asthose a criterion outcomes are it important, these should be for prioritythat setting, should be and noted thatonly the use of sumincluded when of evidence importantsuch outcomes is lackmary burden diseaseformeasures, as disability ing. Thus,life important outcomes (e.g. criticised mortality,for morbidity, adjusted years (DALYs) has been focusing quality of life) should be resource preferred use overand surrogate, indirect on disease rather than interventions, outcomes cholesterol levels, function) may because of(e.g. the assumptions aboutlung values inherentthat in such or may notand correlate with patient important outcomes. measures, because of the technical limitations of such measures (see for example references [17] and [18]). Because the importance of different outcomes can vary What processes be used toattached agree ontopriorities? dramatically andshould the importance different outBatista may and vary Hodge inculture a review conducted years ago comes from to culture, it is 10 important to found only three articles to priority setting for take cultural diversity intopertinent account when deciding which clinical practice guidelines [10]. They suggested the and foloutcomes are important [20-23]. Prenatal screening lowing framework priority setting: genetic counselingforare examples of interventions for which the importance of an outcome (abortion) varies 1. Consult with end and usersacross and other stakeholders before between individuals cultures, because of reliselecting topics. gious beliefs or values [24,25]. End of life decisions are influenced by the roles of decision makers (clinician ver2. Consider feasibility during the consultation. sus patient and family) and cultural differences [20,22]. The choice of using aspirin is related to the values and 3. Document process used to select guideline topics. preferences ofthe diabetic patients and patients place very different values on preventing strokes than their health care The IOM suggested the following procedures providers [1,26]. Cultural differences can be[14]: taken into account through the involvement of stakeholders from â&#x20AC;˘ the usecultures, of Delphi-like procedures for judgments obtaining expert different and may require that about judgmentsbetween or topicthe rankings through correspondence trade-offs benefits and downsides of an intervention are specific for different cultures [27-29]. Values â&#x20AC;˘ the use of questions thatbeareelicited specific,and explicit and conof stakeholders should transparently sistent withinstandard methods for questionnaire construcdescribed recommendations. We offer strategies in tion another article of this series [2]. â&#x20AC;˘ experimentation withshould more formal arrive Ethical considerations also beprocedures taken into to account at group judgments when selecting outcomes. For example, with directly observed therapy for tuberculosis, individual rights to They also suggested there ismay a need more narrefuse therapy (autonomy) havetotodefine be sacrificed for rowly and of precisely topics for guideline development. the benefit society [30]. Explicit identification of ethical They argued thatand thisexplicit would result in moreabout efficient organconsequences, judgments trade-offs ization panels and their work,that resolution of some such as of these, can help to ensure appropriate judgapparent responsive and ments arecontroversies, made, help tomore resolve or clarifyguidelines, disagreements, easier implementation. and facilitate local adaptation of guidelines.

Page 4 of 7 6 (page number not for citation purposes)

the use of 4:18 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/18

How should the importance of outcomes be ranked?

Authors' contributions

HJS prepared the first draft of this introduction. ADO and AF contributed to drafting and revising it.

Judgments about the balance between the benefits and downsides of an intervention require judgments about the relative importance of the different outcomes, either explicitly or implicitly. Ranking outcomes by their relative importance, separated into benefits and downsides in an evidence profile [7,12] can help to focus attention on those outcomes that are considered most important, and help to resolve or clarify disagreements. Research on values and preferences should guide the ranking of outcomes, whenever possible. Guideline panels may want to search for research on the values associated with specific outcomes of interest to inform judgments about their relative importance.

Acknowledgements We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper. The authors thank Mr. John Fox for administrative assistance with the literature review and organizing the literature database.

References 1.

If the importance of outcomes varies across cultures, ranking should be done by people in a specific setting, who can pay due consideration to local values and preferences. If evidence is lacking for an important outcome, this should be acknowledged, rather than ignoring the outcome.

Discussion

There is very limited evidence to inform decisions about how to select and rank outcomes. However, we recommend the use of systematic and transparent methods involving key stakeholders, including consumers and people from different cultures, to help ensure that all important outcomes are considered and facilitate local adaptation of guidelines. Limitations of our work include the possibility that we have missed relevant studies.

5. 6. 7.

Further work

Although it is possible that there is relevant empirical research of which we are not aware, a complete systematic review of the questions addressed in this paper is unlikely to change the conclusion that there is very little research evidence in this area. Evaluations comparing different methods of identifying, selecting and ranking outcomes are needed.

10.

Competing interests

14.

11. 12. 13.

15.

16.

17. 18.

Devereaux PJ, Anderson DR, Gardner MJ, Putnam W, Flowerdew GJ, Brownell BF, Nagpal S, Cox JL: Differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation: observational study. Bmj 2001, 323(7323):1218-1222. Schünemann HJ Fretheim A, Oxman, AD: Improving the Use of Research Evidence in Guideline Development: 10. Integrating values and consumer involvement. Health Res Policy Syst 2006. Woolf SH, DiGuiseppi CG, Atkins D, Kamerow DB: Developing evidence-based clinical practice guidelines: lessons learned by the US Preventive Services Task Force. Annu Rev Public Health 1996, 17:511-538. McKibbon A, Hunt D, Richardson SW, al. : Finding the evidence. In Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice Edited by: Guyatt G RD. Chicago , American Medical Association Press; 2002:16. National Health and Medical Research C: Handbook series on preparing clinical practice guidelines. February 2000. Australia. 2000. National Health and Medical Research C: NHMRC additional levels of evidence and grades for recommendations for developers of guidelines PILOT PROGRAM 2005. . Scottish Intercollegiate Guideline Network. SIGN 50: A guideline developers' handbook. Section 5. :(accessed on March 17, 2006) [http://www.sign.ac.uk/guidelines/fulltext/50/ section5.html]. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, Atkins D: Current methods of the U.S. Preventive Services Task Force: A review of the process. American Journal of Preventive Medicine 2001, 20(3, Supplement 1):21-35. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: Introduction. Health Res Policy Syst 2006. Agency for Healthcare Research and Quality [http:// www.guidelines.gov] Guidelines International Network [http://www.g-i-n.net] National Instutites for Health and Clinical Excellence [http:/ /www.nice.org.uk/pdf/GDM_Allchapters_0305.pdf] Kelson MC: The NICE Patient Involvement Unit. EvidenceBased Healthcare & Public Health 2005, 9:304–307. Owens DK, Nease RF Jr.: Development of outcome-based practice guidelines: a method for structuring problems and synthesizing evidence. Jt Comm J Qual Improv 1993, 19(7):248-263. Bravata DM, McDonald KM, Shojania KG, Sundaram V, Owens DK: Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care. Ann Intern Med 2005, 142(12_Part_2):1056-1065. Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health research. J Epidemiol Community Health 2004, 58(9):794-800. Watts T, Merrell J, Murphy F, Williams A: Breast health information needs of women from minority ethnic groups. J Adv Nurs 2004, 47(5):526-535. AGREE Collaboration (www.agreetrust.org). Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12(1):18-23.

Page 5 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:18

http://www.health-policy-systems.com/content/4/1/18 http://www.health-policy-systems.com/content/4/1/14

19. D, Best D, Briss PA, Eccles M, Falck-Ytter Flottorp S, Guyatt and Atkins the Guidelines International NetworkY,[8]) for existing GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Libsystematic reviews and relevant methodological research erati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, that Oxman address these questions. The HJ, answers to the questions AD, Phillips B, Schunemann Edejer TT, Varonen H, Vist GE, conclusions Williams JW Jr.,based Zaza S:on Grading quality of evidenceconand are our the available evidence, strength of recommendations. Bmj 2004, 328(7454):1490. sideration other organisations are 20. Klessig J: of Thewhat effect WHO of valuesand and culture on life-support decisions. J Medarguments. 1992, 157(3):316-322. doing, andWest logical

3. There is evidence of variation between actual and appropriate care. 4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available.

Blackhall LJ, Murphy ST, Frank G, Michel V, Azen S: Ethnicity and attitudes toward patient autonomy. Jama 1995, For this review we searched PubMed using (clinical prac274(10):820-825. 22. guidelines Ruhnke GW, Wilson SR, Akamatsu Kinoue T, Takashima Y, Goldtice or public health T, guidelines) and (priority stein MK, Koenig BA, Hornberger JC, Raffin TA: Ethical decision setting or setting priorities) and related articles for selected making and patient autonomy: a comparison of physicians references [9,10].in We searched Cochrane and patients Japan and thethe United States. MethodolChest 2000, ogy 118(4):1172-1182. Register using priority or priorities. We reviewed the 23. Weber E Hsee th C: Culture and Individual Judgment and DeciInternational onReview Priorities website the 5 Applied sion of Making. Psychology Conference An International 2000, 49:32-61. in Health Care [11] and references that we had in our files 24. Hunt LM, de Voogd KB, Castaneda H: The routine and the trau[12-15]. matic in prenatal genetic diagnosis: does clinical information inform patient decision-making? Patient Educ Couns 2005, 56(3):302-312. Findings 25. Fischer RL, Schaeffer K, Hunter RL: Attitudes of obstetrics and What criteria should be toward used toabortion establish priorities? a Philagynecology residents participation: area survey. Contraception 2005, 72(3):200-205. The delphia US Institute of Medicine's (IOM) Committee on 26. Montorifor VM,Setting Bryant SC, O'Connor for AM, Guidelines Jorgensen NW,DevelopWalsh EE, Methods Priorities Smith SA: Decisional attributes of patients with diabetes: the mentaspirin in itschoice. study Diabetes of setting for clinical practice Carepriorities 2003, 26(10):2804-2809. 27. Duffy SA,published Jackson FC, Schim SM, Ronis DL, Fowler KE: priority Racial/ethnic guidelines in 1995 argued that the setpreferences, sex preferences, and perceived discrimination ting related processtoshould be open defensible [14] They recend-of-life care. Jand Am Geriatr Soc 2006, 54(1):150-157. ommended general criteria: prevalence,inburden of 28. Suurmond J,six Seeleman C: Shared decision-making an intercultural context. Barriers inthe theproblem, interactionvariability between physicians illness, cost of managing in pracand immigrant patients. Patient Educ Couns 2006, 60(2):253-259. tice, potentialM,ofCuttini a guideline to improve outcomes, 29. Rebagliato M, Broggin L, Berbik health I, de Vonderweid U, G, Kaminski M, Kollee to LA,reduce Kucinskas A, Lenoir S, Levin A, and Hansen potential of a guideline costs. Because data J, Reid M, Saracci R: Neonatal end-of-life decision makusedPersson to make these judgements is often lacking, they suging: Physicians' attitudes and relationship with self-reported gested explicit inopportunities for countries. important unmeasured practices 10 European Jama 2000, 284(19):2451-2459. factors to be considered. They further suggested separate 30. Verma G, Upshur RE, Rea E, Benatar SR: Critical reflections on "tracks" for considering issues for specific populations, evidence, ethics and effectiveness in the management of conditions or concerns. Theyand argued the rationales for tuberculosis: public health globalthat perspectives. BMC Med Ethics 2004,special 5:E2. tracks should be defined clearly; i.e. designating 21.

6. The recommendations will be acceptable to the potential users. 7. Implementation of the guideline is feasible, will not exhaustively use the communities' resources, and barriers to clinical change are not so high that they cannot be overcome. While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]). What processes should be used to agree on priorities? Batista and Hodge in a review conducted 10 years ago found only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the following framework for priority setting:

1. Consult with end users and other stakeholders before selecting topics. 2. Consider feasibility during the consultation.

why they warrant special consideration. They suggested that updating of guidelines should also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.

In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]:

disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation.

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 4 6 of 7 6 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 7. Deciding what evidence to include Andrew D Oxman*1, Holger J Schünemann2 and Atle Fretheim3 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway, 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 3Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway Email: Andrew D Oxman* - oxman@online.no; Holger J Schünemann - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no * Corresponding author

Published: 01 December 2006 Health Research Policy and Systems 2006, 4:19

Received: 07 April 2006 Accepted: 01 December 2006

doi:10.1186/1478-4505-4-19

This article is available from: http://www.health-policy-systems.com/content/4/1/19 © 2006 Oxman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract Background: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the seventh of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. Objectives: We reviewed the literature on what constitutes "evidence" in guidelines and recommendations. Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key question and answers: We found several systematic reviews that compared the findings of observational studies with randomised trials, a systematic review of methods for evaluating bias in non-randomised trials and several descriptive studies of methods used in systematic reviews of population interventions and harmful effects. What types of evidence should be used to address different types of questions? • The most important type of evidence for informing global recommendations is evidence of the effects of the options (interventions or actions) that are considered in a recommendation. This evidence is essential, but not sufficient for making recommendations about what to do. Other types of required evidence are largely context specific. • The study designs to be included in a review should be dictated by the interventions and outcomes being considered. A decision about how broad a range of study designs to consider should be made in relationship to the characteristics of the interventions being considered, what evidence is available, and the time and resources available. • There is uncertainty regarding what study designs to include for some specific types of questions, particularly for questions regarding population interventions, harmful effects and interventions where there is only limited human evidence. • Decisions about the range of study designs to include should be made explicitly. • Great caution should be taken to avoid confusing a lack of evidence with evidence of no effect, and to acknowledge uncertainty. • Expert opinion is not a type of study design and should not be used as evidence. The evidence (experience or observations) that is the basis of expert opinions should be identified and appraised in a systematic and transparent way.

Page 1 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:19

http://www.health-policy-systems.com/content/4/1/19 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing Background The World reviews Health Organization like many other systematic and relevant (WHO), methodological research organisations around the world, has recognised the need that address these questions. The answers to the questions to use rigorousbased processes toavailable ensure that health concare are ourmore conclusions on the evidence, recommendations informed by the best available sideration of whatare WHO and other organisations are researchand evidence. This is the seventh of a series of 16 doing, logical arguments. reviews that have been prepared as background for advice fromthis thereview WHO we Advisory Committee on Health Research For searched PubMed using (clinical practo WHO on how achieve this.guidelines) and (priority tice guidelines or to public health setting or setting priorities) and related articles for selected Recommendations about healththe careCochrane and aboutMethodolintervenreferences [9,10]. We searched tionsRegister or actions that affect or health, such as or enviogy using priority priorities. Wesocial reviewed the Conference Priorities ronmental can be informed by on a wide range website of interventions, the 5th International in Health Care [11] and references trials, that we had in our files of evidence including randomised non-randomised [12-15]. comparative studies, descriptive studies, qualitative research, animal studies and laboratory studies. Discussions of evidence-informed policy and practice can generFindings What criteriaregarding should bewhat used constitutes to establish 'evidence' priorities? [1]. A ate debates The US Institute of Medicine's (IOM) on common understanding of evidence is thatCommittee "evidence conMethods Setting Prioritiesintended for Guidelines cerns factsfor (actual or asserted) for use inDevelopsupport ment in its study[1]. of setting clinical practice of a conclusion" A fact, priorities in turn, isfor something known guidelines published in 1995 argued that the implication priority setby experience or observation. An important ting process should beof open and defensible [14] They recof this understanding evidence is that evidence is used ommended general it criteria: prevalence, burden of to support a six conclusion; is not the same as the concluillness, cost of managing variability in pracsion. Evidence alone doesthe notproblem, make decisions. tice, potential of a guideline to improve health outcomes, and of a guideline to reduceiscosts. Because data This potential understanding of what evidence has several impliused to make these judgements they sugcations. Firstly, expert opinion is often more lacking, than evidence. It gested explicit important combines facts,opportunities interpretationfor of those facts, unmeasured and conclufactors to be is considered. They further sions. There evidence behind expertsuggested opinions.separate Expert "tracks" considering issues for specific populations, opinion for should be used appropriately by identifying the conditions or concerns. They argued that rationales for facts (experience or observations) that arethe the basis of the designating tracksthe should definedtheclearly; i.e. opinions andspecial appraising extentbe to which facts supwhy theyconclusions warrant special port the [2]. consideration. They suggested that updating of guidelines should also be considered as a separate account of issuesconvincing. such as the How need Secondly,"track", not alltaking evidence is equally for corrections and the new evidence. convincing evidence is availability (for effects)ofshould be based on criteria such as: What sort of observations? How well were Oortwijn 25 criteria prioritise they done?identified How consistent are used they?toHow directlyhealth reletechnology assessments categorised these isinto four vant are they? How manyand are there? How strong an assobroad ciation?categories: burden of disease, potential effects, potential costs, and uncertainty regarding application of the technology [15]. about how much confidence to place Thirdly, judgements in different types of evidence (the 'quality' of the eviIn a more recenteither selective revieworfor the NewIt Zealand dence) are made implicitly explicitly. is better Guidelines Group, the following criteria were as to make these judgements systematically and identified explicitly to indicating that a topic is suitable guideline develophelp protect against errors, resolvefor disagreements, faciliment [16]: appraisal, and communicate information. tate critical This, in turn, requires explicit decisions about what types 1. topic to is clinically affecting large numbers of of The evidence considerimportant at all. people with substantial morbidity or mortality (the burden of illness). Fourthly, all evidence is context sensitive, since observations are made in a specific context. A judgement always 2. The topic complex enough to initiate debate about the that recneeds to beismade about their applicability beyond ommendations. context. It is best to make judgements about applicability

3. There is evidence of variationfor between actual and approprisystematically and explicitly, the same reasons that it is ate bestcare. to make judgements about the quality of the evidence systematically and explicitly. 4. There are no existing valid guidelines available to use. Fifthly, global evidence (i.e. the best evidence from 5. There the is anworld) adequate existing evidence available. around is amount the bestof starting point for judgements about effects, likely modifying factors, and (some6. Theatrecommendations will be acceptable to the potential times least) resource utilisation. This argument is based users. on the understanding that all evidence is context sensitive to some extent and, therefore, indirect to some extent. 7. Implementation of the feasible, will not Decisions based on a guideline subset ofis observations areexhausmore tively communities' barriers to clinical proneusetothe random errorsresources, [3], andandjudgements about change so ahigh that they on cannot be overcome. whetheraretonot base conclusion a subset of observations are better informed if the overall observations (all of the While burden disease isare commonly used as a criterion relevant globalof evidence) known [4]. for priority setting, it should be noted that the use of summary of disease measures, such as disability Sixthly,burden local evidence (from the specific setting in which adjusted years (DALYs) been criticised for for focusing decisionslife and actions will has be taken) is needed most on disease rather about than resource use including: and interventions, other judgements what to do, the presbecause of the assumptions values inherent such ence of modifying factors inabout specific settings, needin(prevmeasures, and because of the technical limitations of such alence, baseline risk or status), values, costs and the measures (see for example references [17] and [18]). availability of resources. What processes beboth usedglobal to agree on priorities? Recognising the should need for evidence (of effects) Batista and Hodge in review conducted 10 years ago and local evidence, it isa important to be cautious about found only global three articles pertinent toNonetheless, priority setting for developing recommendations. global clinical practice guidelines [10].when Theydifferent suggested theconfolrecommendations are valuable local lowing framework fortopriority ditions are not likely lead tosetting: different decisions. When different conditions are likely to lead to different deci1. Consult with end usersfor and other stakeholders before sions, global frameworks decisions are still important. selecting These cantopics. reduce unnecessary duplication of efforts. They are particularly important to support low and middle2. Consider feasibility consultation. income countries, withduring limitedthe resources to systematically develop guidelines, to make context specific decisions by 3. Document process used to select guideline topics. providing the the global evidence, a framework for decisions, and practical advice for incorporating local evidence. The IOM suggested the following procedures [14]: WHO's focus is on global recommendations and support• theitsuse of Delphi-like for obtaining expert ing member states to procedures make well-informed decisions. judgments orquestion topic rankings through correspondence The primary that needs to be addressed in this context is: • the use of questions that are specific, explicit and constandard questionnaire •sistent Whatwith types of studymethods designs for should be used toconstrucaddress tion different types of questions about the effects of the different options that are considered when making a recom• experimentation with more formal procedures to arrive mendation? at group judgments We therefore have focused this review on questions about They also suggested there a need define more nareffects, recognising that is there are toparallel questions rowly andwhat precisely guideline development. regarding types topics of studyfordesigns should be used to They argued this would in more efficient organaddress otherthat questions. In result addressing this question we ization of panels andvalidity their work, resolution some have focused on the of different study of designs, apparent controversies, responsive guidelines, assuming that questionsmore about the applicability of and the easier implementation. results of studies to the specific questions of interest will be similar across different study designs. However, it is

Page 4 2 of 7 (page number not for citation purposes)

the use of 4:19 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/19

important to recognise that decisions about what study designs to include may also be influenced by the extent to which relevant studies are available that have used study designs that are most likely to provide valid results. That is, there may sometimes be a trade-off between including studies that are more likely to be valid and ones that are more likely to be directly relevant.

motion and Public Health Task Force, which recommends that:"The study designs to be included in a public health review should be dictated by the interventions being reviewed (methodological appropriateness), and not vice versa" [9]. There is also general, although not unanimous, agreement that the inclusion criteria for a systematic review should specify the study designs that are acceptable for a specific question [10]. However, there are important differences in both guidance and practice with respect to "how low" reviewers should go in deciding what evidence to include [11]. This question is particularly relevant for questions about the effects of population interventions (public health, health promotion, health systems and social interventions) and for evidence of harmful effects [10-20].

What WHO is doing now

The Guidelines for WHO Guidelines (GWG) state: "It is recommended that [a] systematic review be undertaken (http:/ /hiru.mcmaster.ca/cochrane/cochrane/hbook.htm) After the studies have been identified and critically appraised, and the evidence synthesised, evidence should be graded. All evidence, including that on safety, should be clearly laid out in an evidence table. Meta-analysis should be done when the data permit. The final results should be presented in a balance sheet" [5]. The GWG do not address the choice of study designs for different types of questions. In practice it is difficult to know what study designs are considered relevant for different types of WHO recommendations since few WHO guidelines have adhered to the GWG, few have included a systematic review, and many do not include references [6,7].

The Cochrane Handbook for Systematic Reviews of Interventions takes a relatively cautious approach: "The more restrictive authors are in matching questions to particular aspects of design, the less likely they are to find data specific to the restricted question. However, reviewing studies that are unlikely to provide reliable data with which to answer the question is a poor use of time and can result in misleading conclusions." [21] Because Cochrane reviews address questions about the effects of health care, they focus primarily on randomised trials. The Handbook suggests being cautious of including non-randomised studies because of the risk of biased results; the additional work required to identify and appraise non-randomised studies and keep a review up-to-date; and the risk of publication bias. It concludes: "While attention to the risk of bias should guide decisions about what types of study designs to include in a review, individual authors and Collaborative Review Groups must decide what types of studies are best suited to specific questions."

What other organisations are doing

The U.S. Preventive Services Task Force has the following approach to determining what evidence is admissible: The topic team determines the bibliographic databases to be searched and the specific inclusion and exclusion criteria (i.e., admissible evidence) for the literature on each key question. Such criteria typically include study design, population studied, year of study, outcomes assessed, and length of follow-up. Topic teams specify criteria on a topic-by-topic basis rather than adhering to generic criteria. If high-quality evidence is available, the topic teams may exclude lower-quality studies. Conversely, if higher-quality evidence is lacking, the teams may examine lower-quality evidence.

Within the Cochrane Collaboration, several groups have recommended inclusion of a broader range of study designs for health systems and public health interventions and for assessing harmful effects of clinical interventions. The Cochrane Effective Practice and Organisation of Care Group (EPOC) argues that: While cluster randomised trials are the most robust design for quality improvement strategies, some strategies may not be amenable to randomisation – for example, mass media campaigns. Under these circumstances, reviewers may choose to include other designs including quasiexperimental designs. If a review includes quasi-experimental studies – for example, interrupted time series designs for evaluating mass media campaigns, the reviewers need to recognise the weaknesses of such designs and be cautious of over-interpreting the results of such studies. Within EPOC, reviewers can include randomised trials, controlled before and after studies, and interrupted time series [17].

If a search finds a well-performed systematic review that directly addresses the literature on a key question through a given date, the topic team may use this review to capture the literature for those dates. The team can then restrict its own search to dates not covered by the existing systematic review. The topic team documents these strategies for sharpening focus – the analytic framework, key questions, and criteria for admissible evidence – in an initial work plan. This work plan is presented to the Task Force at its first meeting after the topic has been assigned, allowing the Task Force the opportunity to modify the direction and scope of the review, as needed [8].

The Guidelines for Systematic Reviews of Health Promotion and Public Health Interventions Taskforce suggests including a still broader range of study designs: "A wide

This approach is consistent with other guidance for systematic reviews, such as those of the Cochrane Health Pro-

Page 3 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:19

http://www.health-policy-systems.com/content/4/1/19 http://www.health-policy-systems.com/content/4/1/14

and theofGuidelines variety study designs International may be used Network in the evaluation [8]) for of existing public eases, such as SARSoforvariation avian influenza (H5N1), which 3. There is evidence between actual andfor approprihealth activities, systematic reviews ranging and relevant from randomized methodological controlled research trials casecare. reports may be the only human studies that are availate (RCTs) that address to case these studies, questions. with no The single answers method to the being questions able to able, may be based on a combination of indirect human answer are ourall conclusions relevant questions based on about thethe available effectiveness evidence, of all conpubevidence (from the treatment of other similartodiseases), 4. There are no existing valid guidelines available use. lic health interventions." sideration of what WHO [9] and other organisations are case reports, animal studies and in vitro studies. In gendoing, and logical arguments. eral, the issame principles that apply to human studies can 5. There an adequate amount of existing evidence available. The Cochrane Adverse Effects Subgroup identifies three be applied to animal and in vitro studies [23]. possible For this review approaches we searched for incorporating PubMed using adverse (clinical effectpracdata 6. The recommendations will be acceptable to the potential in a guidelines tice review and or summarises public health the advantages guidelines)and anddisadvan(priority The Guide to Community Preventive Services uses data users. tages oforeach setting setting of these priorities) approaches and related as summarised articles for in selected Table from comparative studies – those that compare outcomes 1[18,19]. [9,10]. We searched the Cochrane Methodolreferences among a group exposed to the isintervention out7. Implementation of the guideline feasible, willversus not exhausogy Register using priority or priorities. We reviewed the comesuse inthe a concurrent orresources, historicaland group thattowas not tively communities' barriers clinical on Priorities The U.K.of NHS for ReviewsConference and Dissemination prowebsite the Centre 5th International exposedare ornot wassoless to answer questions about change highexposed that they–cannot be overcome. in Health Care [11] guidance: and references we had in ourspecifiles vides the following "The that inclusion criterion whether interventions are effective [24]. All comparative [12-15]. fying the type of study design stems from the desire to base studiesburden are included in isitscommonly reviews, assessed their While of disease used as a for criterion reviews on the highest quality evidence. There are several areas design suitability threats to internal for priority setting, and it should be noted that theand use external of sumof health care which have not been evaluated with methodologvalidity, and assessed for potential effects study design mary burden of disease measures, suchof as disability Findings What criteria shouldInbethis used to establish priorities? ically sound studies. situation, studies of methodologiand execution on results. adjusted life years (DALYs) has been criticised for focusing The Institute Medicine's (IOM) Committee cally US lower quality of may have to be included. Here it on is on disease rather than resource use and interventions, Methods fornote Setting Priorities forfor Guidelines Developimportant to that the preference one or another study The Campbell Collaboration does not inherent provide specific because of the assumptions about values in such ment its study of on setting priorities for clinical practice design in should depend the nature of questions raised in the guidance on what studyofdesigns shouldlimitations be used to address measures, and because the technical of such guidelines published in 1995 argued that the priority review. Inevitably the decisions regarding inclusion basedseton different types of questions related to[17] the effects of intermeasures (see for example references and [18]). ting open and defensible [14]ofThey recstudyprocess design should will alsobedepend on the availability suitable ventions in the social, behavioral and educational arenas What processes should be used to agree on priorities? ommended sixthegeneral criteria: study designs in literature." [22] prevalence, burden of [25]. Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracfound only three articles pertinent to priority setting for tice, potential of a of guideline to improve health outcomes, We are not aware any specific guidance for what study Methods The methods used to prepare thisThey review are described in clinical practice guidelines [10]. suggested the foland potential of a for guideline to reduce costs. Becausesome data designs to include non-human studies, although the introduction this series setting: [26]. Briefly, the key queslowing frameworktofor priority used to make theserely judgements is and often they sugrecommendations on animal inlacking, vitro studies. For tions addressed in this paper were vetted amongst the gested explicit opportunities for important unmeasured example, treatment recommendations for emerging dis1. Consult with end users and other stakeholders before factors to be considered. They further suggested separate selecting topics. "tracks" for considering issues for specific populations, Table 1: Pros and cons of different approaches for incorporating adverse effect data in a systematic review* conditions or concerns. They argued that the rationales for Method Look in thetracks trials/studies Look inclearly; all retrieved of that forthe studies that specifically 2. Consider feasibility Look during consultation. designating special should be defined i.e. trials/studies the systematic intervention, even in those excluded from the evaluate adverse effects of the why they included warrant inspecial consideration. They suggested review of benefit. analysis of benefit intervention 3. Document the process used to select guideline topics. that updating of guidelines should also be considered as a separate taking of issuesStudies such rejected as the need Protocol "track", Should usually beaccount the minimum from analysis of benefit (e.g. because Design separate strategy to identify studies The IOM suggested following procedures [14]: those for corrections and the availability of new evidence. report adverse effects, including recommendation beneficial outcomes are measured in a different way,thethat which cannot be combined with other studies), may that do not look at beneficial effects. included tohealth allow adverse data of collection. • effect the use Delphi-like procedures for obtaining expert Oortwijn identified 25 criteria used tobeprioritise Two sets of inclusion criteria will be needed – for judgments or topic rankings through correspondence technology assessments and categorised these into four benefit, and for adverse effects broad categories: burden of disease, potential effects, Might amount to a separate review nested traditional Cochrane review • the use of questionswithin that aare specific, explicit and conpotential costs, and uncertainty regarding application of Prostechnology Less demanding More comprehensive than just looking at included Most comprehensive sistent with standard methods for questionnaire constructhe [15]. on time and resources trials tion Does not require new literature Can potentially cover a more representative group May be able to evaluate rare, or long-term, In a moresearch recent selective review for the New Zealand strategy of patients or previously unrecognized adverse effects • full-text experimentation more to arrive Guidelines Group, the following were identified as Time and formal resource procedures intensive Cons Data may be very limited andcriteria Relatively time consuming as articles of all with short-for guideline potentially relevant studies at need checking Data may group judgments indicatingbiased that towards a topiccommon, is suitable developbe limited to well-recognized and commonly seen ment [16]:term harms adverse effects. They also suggested there is a need to define more narBenefit and harm cannot be compared directly as the Special techniques required in synthesizing rowly and precisely topics for guideline development. 1. The topic is clinically important affecting large of sources data comenumbers from different data from a diverse range of sources They argued that this would result in more organpeople with substantial morbidity or mortality (the burden of Increased quantity of dataefficient but greater risk biased and poorresolution quality data of some ization of panels andof their work, illness). Benefit andresponsive harm cannot guidelines, be compared and apparent controversies, more directly as the data come from different easier implementation.sources. 2. The topic is complex enough to initiate debate about the rec-

ommendations.

*Copied from reference [18].

Page 4 of 7 (page number not for citation purposes)

the use of 4:19 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/19

Cochrane reviews, only 138 included evidence on â&#x2030;Ľ 4000 subjects. Of these only 25 (18%) had eligible data on adverse events, while 77 had no harms data, and 36 had data on harms that were non-specific or pertained to < 4000 subjects [17]. Thus, while systematic reviews of randomised trials can provide useful information on adverse effects of clinical interventions, the reporting of adverse effects in both randomised trials and systematic reviews needs to be improved.

authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct a full systematic review. We searched PubMed and three databases of methodological studies (the Cochrane Methodology Register, the US National Guideline Clearinghouse, and the Guidelines International Network for existing systematic reviews and relevant methodological research that address these questions. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

Descriptive reports of reviews of harmful effects have found that a significant investment of effort failed to yield significant new information [18,19,36]. Authors of reviews of social interventions, on the other hand, have argued that restricting the study designs that are included in a review may reduce the value of the review and reinforce the "inverse evidence law" whereby the least is known abut the effects of interventions most likely to influence whole populations. However, this argument relates more to the importance of mapping out the available evidence than to producing reliable estimates of the effects of interventions [11].

For this review we searched PubMed using (clinical practice guidelines or public health guidelines or systematic reviews) and (study designs) and related articles for references. We searched the Cochrane Methodology Register using the key word study design, and we checked the reference lists of the reports that we retrieved. The searches were conducted in February and March 2006.

Findings

We found several systematic reviews that compared the findings of observational studies with randomised trials [27-33], and a systematic review of methods for evaluating bias in non-randomised trials [34]. We also found several descriptive studies of methods used in systematic reviews of population interventions and harmful effects.

Discussion

While there is broad agreement that the study designs to be included in a review should be dictated by the interventions being reviewed, there is uncertainty regarding what study designs to include for some specific types of questions. For any question, as the range of study designs that are included is broadened, an increasing amount of work is required to derive decreasingly reliable estimates of the effects of interventions. A decision about how broad a range of study designs to consider must be made in relationship to the characteristics of the interventions, what evidence is available, and the time and resources available.

Systematic reviews of the results of randomised trials compared with observational studies have differed in the methods they have used, and, to some extent, in their conclusions, but have generally found that it is not possible to predict differences in the size, or even the direction, of estimates of treatment effects for the same intervention when it is generated in randomized and non-randomized studies. However, especially in the more recent reports [30-33], there is the suggestion that these disparities decrease when investigators have controlled for known confounders (between risk/responsiveness and treatment).

For any question there is a cut-off point beyond which broadening the types of studies that are considered requires a substantial investment of effort that will not yield additional information that will inform decisions in a meaningful way. In many cases, it is likely to be prudent to acknowledge the limits of what is known from a restricted range of study designs, rather than to invest additional resources that are unlikely to do more than confirm the limits of what is known. Whatever decision is taken about the range of study designs to include should be made explicit, and great caution should be taken to avoid confusing a lack of evidence with evidence of no effect.

The review of methods for evaluating bias in non-randomised trials found six tools that were thought to be suitable for use in systematic reviews [34]. Their review of 511 systematic reviews that included non-randomised studies found that only 169 (33%) assessed study quality. A more recent survey of methods used in systematic reviews of adverse effects found that although more than three quarters (185/243) reviews sought to include data from sources other than randomised controlled trials, fewer than half (106/256) assessed the quality of the studies that were included [35].

Further work

There is a rapidly growing number of reviews and studies comparing the results of different study designs. High priority should be given to generating and periodically

A study that considered the potential of randomised trials to provide evidence on specific harms found that of 1727

Page 5 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:19

http://www.health-policy-systems.com/content/4/1/19 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines updating a commonInternational data set of studies Network to update [8]) for and existing reconcile different systematic reviews conclusions and relevant among methodological these reviews.research Priority should that address also these be given questions. to broadening The answers thetoscope the questions of these comparisons are our conclusions to include based a wider on therange available of questions evidence,and con-a wider rangeofof what sideration study WHO designs, and including other organisations animal and laboare ratory studies. doing, and logical Additional arguments. studies, and systematic reviews of studies, that more rigorously assess the added cost and value For this of review including we broader searchedranges PubMed of study using designs (clinicalwould prachelpguidelines tice to informordecisions public health aboutguidelines) when it isand likely (priority to be important setting or setting and worthwhile priorities) and to use related more articles diverse for types selected of study designs. references [9,10]. There Weissearched a need to thedevelop Cochrane more Methodoldetailed guidance ogy Register regarding using priority decisions or priorities. for which We study reviewed designsthe to Conference on Priorities include for website of the different 5th International types of questions for incorporation Care [11]for andWHO references that we had our files in Health the Guidelines Guidelines. Thisinguidance, [12-15]. which is particularly needed for harms and interventions targeted at populations, should be based on both empirical evidence and conceptual arguments. Findings What criteria should be used to establish priorities? The US Institute of Medicine's (IOM) Committee on Competing interests ADO and for AF work forPriorities the Norwegian Knowledge Centre Methods Setting for Guidelines Developforthein Health Services, an agency funded by the practice Norwement its study of setting priorities for clinical gian government thatinproduces systematic guidelines published 1995 argued that thereviews priority and sethealth technology three authors are conting process shouldassessments. be open andAll defensible [14] They rectributors to the Collaboration. ADO and HJS ommended six Cochrane general criteria: prevalence, burden of are members the GRADE HJS is illness, cost ofofmanaging theWorking problem,Group. variability indocupracments editor and chair of the documents development tice, potential of a guideline to improve health outcomes, implementation committee for the American and potential of a guideline to reduce costs. Because Thodata racic Society senior editor ofisthe American of used to makeand these judgements often lacking,College they sugChest Physicians' Antithrombotic Thrombolytic Thergested explicit opportunities for and important unmeasured apy Guidelines. factors to be considered. They further suggested separate "tracks" for considering issues for specific populations, conditions contributions or concerns. They argued that the rationales for Authors' ADO prepared the first draft of this and i.e. AF designating special tracks should be review. definedHJS clearly; contributed to drafting andconsideration. revising it. why they warrant special They suggested that updating of guidelines should also be considered as a separate "track", taking account of issues such as the need Acknowledgements We acknowledge WHO and the Norwegian for gratefully corrections and the availability of new Knowledge evidence.Centre

Oxman Lavis J,ofFretheim A: The use ofactual research in There is AD, evidence variation between andevidence appropriWHO recommendations. . ate care. 8. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM,

7. 3.

Atkins D, for the Methods Word Group, third U.S. Preventive Serv-

ices Task Current methods of theavailable U.S. Preventive 4. There are Force: no existing valid guidelines to use. Serv-

ices Task Force: a review of the process. Am J Prev Med 2001, 20(3S):21-35. 9. Jackson N, Waters E, for the Guidelines for evidence Systematic available. Reviews in 5. There is an adequate amount of existing Health Promotion and Public Health Taskforce: Criteria for the systematic review of health promotion and public health 6. The recommendations will beInternational acceptable to 20:367-74. the potential interventions. Health Promotion 2005, 10. users.Sheldon T: Making evidence synthesis more useful for management and policy-making. J Health Serv Res Policy 2005, 10(Suppl 1):1-5. 7. Implementation ofHamilton the guideline is feasible, will not exhaus11. Ogilvie D, Egan M, V, Petticrew : Sysematic reviews of effects of social interventions: best available evitivelyhealth use the communities' resources, and2.barriers to clinical dence: how low should you go? J Epidemiol Community Health change are59:886-92. not so high that they cannot be overcome. 2005, 12. Nutbeam D: The challenge to provide 'evidence' in health promotion. Health Promotion International 1999, 14:99-101. While burden of disease is commonly used as a criterion 13. Petticrew M: Presumed innocent. Why we need systematic for priority shouldAm beJ Prev noted use of3):2-3. sumreviews setting, of social it policies. Medthat 2003,the 23(suppl 14. M, Roberts H: Evidence, hierarchies, maryPetticrew burden of disease measures, such and as typologies: disability horses for courses. J Epidemiol Community Health 2003, 57:527-9. adjusted life L,years (DALYs) has criticised forevaluating focusing 15. Rychetnik Frommer M, Hawe P, been Shiell A: Criteria for evidencerather on public health interventions. J Epidemiol Community on disease than resource use and interventions, Healthof2002, because the 56:119-27. assumptions about values inherent in such 16. Kramer MS: Randomized trials and public health intervenmeasures, andto because the technical limitations of insuch tions: time end theof scientific double standard. Clinics Perinatology(see 2003,for 30:351-61. measures example references [17] and [18]). 17. Grimshaw J, McAuley LM, Bero LA, Grilli R, Oxman AD, Ramsay , Vale L, Zwarenstein M: Systematic reviews of the effectiveness What should be strategies used to agree on priorities? of processes quality improvement and programmes. Qual Saf Health CareHodge 2003, 12:298-303. Batista and in a review conducted 10 years ago 18. Loke YK, Price D, Herxheimer A, editors: Including adverse found only three articles pertinent to priority for effects. Cochrane Handbook for Systematic Reviews ofsetting Interventions 4.2.5 practice [updated May 2005] . Appendix clinical guidelines [10]. 6b They suggested the fol19. McIntosh HM, Nerys NF, Woolacott F, Bagnall AM: Assessing lowing framework for priority setting: harmful effects in systematic Reviews. BMC Medical Research Methodology 2004, 4:19. 20. Papanikolaou Ioannidis of large-scale evi1. Consult withPN, end users JPA: andAvailability other stakeholders before dence on specific harms from systematic reviews of randselecting topics. omized trials. Am J Med 2004, 117:582-9. 21. Higgins JPT, Green S, editors: 4.2.4 What types of study designs? Cochrane Handbook for Systematic Reviews of Interventions [updated May 2. Consider feasibility during the consultation. 2005] [http://www.cochrane.org/resources/handbook/hbook.htm]. Section 4 22. Khan KS, Kleijnen J: STAGE II. Conducting the review.topics. PHASE 3. Document the process used to select guideline 4. Selection of studies. Undertaking Systematic Reviews of Research on Effectiveness. CRD's Guidance for those Carrying Out or Commissioning The Reviews. IOM suggested the following CRD Report Number 4 procedures 2nd edition. [14]: 2001 [http:// www.york.ac.uk/inst/crd/pdf/crd4_ph4.pdf]. 23. Festing MFW, Altman DG: Guidelines for the Design and Statis• thetical useAnalysis of Delphi-like procedures obtainingAnimals. expert of Experiments Usingfor Laboratory ILAR Journal judgments or 2002, topic43:244-58. rankings through correspondence 24. Briss PA, Zaza S, Pappaioanou M, et al.: Developing an evidencebased guide to community preventive services-methods. Am • theJ Prev useMed of 2000, questions that are specific, explicit and con18(1S):35-43. 25. Campbell Systematic Reviews. for theconstrucPreparasistent with standard methods forGuidelines questionnaire tion of Review Protocols. (Version 1.0: January 1, 2001) tion [http://www.campbellcollaboration.org/ c2_protocol_guidelines%20doc.pdf] 26. Oxman AD, Fretheim Schünemann HJ:procedures Improving the use of • experimentation withA,more formal to arrive research evidence in guideline development: introduction. at group Healthjudgments Res Policy Syst 2006. 27. Heinsman DT, Shadish WR: Assignment methods in experimenwhen do nonrandomized experiments approximate Theytation: also suggested there is a need to define more naranswers from randomized experiments? Psychological Methods rowly and precisely topics for guideline development. 1996, 1:154-69. 28. A, that McKee M, would Black N,result McPherson K, Sanderson Bain C: TheyBritton argued this in more efficientC,organChoosing between randomised and non-randomised studies: ization of panels and Health theirTechnol work,Assessment resolution of some a systematic review. 1998, 2:#13. 29. Kunz R,controversies, Vist G, Oxman AD: Randomisation protect against apparent more responsive to guidelines, and selection bias in healthcare trials. The Cochrane Database of easier implementation. Methodology Reviews 2002.

forthe Health Services for financial support, the other authors of these reviews for their contributions, the other used members the ACHR health SURE, Oortwijn identified 25 criteria to ofprioritise and all of those who have provided us with feedback on earlier versions of technology assessments and categorised these into four this paper and the others in this series.

broad categories: burden of disease, potential effects, potential costs, and uncertainty regarding application of References the technology [15]. 1. Lomas J, Culyer T, McCutcheon C, McAuley L, Law S: Conceptualizing and Combining evidence for health system guidance.

Ottawa: Canadian Health Services Research Foundation; 2005. In aSchünemann more recent selective review for the New Zealand 2. HJ, Fretheim A, Oxman AD: Improving the Use of Guidelines Group, the in following were identified as Research Evidence Guidelinecriteria Development: 9. Grading evidence andarecommendations. Res Policy Syst 2006. indicating that topic is suitable Health for guideline develop3. Counsell CE, Clarke MJ, Slattery J, Sandercock PA: The miracle of mentDICE [16]:therapy for acute stroke: fact or fictional product of subgroup analysis? BMJ 1994, 309:1677-81.

Oxman G: When to believe a subgroup analysis. of In The topicAD, is Guyatt clinically important affecting large numbers Users' Guide to the Medical Literature. A Manual for Evidence-Based Clinipeople substantial or mortality (the AMA burden of cal with Practice Edited by: morbidity Guyatt G, Rennie D. Chicago: Press; 2002:553-65. illness).

4. 1.

Global Programme on Evidence for Health Policy. Guidelines for WHO Guidelines. Geneva: World Health Organisation; 2. The topic is complex enough to initiate debate about the rec2003. (EIP/GPE/EQC/2003.1) 6. Panisset U: A review of WHO recommendations published in ommendations. 2005. . 5.

Page 4 6 of 7 (page number not for citation purposes)

the use of 4:19 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/19

30.

31.

32.

33. 34. 35. 36.

MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AMS: A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technol Assessment 2000, 4(34):. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MB, Contopoulos Ioannidis DG, Lau J: Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA 2001, 286:821-30. King M, Nazareth I, Lampe F, Bower P, Chandler M, Morou M, Sibbald B, Lai R: Impact of participant and physician intervention preferences on randomized trials; a systematic review. JAMA 2005, 293:1089-99. Vist GE, Hagen KB, Devereaux PJ, Bryant D, Kristoffersen DT, Oxman AD: Systematic review to determine whether participation in a trial influences outcome. BMJ 2005, 330:1175. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al.: Evaluating non-randomised intervention studies. Health Technol Assess 2003, 7(27):1-173. Golder SP, Loke YK, McIntosh HM: Room for improvement? A survey of the methods used in systematic reviews of adverse effects. BMC Medical Research Methodology 2006, 6:3. Loke YK, Derry S: Incorporating adverse effects data into reviews: how to get started [abstract]. 9th Annual Meeting for UK Contributors to the Cochrane Collaboration: Coventry 2003.

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 7 of 7 (page number not for citation purposes)

Health Research Policy and Systems Health Research Policy and Systems 2006, 4:14

http://www.health-policy-systems.com/content/4/1/14 BioMed Central

and the Guidelines International Network [8]) for existing 3. There is evidence of variation between actual and appropriOpen Access Review systematic reviews and relevant methodological research ate care. that address these questions. The answers to the questions are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. sideration of what WHO and other organisations are 1 2 an adequate 1 Andrew D Oxman* Atle amount Fretheim doing, and logical arguments. , Holger J Schünemann 5. There isand of existing evidence available.

Improving the use of research evidence in guideline development: 8. Synthesis and presentation of evidence

For this 1review we searched PubMed using (clinical prac6. The recommendations will be acceptable to the potential Address: Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway and 2INFORMA, S.C. tice guidelines or public health guidelines) and (priority Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome,users. Italy setting or setting priorities) and related articles for selected Email: Andrew D Oxman* - oxman@online.no; Holger J Schünemann - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhaus* Corresponding author ogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical website of the 5th International Conference on Priorities change are not so high that they cannot be overcome. in Health Care [11] and references that we had in our files [12-15]. While burden of2006 disease is commonly used as a criterion Published: 05 December 2006 Received: 07 April Accepted: 05setting, December 2006 for priority it should be noted that the use of sumHealth Research Policy and Systems 2006, 4:20 doi:10.1186/1478-4505-4-20 mary burden of disease measures, such as disability Findings This article is available from: http://www.health-policy-systems.com/content/4/1/20 What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing © 2006 et al; licensee BioMed Central Ltd. Committee on The USOxman Institute of Medicine's (IOM) on disease rather than resource use and interventions, This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). ting process should be open and defensible [14] They recWhat processes should be used to agree on priorities? ommended six general criteria: prevalence, burden of Abstract Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracBackground: The World Health Organization (WHO), like many other organisations around the world, has recognised found only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, the need to use more rigorous processes to ensure that health care recommendations are informed by the best available clinical practice guidelines [10]. They suggested the foland potential of a guideline to reduce costs. Because data research evidence. This is the eighth of a series of 16 reviews that have been prepared as background for advice from lowing framework for priority setting: used to make these judgements is often lacking, they sugthe WHO Advisory Committee on Health Research to WHO on how to achieve this. gested explicit opportunities for important unmeasured Objectives: We reviewed literature on theseparate synthesis and of end research focusing on four before key 1.presentation Consult with usersevidence, and other stakeholders factors to be considered. Theythe further suggested questions. selecting topics. "tracks" for considering issues for specific populations, conditions or concerns. TheyPubMed argued that rationales forof methodological studies for existing systematic reviews and Methods: We searched and the three databases 2. Consider the consultation. designating special tracks should be defined clearly; i.e. relevant methodological research. We did not conduct systematic reviews feasibility ourselves. during Our conclusions are based on the whyavailable they warrant special consideration. They suggested evidence, consideration of what WHO and other organisations are doing and logical arguments. 3. Document the process used to select guideline topics. that updating of guidelines should also be considered as a Key questions and account answers:ofWe found twoasreviews of instruments for critically appraising systematic reviews, several separate "track", taking issues such the need studies of the of using extensive searches for reviews and determining when it isprocedures important to update The IOM suggested the following [14]: for corrections andimportance the availability of new evidence. reviews, and consensus statements about the reporting of reviews that informed our answers to the following questions. • the use of Delphi-like procedures for obtaining expert Oortwijn identified 25 criteria used to prioritise health appraised? How should existing systematic reviews be critically judgments or topic rankings through correspondence technology assessments and categorised these into four • Because preparing systematic reviews can take over a year and require capacity and resources, existing reviews should broad categories: burden of disease, potential effects, be used when possible and updated, if needed. • the use of questions that are specific, explicit and conpotential costs, and uncertainty regarding application of standardshould methods for questionnaire constructhe technology [15]. such as A MeaSurement Tool to Assess sistent • Standard criteria, Reviewswith (AMSTAR), be used to critically appraise existing systematic reviews, together with an assessment of the relevance of the review to the questions being asked. tion In a more recent selective review for the New Zealand When and how should WHO undertake or commission new reviews? • experimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as • Consideration should given to commissioning a newjudgments review whenever a relevant, up-to-date review at group indicating that a topic is be suitable forundertaking guideline or developof [16]: good quality is not available. ment also suggested there is The a need to define more nar• When time or resources are limited it may be necessary toThey undertake rapid assessments. methods that are used rowly and precisely topics for guideline development. 1. The topic is clinically important affecting large numbers of to do these assessments should be reported, including important limitations and uncertainties and explicit consideration They argued that this would result in more efficient organpeople withneed substantial morbidity or mortality (thesystematic burden ofreview. of the and urgency of undertaking a full ization of panels and their work, resolution of some illness). • Because WHO has limited capacity for undertaking systematic reviews, reviews will more often need to be commissioned apparent controversies, responsive guidelines, and when a new review is needed. Consideration should be given to establishing collaborating centres to undertake or easier implementation. 2. The topic is complex enough to initiate debate about the recsupport this work, similar to what some national organisations have done. ommendations.

Page Page14ofof10 7 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:20 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/20

How should the findings of systematic reviews be summarised and presented to committees responsible for making recommendations? • Concise summaries (evidence tables) of the best available evidence for each important outcome, including benefits, harms and costs, should be presented to the groups responsible for making recommendations. These should include an assessment of the quality of the evidence and a summary of the findings for each outcome. • The full systematic reviews, on which the summaries are based, should also be available to both those making recommendations and users of the recommendations. What additional information is needed to inform recommendations and how should this information be synthesised with information about effects and presented to committees? • Additional information that is needed to inform recommendations includes factors that might modify the expected effects, need (prevalence, baseline risk or status), values (the relative importance of key outcomes), costs and the availability of resources. • Any assumptions that are made about values or other factors that may vary from setting to setting should be made explicit. • For global guidelines that are intended to inform decisions in different settings, consideration should be given to using a template to assist the synthesis of information specific to a setting with the global evidence of the effects of the relevant interventions.

Background

present that information to the group of people who will consider that evidence, together with other evidence and judgements, to develop recommendations. In addition, consideration needs to be given to the additional information that is needed and how that should be summarised and presented.

In this paper we address the following questions: • How should existing systematic reviews be critically appraised and used?

A summary of the best available research evidence is essential, though not sufficient to inform recommendations. To reduce the risk of bias and errors that occur by chance, and to facilitate critical appraisal of syntheses of evidence, reviews should be systematic and should explicitly report the methods that were used [1]. However, systematic reviews require resources, take time, and may not always be warranted or possible. Moreover, unnecessary duplication of systematic reviews should be avoided, given the large unmet need for systematic reviews of a wide range of questions and the need to keep reviews upto-date [2,3].

• When and how should WHO undertake or commission new reviews? • How should the findings of systematic reviews be summarised and presented to committees responsible for making recommendations? • What additional information is needed to inform recommendations and how should this information be synthesised with information about effects and presented to committees?

The first step in considering the needs for systematic reviews to inform recommendations is to critically appraise existing reviews to determine if they provide an adequate summary of the relevant evidence that is needed, particularly evidence of the effects of the different options (interventions) that are being considered. If they do not, consideration must then be given to whether a new review should be undertaken and how best to obtain a new review. Once an adequate summary of the evidence is available, consideration must be given to how best to

Related questions on priority setting for guidelines or recommendations and reporting of guidelines are addressed in other papers in this series [4,5].

What WHO is doing now

Of 62 WHO documents that were indexed as guidelines in 2005, only two reported a systematic review and less than 40% included references [6]. Although it is possible that systematic reviews are being used and this is not being

Page 2 of 10 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:20

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/20

and reported, the Guidelines this is unlikely. International With some Network notable [8]) for exceptions, existing systematic for the most reviews part and recommendations relevant methodological are currently research being that made address without these adequate questions. use of The existing answers systematic to the questions reviews are andour systematic conclusions reviews basedare on rarely the available being evidence, undertaken conor sideration commissioned of what by WHO WHOcommittees and other that organisations make recomare doing, mendations. and logical However, arguments. the situation may be somewhat better than what is reported in published guidelines. For When thisasked reviewabout we searched the use PubMed of evidence using of (clinical effects specifipractice callyguidelines in an interview or public health study guidelines) [7], many and departments (priority setting reported or setting using priorities) background and related documents. articlesThese for selected were references reported to [9,10]. have We been searched prepared the in Cochrane a varietyMethodolof ways, ogy including Registerasusing unpublished priority orworking priorities. papers, We reviewed documents the th International Conference on Priorities website similar to ofthose the 5used by the Scottish Intercollegiate Guidein Health Care [11] and references that weprepared had in our lines Network (SIGN), and documents byfiles the [12-15]. participating experts. Only two departments reported using systematic reviews specifically, while several others reported using systematic reviews along with a range of Findings What should be used to establish priorities? other criteria documents. Others reported leaving the use of eviThe USupInstitute of Medicine's (IOM) Committee on dence to the experts, a lack of documentation, evidence Methods Priorities Guidelines Developof effects for not Setting being relevant forfor some recommendations, ment in its astudy of setting priorities for clinical practice and using mixture of "epidemiological data, trial data, guidelines published in 1995 argued(common that the priority opinions based on logical reasoning sense) setand ting process should be open and defensible [14] They recclinical experience." ommended six general criteria: prevalence, burden of illness, cost of managing theusing problem, variability in pracNo departments reported concise summaries of tice, potential of a guideline improve health outcomes, findings or "balance sheets"tofor the most important outand potential of a harms guideline reducefor costs. comes (benefits, andtocosts) the Because options data that used make theseWHO judgements often lacking, they sugwere to considered. groupsisthat develop recommengested forcomposed importantofunmeasured dationsexplicit are, foropportunities the most part, experts in a factors to be considered. They further suggested separate particular content area and not supported by experts in "tracks" formethodological considering issues specific populations, particular areasfor (e.g. systematic reviews) conditions or concerns. They arguedskills that the for or by staff with particular technical (e.g.rationales information designating special tracks should beappears definedtoclearly; i.e. retrieval). Relatively little attention have been why warrant special consideration. They global suggested giventhey to how best to help member states adapt recthat updating of guidelines shouldof also be considered as a ommendations, taking account local needs, values, separate taking account of issues such as the need resources"track", and conditions. for corrections and the availability of new evidence.

period to 18 of months. NICE reviews areand available in 3. Thereof is 12 evidence variation between actual approprithecare. full version of its guidelines. ate Other 4. Thereorganisations are no existingthat validproduce guidelinesguidelines available tosometimes use. use existing systematic reviews, sometimes prepare their own systematic reviews, and sometimes commission 5. There is an adequate amount of existing evidence available. reviews. The U.S. Preventive Services Task Force, for example, commissions systematic reviews from Evidence-based 6. The recommendations will be acceptable to the potential Practice Centers (EPCs) for updates of its guidelines [12]. users. The Agency for Healthcare Research and Quality (AHRQ) has contracts with from which itwill commissions 7. Implementation of 13 the EPCs guideline is feasible, not exhaussystematic AHRQresources, does notand produce tively use thereviews. communities' barriersguidelines, to clinical but stakeholder organisations that be request the reviews change are not so high that they cannot overcome. may produce guidelines. Other health technology assessment burden (HTA) agencies, may or used may as not produce While of diseasewhich is commonly a criterion guidelines, have staff that be undertake reviews, for priority setting, it should noted that the use convene of sumexpertburden groups that undertake reviews such together with supmary of disease measures, as disability port from staff, or(DALYs) commission systematic reviews [13]. adjusted life years has been criticised for focusing on disease rather than resource use and interventions, Systematic reviews of the effects of interventions because of the assumptions about values inherent in are sucha major focus most of organisations developofguidemeasures, andfor because the technicalthat limitations such lines. Because most organisations develop recommendameasures (see for example references [17] and [18]). tions for a specific country or setting, they are able to take What processes should beinformation used to agree on priorities? into account additional relevant to the speBatista and Hodge a review 10 years ago cific context for in which the conducted recommendations are found onlyincluding three articles pertinent to priority setting for intended, factors that might affect the applicaclinical guidelines [10]. They suggested folbility ofpractice the evidence in specific settings, need the (prevalowing for priority setting: lence, framework baseline risk or status), values, costs and the availability of resources. 1. Consult with end users and other stakeholders before selecting topics. Methods The methods used to prepare this review are described in the introduction to this seriesthe [14]. Briefly, the key ques2. Consider feasibility during consultation. tions addressed in this paper were vetted amongst the authors and the theprocess ACHR used Subcommittee on thetopics. Use of 3. Document to select guideline Research Evidence (SURE). We did not conduct a full systematic We the searched PubMed and three databases The IOMreview. suggested following procedures [14]: of methodological studies (the Cochrane Methodology the US National Guideline Clearinghouse â&#x20AC;˘Register the use[15], of Delphi-like procedures for obtaining expert [16], and the Guidelines International Network [17]) for judgments or topic rankings through correspondence existing systematic reviews and relevant methodological address these questions. answers the â&#x20AC;˘research the use that of questions that are specific,The explicit andtoconquestions our conclusions based on the available evisistent withare standard methods for questionnaire construcdence, consideration of what WHO and other organisation tions are doing, and logical arguments. â&#x20AC;˘ experimentation with more formal procedures to arrive For this judgments review we knew of two previous systematic at group reviews of instruments for critically appraising systematic reviews through personal andmore studies of They also suggested there contacts is a need[18,19], to define narhow toand present the results systematic reviews to policy rowly precisely topicsoffor guideline development. makers [20], public [21], efficient and users of They argued thatthe thisgeneral would result in more organCochrane [22]. their We used these studies and their ization of reviews panels and work, resolution of some reference controversies, lists to identify related articles guidelines, in PubMed.and We apparent more responsive searched the Cochrane Methodology Register using the easier implementation. key word 'Presentation of reviews: General' and we

What other organisations are doing

In contrast, in a survey 101 organisations that produce Oortwijn identified 25 of criteria used to prioritise health clinical practice guidelines of the these 58 respondents technology assessments and95% categorised into four reported that they burden provide of guideline with systembroad categories: disease,panels potential effects, atic reviews [8]. and In another surveyregarding of 18 prominent organpotential costs, uncertainty application of isations that develop the technology [15]. clinical practice guidelines, all but one reported using systematic reviews [9]. In a more recent selective review for the New Zealand The UK National Centre for Health Clinical ExcelGuidelines Group, the following criteriaand were identified as lence (NICE), example routinely undertakesdevelopsystemindicating that for a topic is suitable for guideline atic reviews ment [16]: to inform its guideline panels [10]. NICE has seven professionally led National Collaborating Centres to The manage development of affecting clinical large guidelines [11]. 1. topic isthe clinically important numbers of Each Centre has a range of skills abilities, people with substantial morbidity or and mortality (the including burden of systematic reviewing. The Centres are responsible for illness). identifying the best and most relevant evidence available. They first consultation draft of a guideline over 2. Thewrite topic the is complex enough to initiate debate about the rec-a ommendations.

Page Page34ofof10 7 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:20 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/20

checked the reference lists of the reports that we retrieved. We searched for literature on priority setting for guidelines and health technology assessments for another report [4]. In addition, we searched broadly for literature on commissioning systematic reviews in PubMed (commissioning systematic reviews) and using Google ("commissioning systematic reviews" and "updating systematic reviews") and in the Cochrane Methodology Register using the terms 'commissioning' and 'updating systematic reviews'. The searches were conducted in March 2006.

Based on the second expert consultation, the AMSTAR 2005 was selected as the best instrument for appraising systematic reviews (Table 1). A description of the rationale for selecting that instrument is not available. When and how should WHO undertake or commission new reviews? There is wide agreement that guidelines should be informed by systematic reviews of the best available evidence among organisations that develop clinical practice guidelines and, increasingly, among organisations that develop guidance for population interventions (public health, health promotion, health systems and social interventions) [8,9,28-34]. Thus, priorities for systematic reviews are set, to some extent, when a decision is first made to develop recommendations. We reviewed the methodological literature relevant to priority setting for guidelines and health technology assessments, which overlaps largely with priority setting for systematic reviews, in our review on setting priorities for developing recommendations [4]. Additional questions related to undertaking or commissioning new reviews include: If there is a systematic review is it of good enough quality and recent enough that a new review is unlikely to be needed? Are there sufficient time and resources to commission or undertake a new review, if one is needed? If there is time, resources and a need for a new review, what is the best approach to getting the work done?

Findings

How should existing systematic reviews be critically appraised? The first of two reviews of different instruments for critically appraising systematic reviews found 20 systems concerned with the appraisal of systematic reviews or metaanalyses, including one scale, 10 checklists, and nine guidance documents [18]. The authors identified seven key domains that they considered important to appraise: study question, search strategy, inclusion and exclusion criteria, data abstraction, study quality, data synthesis and analysis, and funding or sponsorship. One checklist fully addressed all seven domains [23]. A second checklist also addressed all seven domains but merited only a "Partial" score for study question and study quality [24]. Two additional checklists and the one scale addressed six of the seven domains [25-27]. These latter two checklists excluded funding; the scale omitted data abstraction and had a "Partial" score for search strategy. The authors concluded that based on coverage of the seven domains that they considered key, these five systems (four checklists and one scale) represented "best practice" (i.e. were the best available instruments) for appraising systematic reviews. Although they considered other aspects of the systems, such as the methods used to select items and inter-rater reliability, they did not take these factors into consideration in their selection of these five systems, nor did they consider the suitability of the different systems for specific purposes.

The first of these questions can be answered by considering the criteria discussed above and the likelihood of whether new research is likely to have been completed. Under some circumstances, it may not be warranted or possible to undertake or commission a systematic review even if there is not a previous systematic review; for example, for emerging diseases when it is known that the available evidence is sparse and when decisions must be made urgently. We address which evidence should be used to address different types of questions in another paper in this series [35]. As we suggest in that paper, there is a cut-off point beyond which broadening the types of studies that are included requires a substantial investment of effort that will not yield additional information that usefully informs decisions. Similarly, there is a cut-off point beyond which more extensive searches are unlikely to yield additional useful studies.

The second review used a detailed process to evaluate and select a system and expanded the work by AHRQ up until the year 2005 [19]. They identified approximately 240 quality assessment instruments for systematic reviews, randomized controlled trials and observational studies as well as nearly 50 evidence grading systems. The instruments and systems identified were evaluated by type of study using the AHRQ evaluation grids from the first review, and considering descriptive items for most potential instruments and systems. The highest scoring instruments and systems from each grid represented the proposed selections. The proposed selections were then sent to the same experts that were contacted to review and provide comment during the initial expert consultation.

An assessment of 159 systematic reviews with comprehensive literature searches found that the importance of trials that are difficult to locate may vary, but that generally in situations where resources are limited, thorough quality assessments should take precedence over extensive literature searches and translations of articles [36,37]. Consist-

Page 4 of 10 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:20

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/20

Table 1: Guidelines A MeaSurement Tool to Assess Reviews (AMSTAR), 2005 (from COMPUS [19]) of variation between actual and appropriand the International Network [8]) for existing 3. There is evidence systematic reviews and relevant methodological research ate care. 1. Was an 'athese priori'questions. design provided? that address The answers to the questions are conclusions based on the available evidence, 4.conduct There are noreview. existing valid guidelines available to use. Theour research question and inclusion criteria should be establishedconbefore the of the sideration of what WHO and other organisations are Yes and logical arguments. doing, 5. There is an adequate amount of existing evidence available.

For this review we searched PubMed using (clinical prac6. The recommendations will be acceptable to the potential answer tice Can't guidelines or public health guidelines) and (priority users. setting or setting priorities) and related articles for selected Not applicable references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhaus2. Were there duplicate study selection and data extraction? ogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical International Conference on and Priorities website of the 5thleast change are not so high that should they cannot be overcome. There should be at two independent data extractors the consensus procedure for disagreements be reported. in Health Care [11] and references that we had in our files Yes [12-15]. While burden of disease is commonly used as a criterion No for priority setting, it should be noted that the use of summary burden of disease measures, such as disability Findings Can't answer What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing The Not USapplicable Institute of Medicine's (IOM) Committee on on disease rather than resource use and interventions, Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such 3. Was a comprehensive literature search performed? ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such At least twopublished electronic sources should be searched. Thepriority report must years and databases Central, EPOC, and[17] MEDLINE). Key words guidelines in 1995 argued that the set-includemeasures (see for (e.g., example references and [18]). and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting ting process should be open and defensible [14] They reccurrent contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies What processes should be used to agree on priorities? ommended six general criteria: prevalence, burden of found. Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracfound only three articles pertinent to priority setting for tice,Yes potential of a guideline to improve health outcomes, clinical practice guidelines [10]. They suggested the folandNo potential of a guideline to reduce costs. Because data lowing framework for priority setting: used to make these judgements is often lacking, they sugCan't answer gested explicit opportunities for important unmeasured 1. Consult with end users and other stakeholders before factors be considered. They further suggested separate Not to applicable selecting topics. "tracks" for considering issues for specific populations, 4. Was the status of publication (i.e., grey literature) used as an exclusion criterion? conditions or concerns. They argued that the rationales for The authors should state that they searched reports clearly; regardlessi.e. of their 2. publication type. The authorsduring should the stateconsultation. whether or not they Consider feasibility designating special tracks should be for defined excluded reports (from the consideration. systematic review),They based suggested on their publication status. why theyany warrant special 3. Document the process used to select guideline topics. thatYes updating of guidelines should also be considered as a separate "track", taking account of issues such as the need No The IOM suggested the following procedures [14]: for corrections and the availability of new evidence. Can't answer

â&#x20AC;˘ the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence

Oortwijn identified 25 criteria used to prioritise health Not applicable technology assessments and categorised these into four broad categories: burden of disease, potential effects, 5. Was a list of studies (included and excluded) provided? potential costs, and uncertainty regarding application of A list of included and excluded studies should be provided. the technology [15].

â&#x20AC;˘ the use of questions that are specific, explicit and consistent with standard methods for questionnaire construction

Yes

In a more recent selective review for the New Zealand No Guidelines Group, the following criteria were identified as Can't answer indicating that a topic is suitable for guideline development [16]:

â&#x20AC;˘ experimentation with more formal procedures to arrive at group judgments

Not applicable

They also suggested there is a need to define more narWere theischaracteristics of the included provided? rowly and precisely topics for guideline development. 1.6.The topic clinically important affecting studies large numbers of They argued that this would result in more efficient organpeople with substantial morbidity or mortality (the burden of In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges ization ofdata, panels their work, resolution of some illness). of characteristics in all the studies analyzed (e.g., age, race, sex, relevant socioeconomic diseaseand status, duration, severity, or other diseases) should be reported. apparent controversies, more responsive guidelines, and easier implementation. 2. The topic is complex enough to initiate debate about the recYes ommendations.

Page Page54ofof10 7 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:20 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/20

Table 1: A MeaSurement Tool to Assess Reviews (AMSTAR), 2005 (from COMPUS [19]) (Continued)

No Can't answer Not applicable 7. Was the scientific quality of the included studies assessed and reported? 'A priori' methods of assessment should be reported (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant. Yes No Can't answer Not applicable 8. Was the scientific quality of the included studies used appropriately in formulating conclusions? The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations. Yes No Can't answer Not applicable 9. Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess the homogeneity (i.e., Chi-squared test for homogeneity, I2). If heterogeneity exists, random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e., is it sensible to combine?). Yes No Can't answer Not applicable 10. Was the likelihood of publication bias assessed? An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot) and statistical tests (e.g., Egger regression test). Yes No Can't answer Not applicable 11. Was the conflict of interest stated? Potential sources of support should be clearly acknowledged in both the systematic review and the included studies. Yes No Can't answer Not applicable Source: AMSTAR 2005 (Beverley Shea, CIET, Institute of Population Health, Ottawa: personal communication, 2005 Oct)

Page 6 of 10 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:20

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/20

and ent the with Guidelines this, another International assessment Network of Cochrane [8]) for existing reviews systematic found thatreviews additional and relevant databasemethodological searching beyond research the that Cochrane addressCentral these questions. Register The of Controlled answers to the Trials questions (CENare TRAL) our retrieved conclusions onlybased a small onpercentage the available of extra evidence, trials,conand sideration that contacting of what authors WHO and and manufacturers other organisations to find unpubare doing, lished and trialslogical appeared arguments. to be a more effective method of obtaining additional better quality trials [38]. For this review we searched PubMed using (clinical practice Similarly, guidelines a third or public assessment healthof guidelines) 20 Technology and (priority Assesssetting ment or Reports settingby priorities) NICE found and related thatarticles a more for selected selective references approach [9,10]. to database We searched searching thewould Cochrane suffice Methodolin most ogy casesRegister and would usingsave priority resources, or priorities. whereasWe searching reviewedother the Priorities website sources,of including the 5th International contact with Conference experts and on checking refin Health Careappeared [11] and to references that productive we had in our files erence lists, be a more way of [12-15]. identifying further studies [39]. Searching additional databases beyond the Cochrane Library, MEDLINE, EMBASE and SCI, plus BIOSIS limited to meeting abstracts only, Findings What criteriafound shouldtobebeused to establish priorities? was seldom effective in retrieving additional The US for Institute of in Medicine's (IOM) Committee on studies inclusion the clinical and cost-effectiveness Methods forTechnology Setting Priorities for Guidelines Developsections of Assessment Reports (apart from ment in of its cancer study of setting priorities for clinical practice reviews therapies, where a search of the ASCO guidelines published in 1995 argued that the priority setdatabase was recommended). ting process should be open and defensible [14] They recommended generalforcriteria: prevalence, of Information six retrieval systematic reviews burden for public illness, cost other of managing the problem, variability in prachealth and non-clinical interventions may be more tice, potential of a guideline to improve health outcomes, elusive than retrieval for reviews in clinical medicine, due and potential of a guideline to reduce costs. Because data to the interdisciplinary nature of the research, use of used to make these judgements is often lacking, they sugresearch designs other than randomised trials, and limitagested explicit opportunities for important unmeasured tions of what and how the research is indexed. While it factors be considered. They further suggested strategies separate may betoimportant to consider other databases, "tracks" for database considering issues are for likely specific populations, other than searching to be important conditions or concerns. They argued that the rationales for [40,41]. Moreover, database searching in public health designating special tracks should be defined clearly;skills i.e. and other non-clinical areas may require specialised why they warrant special of consideration. due to technical demands the databasesThey to besuggested searched, that of guidelines should also be considered as a lackupdating of standardization of the vocabulary, and the relative separate "track", taking account [42]. of issues such as the need scarcity of rigorous evaluations Information retrieval for corrections the availability of newtoevidence. specialists mayand require a broad exposure databases, the grey literature and the terminology that is used. Oortwijn identified 25 criteria used to prioritise health technology assessments categorised these into four Several investigators haveand addressed the question of when broad categories: burden disease, [37,43-47]. potential effects, a review or guideline needsofupdating French potential costs, and uncertainty of and colleagues found that of a regarding sample ofapplication 254 updated the technology [15].23 (9%) had a change in conclusion Cochrane reviews [43]. Another survey of Cochrane reviews found that of In a more recent selective review theofNew Zealand 104 updated reviews in the firstforhalf 2003, 77% Guidelines Group, the following criteria were identifiedthe as included no new data or data insufficient to influence indicating a topic suitable for guideline developconclusion.that In 16% newisdata had some impact on conclument sions [16]: without major change, and in only 5% new data resulted in major changes in conclusions [44]. 1. The topic is clinically important affecting large numbers of people withand substantial morbidity or other mortality (thefound burdenthat of Johnston colleagues, on the hand, illness). an updating strategy for cancer practice guidelines found 80 pieces of new evidence over a one-year period relating 2. topic complex enough to initiate about the rectoThe 17 of 20 isguidelines [45]. On averagedebate four pieces of new ommendations. evidence were found per guideline, but there was consid-

3. Therevariation is evidence of variation between Of actual approprierable across the guidelines. the and 80 pieces, 19 ate care. contributed to modifications of clinical recommendations in six practice guidelines, whereas the remaining evi4. Theresupported are no existing guidelines available to use.In this dence the valid original recommendations. case the updating process was resource intensive, but 5. There is an adequate amountHowever, of existing itevidence available. yielded important findings. was possible to reduce the scope of the sources searched routinely to 6. The recommendations willLibrary be acceptable to the proceedpotential MEDLINE, the Cochrane and meeting users. ings. Another review of 17 guidelines published by AHRQ found that for seven guidelines new evidence and expert 7. Implementation of the an guideline is feasible, will notsix exhausjudgement indicated update was needed, were tively communities' resources, and three barriers to considclinical founduse to the be in need of a minor update, were change arevalid, not soand highnothat they cannot overcome. ered still conclusion wasbedrawn for one [47]. The authors found that no more than 90% of the guideWhile burden disease commonly usedestimated as a criterion lines were stillof valid after is 3.6 years and they that for priority setting, it should be outdated noted that of sumabout half the guidelines were inthe 5.8use years. They mary burden disease measures, such as disability concluded thatof guidelines should be reassessed every adjusted life years (DALYs) has been criticised for focusing three years. on disease rather than resource use and interventions, because of the assumptions abouttime-consuming. values inherent inMany such Comprehensive reviews are measures, and because of the technical limitations ofestabsuch health technology assessment (HTA) agencies have measures (see assessment for exampleprocesses, references particularly [17] and [18]). lished rapid for new technologies [48-51]. There is no common definition of What should be there used to on priorities? "rapidprocesses assessment" and is agree variation in the scope, Batista andand Hodge review conducted 10 While years ago methods time in to acomplete assessments. the found only three articles pertinent priority setting evifor concept is intuitively sound, there to is little empirical clinical practice guidelines They or suggested therapid foldence comparing alternative[10]. methods comparing lowing framework priority setting: assessments with for more comprehensive methods. Milne and colleagues have described a range of HTA responses 1. Consultinwith users and 2–3 otherpage stakeholders before available the end UK, including assessments that selecting topics. rapid systematic reviews that take 8–10 take six weeks, weeks, technology assessment reviews that take six 2. Consider feasibility duringand thefull consultation. months, Cochrane reviews, HTA reports that take 3 years [52]. They identify three factors that determine the 3. Document the process used to selectwant, guideline topics. HTA response: what decision-makers including the time scale for decision making; the characteristics of the The IOM suggested the the following procedures technology, including importance of the [14]: uncertainty, the importance of the potential benefits, the rate of diffu•sion, the use Delphi-like for obtaining expert andofhow much isprocedures already known from previous judgments or topic rankings through correspondence assessments; and the resources available for an assessment. • the use of questions that are specific, explicit and consistent for questionnaire We didwith notstandard find anymethods evaluations of alternativeconstrucmethods tion for commissioning reviews or of comparisons between commissioning reviews and doing them in house. A sur•vey experimentation with more formal reviews procedures to arrive of people preparing Cochrane in Australia at group judgments (with a response rate of 92/112) found that the most critical barriers to completion of a Cochrane review were lack They also suggested there is a need to define more narof time (80%), lack of financial support (36%), methodrowly andproblems precisely (23%) topics for ological andguideline problemsdevelopment. with group They argued that this would result in more efficient organdynamics (10%) [53]. ization of panels and their work, resolution of some apparent controversies, more responsive guidelines, and easier implementation.

Page Page74ofof10 7 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:20 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/20

How should the findings of systematic reviews be summarised and presented to committees responsible for making recommendations? The Conference on Guideline Standardization (COGS) developed an 18-item checklist for the reporting of guidelines [29]. The checklist includes the method for synthesizing evidence (how evidence was used to create recommendations, e.g., evidence tables, meta-analysis, decision analysis) and the recommendation grading criteria (the criteria used to rate the quality of evidence that supports the recommendations and the system for describing the strength of the recommendations).

[62,63]. Because factors such as modifying factors, needs and the availability of resources can vary greatly from setting to setting, methods for incorporating this information in global guidelines are particularly challenging. We did not find any evaluations of methods for addressing these challenges.

Discussion

There is broad agreement on the need for systematic reviews to inform recommendations and on criteria for critically appraising systematic reviews. Several criteria have been identified that need to be considered when deciding whether a new systematic review is needed, including the needs of decision makers, the nature of the problem and the relevant interventions, and the availability of resources.

The GRADE Working Group recommends the use of evidence profiles including detailed descriptions of the judgements used to assess the quality of evidence for each important outcome and a summary of the findings for each important outcome [54,55]. More recently the Cochrane Collaboration has developed summary of findings tables, based in part on GRADE evidence profiles [22,56].

The available evidence suggests that, generally, in situations where time or resources are limited, thorough quality assessments should likely take precedence over extensive literature searches. When a full systematic review is not undertaken, for example because of the need for a rapid response, explicit consideration should be given to the need and urgency of undertaking a full systematic review and putting in place appropriate mechanisms for timely updating of the recommendations.

All of these methods of presenting evidence to decision makers are based on consultations informed by evidence, such as comparisons of different ways of presenting evidence. We did not find comparisons of different ways of presenting evidence to groups developing recommendations.

The frequency with which reviews or guidelines need to be updated is likely to vary, but as a rough rule of thumb, based in part on a study of clinical practice guidelines, the need for updating should be considered routinely after three years and more often for areas that are developing rapidly.

In addition to summaries of the main findings, such as evidence profiles, the full systematic reviews should be available to both those making recommendations and to users of the recommendations [29]. These full systematic reviews should adhere to standards such as those recommended in the QUOROM statement [57].

Further work

Both the Agency for Healthcare Research and Quality and the Canadian Coordinating Office for Health Technology Assessment have funded projects on updating systematic reviews [64,65]. These reports should help to fill in some of the gaps in this review regarding when and how to undertake or commission an update of a review. Further work is needed on several of the other questions asked in this review, including evaluation of methods for rapid assessments, how best to present evidence to groups making recommendations and, importantly for WHO, how best to take into consideration information that varies from setting to setting when making global recommendations.

What additional information is needed to inform recommendations and how should this information be synthesised with information about effects and presented to committees? Although there are a number of descriptive papers and guidelines for what additional information is needed in addition to systematic reviews of the effects of the options that are being considered, we did not find comparisons of alternative ways of synthesising this information and presenting it to groups making recommendations. As discussed in another article in this series [58], additional information that needs to be considered in a recommendation includes factors that might modify the expected effects, need (prevalence, baseline risk or status), values [59], costs and the availability of resources.

Competing interests

Methods of integrating this additional information and judgements include formal and informal consensus methods [60,61], decision analyses, and economic analyses

Page 8 of 10 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:20

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/20

and are members the Guidelines of theInternational GRADE Working Network Group. [8])HJS for existing is docusystematic ments editor reviews and and chairrelevant of the documents methodological development research that and address implementation these questions. committee The answers for thetoAmerican the questions Thoare racic our Society conclusions and senior based editor on the of available the American evidence, College conof sideration Chest Physicians' of what Antithrombotic WHO and other and Thrombolytic organisationsTherare doing, apy Guidelines. and logical arguments.

17.There Guidelines International Network [http://www.g-i3. is evidence of variation between(GIN) actual and approprin.net] ate care. 18. West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al.: Sys-

tems to rate the strength of scientific evidence [Evidence

report/technology assessment no available 47]. 2002 4. There are no existing valid guidelines to use. [http://

www.thecre.com/pdf/ahrq-system-strength.pdf]. Rockville (MD): Agency for Healthcare Research and Quality; AHRQ Publication No 02-E016 5. There is an adequate amount of existing evidence available. 19. Proposed Evaluation Tools for COMPUS [https://www.cco hta.ca/compus/ 6. The recommendations will be acceptable to the potential compus_pdfCOMPUS_Evaluation_Methodology_draft_e.pdf]. users.Ottawa: Canadian Coordinating Office for Health Technology Assessment November 29, 2005. 20. Lavis JN, Davies HTO, Oxman AD, Denis JL, Golden-Biddle K, Ferlie 7. Implementation of the guideline is feasible, not exhausE: Towards systematic reviews that informwill healthcare manpolicymaking. Journal ofand Health Services to Research and tivelyagement use the and communities' resources, barriers clinical Policy 2005, 10(Suppl 1):35-48. change are not high that be overcome. 21. Glenton C, so Underland V, they Kho cannot M, Oxman AD: Translating evidence into patient information – challenges and possible solutions when extracting and presenting the results of sysWhile burden of disease is commonly used as a criterion tematic reviews. J Clin Epidemiol in press. for should beJ:noted thatofthe use of Tables sum22. priority Glasziou setting, P, OxmanitAD, Higgins Summary Findings Cochrane Reviews: Draft Specification RevMan marywithin burden of disease measures, such as for disability 5.0. December 2004. Obtaining a consensus on the content and methadjusted years (DALYs) criticised forReport focusing ods oflife a Summary of Findings has table been for Cochrane Reviews. to the Cochrane Collaboration Steering Group 2005. reports availon disease rather than resource use(unpublished and interventions, able of from theassumptions author) because the about values inherent in such 23. Sacks HS, Reitman D, Pagano D, Kupelnick B: Metaanalysis: an measures, and the63:216-24. technical limitations of such update. Mt because Sinai J Medof 1996, 24. Auperin A, Pignon JP, Poynard T: Review[17] article: measures (see for example references andcritical [18]).review of meta-analyses of randomized clinical trials in hepatogastroenterology. Alimentary Pharmacol Ther 1997, 11:215-25. What processes shouldAN, be used to agree 25. Irwig L, Tosteson Gatsonis C, et on al.: priorities? Guidelines for metaanalyses evaluating diagnostic tests. Ann10 Intern Med 1994, Batista and Hodge in a review conducted years ago 120:667-76. found only three articles pertinent to priority setting for 26. Khan KS, Ter Riet G, Glanville J, Sowden AJ, Kleijnen J: Undertaking Systematic of Research on Effectiveness. clinical practiceReviews guidelines [10]. They suggested theCRD's folGuidance for Carrying Out or Commissioning Reviews: York, Englowing framework for priority setting: land: University of York, NHS Centre for Reviews and Dissemination; 2000. 27.Consult Stroup DF, Berlin JA, users Mortonand SC, etother al.: Meta-analysis of observa1. with end stakeholders before tional studies in epidemiology: a proposal for reporting. selecting topics. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000, 283:2008-12. 28.Consider The AGREE Collaboration: Development and validation of an 2. feasibility during the consultation. international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care the 2003,process 12:18-23.used to select guideline topics. 3. Document 29. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM: Standardized Reporting of Clinical Practice GuideThe IOM the from following procedures [14]: on Guideline lines: suggested A Proposal the Conference Standardization. Ann Intern Med 2003, 139:493-8. 30. Briss PA, Zaza S, Pappaioanou M, et al.: Developing an evidence• thebased use of Delphi-like procedures for obtaining expert guide to community preventive services-methods. Am J Prev Med judgments or2000, topic18(1S):35-43. rankings through correspondence 31. Dobbins M, Thomas H, O'Brien MA, Duggan M: Use of systematic reviews in the development of new provincial public health • thepolicies use of in questions specific, explicit Ontario. that Intl are J Technol Assess Health and Care con2004, 20:399-404. sistent with standard methods for questionnaire construc32. McMichael C, Waters E, Volmink J: Evidence-based public health: tion what does it offer developing countries? J Public Health 2005, 27:215-21. Chan KS, Morton with SC, Shekelle Systematic reviews evi•33.experimentation more PG: formal procedures to for arrive dence-based management: how to find them and what to do at group with judgments them. Am J Manag Care 2004, 10:806-12. 34. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E: systematic reviews that inform health care manTheyTowards also suggested there is a need to define more naragement and policy-making. J Health Serv Res Policy 2005, rowly10(Suppl and precisely 1):35-48. topics for guideline development. 35. Oxman Fretheim A, Schünemann Improving theorganUse of They arguedAD, that this would result inHJ:more efficient Research Evidence in Guideline Development: 7. Deciding ization panelstoand theirHealth work, resolution whatofevidence include. Res Policy Syst 2006.of some 36. Egger M, Juni P, Bartlett C,more et al.: How important are comprehenapparent controversies, responsive guidelines, and literature searches and the assessment of trial quality in easiersive implementation. systematic reviews? Empirical study. Health Technology Assessment 2003, 7(1): [http://www.ncchta.org/execsumm/summ701.htm].

For this review we searched PubMed using (clinical pracAuthors' contributions ADO preparedor the first draft this review.and HJS(priority and AF tice guidelines public healthofguidelines) contributed to drafting andand revising it.articles for selected setting or setting priorities) related references [9,10]. We searched the Cochrane Methodology Register using priority or priorities. We reviewed the Acknowledgements We gratefully acknowledge WHO and theConference Norwegian Knowledge Centre on Priorities website of the 5th International forthe Health Services financial support, that the other authors these in Health Care [11]forand references we had in of our files reviews for their contributions, the other members of the ACHR SURE, [12-15]. and all of those who have provided us with feedback on earlier versions of this paper and the others in this series.

Findings What criteria should be used to establish priorities? References

The US Institute of Medicine's (IOM)reviews. Committee on 1. Mulrow CD: Rationale for systematic BMJ 1994, 309:597-9. Methods for Setting Priorities for Guidelines Develop2. Mallett Clarke of M: How many Cochrane needed ment in itsS,study setting priorities forreviews clinicalarepractice to cover existing evidence on the effects of health care interguidelines published 1995 argued that the priority setventions? ACP J Clubin 2003, 139(1):A11. 3. process Chinnock should P, Siegfried Clarke M: defensible Is evidence-based medicine ting beN, open and [14] They recrelevant to the developing world? Systematic reviews have ommended six general criteria: prevalence, burden of yet to achieve their potential as a resource for practitioners illness, cost of managing variability in developing countries.the Evidproblem, Based Complement Alternat in Medprac2005, 2:321-4. tice, potential of a guideline to improve health outcomes, 4. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of and potential of a guideline to reduce costs. Because data Research Evidence in Guideline Development: 2. Priority Resjudgements Policy Syst 2006.is often lacking, they sugused setting. to makeHealth these 5. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of gested explicit opportunities for important unmeasured Research Evidence in Guideline Development: 14. Reporting factors to be considered. They suggested separate guidelines. Health Res Policy Syst further 2006. 6. Panisett A review of WHO published in "tracks" forU:considering issuesrecommendations for specific populations, 2005. . conditions or concerns. TheyA:argued the rationales forin 7. Oxman AD, Lavis J, Fretheim The usethat of research evidence WHO recommendations. . designating special tracks should be defined clearly; i.e. 8. Moynihan R, Oxman AD, Lavis JN, Paulsen E: Evidence-Informed why they warrant special consideration. They suggested Health Policy: Using Research to Make Health Systems that updating should also be considered as a Healthier. of In guidelines A review of organizations that support the use of research evidence in developing guidelines, technology assessments, andthe health polseparate "track", taking account of issues such as need icy, for the WHO Advisory Committee on Health Research Oslo: Norwefor corrections and the availability of new 2006. evidence. gian Knowledge Centre for the Health Services; 9.

Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evi-

dence-based clinical international survey of 18 Oortwijn identified 25practice: criteria an used to prioritise health clinical guideline programs. Int J Qual Health Care 2003, 15:31-45. technology assessments and categorised these into four 10. Reviewing and grading the evidence NICE: Guideline Development Methods. National Institute for Clinical Excellence, potential February 2004effects, (updated broad categories: burden of disease, March 2005) [http://www.nice.org.uk/pdf/ potential costs, and uncertainty regarding application of GDM_Chapter7_0305.pdf]. the technology [15]. 11. National Institute for Clinical Excellence: The Guideline Development Process: An Overview for Stakeholders, the Public and the NHS 2004 [http://www.nice.org.uk/pdf/

In a GDP_An_Overview_for_Stakeholders_the_Public_and_the_NHS.pd more recent selective review for the New Zealand f]. Guidelines Group, the following criteria were identified as 12. Harris RP, Helfand M, Woolf SH, Lohr KN, CD, Teutsch SM, indicating that a topic is suitable for Mulrow guideline developAtkins D, for the Methods Word Group, third U.S. Preventive Servmentices [16]: Task Force: Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prev Med 2001,

20(3S21-35 [http://www.elsevier.com/locate/ajpmonline]. 1. topic is clinically important affecting large for numbers of 13.The Glanville J, Sowden AJ: Identification of the need a review. people[http://www.york.ac.uk/inst/crd/pdf/crd4_ph0.pdf]. with substantial morbidity or mortality (the burden of 14. Oxman AD, Fretheim A, Schünemann HJ: Improving the use of illness).

research evidence in guideline development: introduction. Health Res Policy Syst 2006. 15.The Cochrane Methodology 2. topic is complex enoughRegister to initiate[http://www.cochrane.org/ debate about the recaccess_data/cmr/accessDB_cmr.asp] ommendations. 16. National Guideline Clearinghouse [http://www.guidelines.gov]

Page Page94ofof10 7 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:20 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/20

37.

38. 39.

40. 41.

42. 43.

44. 45.

46. 47.

48. 49. 50. 51.

52. 53.

54. 55. 56. 57.

58. 59.

Gartlehner G, West SL, Lohr KN, Kahwati L, Johnson JG, Harris RP, et al.: Assessing the need to update prevention guidelines: a comparison of two methods. Int J Qual Health Care 2004, 16:399-406. Royle P, Milne R: Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care 2003, 19:591-603. Royle P, Waugh N: Literature searching for clinical and costeffectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technology Assessment 2003, 7(34): [http://www.ncchta.org/execsumm/summ734.htm]. Beahler CC, Sundheim JJ, Trapp NI: Information retrieval in systematic reviews: challenges in the public health arena. Am J Prev Med 2000, 18(4 Suppl):6-10. Ogilvie D, Hamilton V, Egan M, Petticrew M: Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health 2005, 59:804-8. Alpi KM: Expert searching in public health. J Med Libr Assoc 2005, 93:97-103. French SD, McDonald S, McKenzie JE, Green SE: Investing in updating: how do conclusions change when Cochrane systematic reviews are updated? BMC Medical Research Methodology 2005, 5:33. Bastian H, Doust J: When does an updated meta-analysis have enough content to justify re-reading? [abstract]. Barceona: XI Cochrane Colloquium. October 26–31, 2003. Johnston ME, Brouwers MC, Browman GP: Keeping cancer guidelines current: results of a comprehensive prospective literature monitoring strategy for twenty clinical practice guidelines. Int J Technol Assess Health Care 2003, 19:644-55. Barrowman NJ, Fang M, Sampson M, Moher D: Identifying null meta-analyses that are ripe for updating. BMC Medical research Methodology 2003, 3:1. Shekelle PG, Ortiz E, Rhodes S, Morton SC, Eccles MP, Grimshaw JM, Woolf SH: Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines. How quickly do guidelines become outdated? JAMA 2001, 286:1461-7. Stevens A, Colin-Jones D, Gabbay J: 'Quick and clean': authoritative health technology assessment for local health care contracting. Health Trends 1995, 27:37-42. Hailey D, COrabian P, Harstall C, Schneider W: The use and impact of rapid health technology assessments. Int J Technol Assess Health Care 2000, 16:651-6. Perleth M, Luhmann D, Givis B, Droste S: Rapid Assessments – quick evaluation of medical technology. Gesundheitswesen 2001, 63(Suppl 1):S79-84. Canadian Coordinating Office for Health Technoloty Assessment: Rapid response services in health technology assessment: information from INAHTA members and additional information. Ottawa: CCOHTA; 2004. Milne R, Clegg A, Stevens A: HTA responses and the classic HTA report. J Public Health Med 2003, 25:102-6. Piehl JH, Green S, Silagy C: Training practitioners in preparing systematic reviews: a cross-sectional survey of participants in the Australasian Cochrane Centre training program. BMC Health Services Research 2002, 2:11. GRADE Working Group: Grading quality of evidence and strength of recommendations. BMJ 2004, 328:1490. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 9. Grading evidence and recommendations. Health Res Policy Syst 2006. Vist G: Summaries of findings in Cochrane reviews. The Cochrane Collaboration Methods Groups Newsletter 2005, 9:5-6 [http:// www.cochrane.org/newslett/MGNews_June_2005.pdf]. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999, 354:1896-900. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include. Health Res Policy Syst 2006. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 10. Integrat-

60. 61. 62.

63.

64. 65.

ing values and consumer involvement. Health Res Policy Syst 2006. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 5. Group processes. Health Res Policy Syst 2006. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, et al.: Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1988, 2:. Edejer TTT: Improving the Use of Research Evidence in Guideline Development: 11. Incorporating considerations of cost-effectiveness, affordability and resource implications. Health Res Policy Syst 2006. Guyatt G, Yayward R, Richardson WS, Green L, Wilson M, Sinclair J, et al.: Moving from evidence to action. In Users' Guides to the Medical Literature Edited by: Guyatt G, Rennie D. Chicago: AMA Press; 2002:175-99. Agency for Healthcare Research and Quality. Updating Systematic Reviews. Expected Release Date: early 2007 [http:// www.ahrq.gov/clinic/tp/sysrevtp.htm] Moher D: When and How to Update Systematic Reviews: A Methodological Systematic Review and International Conference. Canadian Coordinating Office for Health Technology Assessment. 2005 HTA Capacity Building Grants Program. [https://www.ccohta.ca/HTA/ hta_grants_2005_descriptions_e.cfm].

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 10 of 10 (page number not for citation purposes)

Health Research Policy and Systems Health Research Policy and Systems 2006, 4:14

http://www.health-policy-systems.com/content/4/1/14 BioMed Central

and the Guidelines International Network [8]) for existing 3. There is evidence of variation between actual and appropriOpen Access Review systematic reviews and relevant methodological research ate care. that address these questions. The answers to the questions are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. sideration of what WHO and other organisations are 1, Atle Fretheim25.and 2 Holger J Schünemann* Oxman doing, and logical arguments. There Andrew is an adequateD amount of existing evidence available.

Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations

For this 1review we searched PubMed using (clinical prac6. The recommendations will be acceptable to the potential Address: INFORMA/CLARITY Research Group, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and tice guidelines or public health guidelines) and (priority users.plass, N-0130 Oslo, Norway 2Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs setting or setting priorities) and related articles for selected Email: Holger J Schünemann* - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@kunnskapssenteret.no; Andrew D Oxman - oxman@online.no references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhaus* Corresponding author ogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical website of the 5th International Conference on Priorities change are not so high that they cannot be overcome. in Health Care [11] and references that we had in our files [12-15]. While burden of2006 disease is commonly used as a criterion Published: 05 December 2006 Received: 07 April Accepted: 05setting, December 2006 for priority it should be noted that the use of sumHealth Research Policy and Systems 2006, 4:21 doi:10.1186/1478-4505-4-21 mary burden of disease measures, such as disability Findings This article is available from: http://www.health-policy-systems.com/content/4/1/21 What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing © 2006 al; licensee BioMed (IOM) Central Ltd. The USSchünemann Instituteet of Medicine's Committee on on disease rather than resource use and interventions, This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). ting process should be open and defensible [14] They recWhat processes should be used to agree on priorities? ommended Abstract six general criteria: prevalence, burden of Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracBackground: The World Health Organization (WHO), like many other organisations around the world, has recognised the found only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, need to use more rigorous processes to ensure that health care recommendations are informed by the best available research clinical practice guidelines [10]. They suggested the foland potential of a guideline to reduce costs. Because data evidence. This is the ninth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory lowing framework for priority setting: used to make these judgements is often lacking, they sugCommittee on Health Research to WHO on how to achieve this. gested explicit opportunities for important unmeasured Objectives: reviewedThey the literature on gradingseparate evidence and1.recommendations in users guidelines. Consult with end and other stakeholders before factors to be We considered. further suggested selecting topics. "tracks" for considering issues for specific populations, Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant conditions or concerns. the rationales for methodological research.They We argued did notthat conduct a full systematic review ourselves. Our conclusions are based on the available 2. Consider during the consultation. designating special tracks should be defined clearly; i.e. evidence, consideration of what WHO and other organisations are doing and feasibility logical arguments. why they warrant special consideration. They suggested Keyupdating questions answers: Should grade the of evidence the strength of recommendations? 3. Document theand process used to select guideline topics. that ofand guidelines should alsoWHO be considered as aquality separate taking accountneed of issues such how as themuch needconfidence they can place in the underlying evidence and the • Users "track", of recommendations to know IOM suggested the complex following procedures [14]: for corrections andThe thedegree availability of new evidence. recommendations. of confidence depends on a numberThe of factors and requires judgments. These judgments should be made explicitly in WHO recommendations. A systematic and explicit approach to making judgments about the quality • theerrors, use offacilitate Delphi-like procedures obtaining expert Oortwijn 25 criteria used to prioritise of evidenceidentified and the strength of recommendations can health help to prevent critical appraisal offor these judgments, and judgments or topic rankings through correspondence technology assessments and categorised these into four can help to improve communication of this information. broad categories: burden of disease, potential effects, What criteria should be used to grade evidence and recommendations? • the use of questions that are specific, explicit and conpotential costs, and uncertainty regarding application of sistent withbestandard methods forused questionnaire constructhe technology [15]. • Both the quality of evidence and the strength of recommendations should graded. The criteria to grade the strength of recommendations should include the quality of the underlying tion evidence, but should not be limited to that. In a more recent selective review for the New Zealand • The approach to grading should be one that has wide international support and is suitable for a wide range of different types • experimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as of recommendations. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, which at group judgments indicating that a topic is suitable for guideline developis currently suggested in the Guidelines for WHO Guidelines, is being used by an increasing number of other organizations ment [16]: internationally. It should be used more consistently by WHO. Further developments of this approach should ensure its wide They also suggested there is a need to define more narapplicability. rowly and precisely topics for guideline development. 1. The topic is clinically important affecting large numbers of Should WHO use the same grading system all ofofits recommendations? They argued that this would result in more efficient organpeople with substantial morbidity or mortality (the for burden ization of panels their resolution of some illness). • Although there are arguments for and against using the same grading system across and a wide rangework, of different types of controversies, more responsive and recommendations, WHO should use a uniform grading system toapparent prevent confusion for developers and usersguidelines, of easier implementation. 2. The topic is complex enough to initiate debate about the recrecommendations. ommendations.

Page 41 of of 77 Page (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:21 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/21

Background

â&#x20AC;˘ What criteria should be used to grade evidence and recommendations?

â&#x20AC;˘ Should WHO use the same grading system for all of its recommendations? Questions related to what evidence should be included, how it should be synthesized and reported are addressed in other papers in this series [3-5].

What WHO is doing now?

For over 25 years a growing number of organisations have employed various systems to grade the quality of evidence (sometimes called levels of evidence) and the strength of recommendations [1]. Unfortunately, different organisations use various grading systems, which may lead to confusion among consumers.

WHO groups (e.g. WHO Europe) have acknowledged the need for evaluating or developing a grading system [6,7], and the Guidelines for WHO Guidelines recommend using a specific, uniform grading system [8]. However, this system, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, has scarcely been used within WHO [9,10]. Some WHO groups have developed their own grading systems [11,12], despite of the guidelines for WHO guidelines suggestion to use GRADE. Most have not explicitly graded either the quality of evidence or the strength of recommendations [13,14].

Groups making recommendations always make judgements about the quality of evidence and the balance of benefits and downsides (harms, burden and costs). Frequently these judgements are made implicitly rather than explicitly and judgements about the quality of evidence are confused with judgements about the balance of benefits and downsides. Many systems that are used to grade the quality of evidence and the strength of recommendations also confuse these judgements by equating the strength of recommendation with the quality of evidence, for example by grading recommendations for which there is high quality evidence as strong, without explicitly considering the balance of benefits and downsides.

What other organisations are doing

Most, but not all organizations that develop guidelines use a grading system to express the strength of a recommendation or the quality of evidence. For example, the US Preventive Services Task Force (USPSTF) uses a grading system that assigns one of three grades of evidence: good, fair, or poor [15]. The Task Force uses its assessment of the evidence and magnitude of net benefit to make a recommendation, coded as a letter: from A (strongly recommended) to D (recommend against). The UK National Institute for Health and Clinical Excellence (NICE) has not yet made a decision as to which grading system to use [16]. The Scottish Intercollegiate Guideline Network (SIGN) has developed its own grading system for application to SIGN guidelines [17]. The Australian Medical Research Council is currently developing a grading system that will probably include grading recommendations according to strength of recommendations and quality of evidence [18]. The US Task Force on Community Preventive Services uses a system in which the quality of the evidence of effectiveness links directly the strength of the recommendation [19,20]. Professional organizations use a variety of systems, many of them, however, based on two prominent grading approaches: the system derived from the Canadian Task Force on the Periodic Health Examination [1,21] and a successor of that system, the Oxford Centre for Evidence Based Medicine approach [22].

Knowing the quality of evidence is essential, but not sufficient for making judgements about the strength of a recommendation. For instance, high quality evidence from well executed randomized controlled trials showed that oral anticoagulation administered for more than one year reduces the risk for recurrent thromboembolic events in patients after a first episode of spontaneous deep venous thrombosis. However, because oral anticoagulation is associated with harms (bleeding risk), burden (taking medication and monitoring anticoagulation levels) and cost (anticoagulation clinics or monitoring devices) the recommendation to anticoagulate all patients is weak because the benefits and downsides are finely balanced and individual patients will make different choices [2]. Both judgements about the quality of evidence and about the strength of a recommendation are complex and require consideration of a number of factors. In this paper we addressed the following questions: â&#x20AC;˘ Should WHO grade the quality of evidence and the strength of recommendations?

More recently, medical societies have begun to form collaborations within specialties to develop grading systems

Page 2 of 7 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:21

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/21

and on their the Guidelines own. For example International a group Network of specialty [8]) for societies existing in systematic rehabilitation reviews sciences and relevant formed methodological a panel to develop research an that approach address tothese grading questions. the quality The answers of evidence to the and questions strength are of recommendations our conclusions based [23].on This thepanel available developed evidence, a set conof sideration criteria for of grading what the WHO strength and of other bothorganisations the evidence and are doing, the recommendation. and logical arguments. Similarly, the world leading urology associations have come together to adopt a uniform For grading this review systemwe and searched approach PubMed that would using (clinical be useful pracfor tice urologists guidelines around or public the world health rather guidelines) than each and association (priority setting using aordifferent setting priorities) grading system and related [24]. articles This latter for selected collaboreferences ration named, [9,10]. Evidence We searched Based Urology, the Cochrane is exploring Methodolusing ogy the GRADE Register approach. using priority The GRADE or priorities. approach We reviewed is being used the Internationalaround Conference on Priorities website increasingly of the by5th organisations the world [25-28], in Health Care [11]cases and references we had in our filesIt although in some with slightthat modifications [29]. [12-15]. has been used for public health questions such as the pharmacological management of human influenza A(H5N1) infection (avian flu) [30], although it more Findings What criteria should used establish priorities? commonly has been be used forto clinical questions up to now. The US Institute Medicine's (IOM) care Committee on A group of family of practice and primary journals has Methods for Setting Priorities also developed a system to gradefortheGuidelines strength ofDevelopa recomment in its study mendation [31]. of setting priorities for clinical practice guidelines published in 1995 argued that the priority setting process should be open and defensible [14] They recMethods The methodssix used to prepare thisprevalence, review are described in ommended general criteria: burden of the introduction to this series [32]. Briefly, the key quesillness, cost of managing the problem, variability in practionspotential addressed this paper were vetted amongst the tice, of ainguideline to improve health outcomes, authors and of thea guideline ACHR Subcommittee onBecause the Use of and potential to reduce costs. data Research Evidence We did not conduct a fullsugsysused to make these (SURE). judgements is often lacking, they tematicexplicit review.opportunities We reviewed existing guidelines for guidegested for important unmeasured lines totoidentify grading system currently in use.separate We also factors be considered. They further suggested searchedfor PubMed using (grading (methods) "tracks" considering issues forsystem) specificand populations, (MESH headings/keywords) for systematic reviews and conditions or concerns. They argued that the rationales for studies of methods for grading thebe quality of evidence. In designating special tracks should defined clearly; i.e. addition, we searched maintained by the why they warrant special databases consideration. They suggested Agency for Healthcare Research andbeQuality (AHRQ, that updating of guidelines should also considered as a [33]) and the Guidelines International Network separate "track", taking account of issues such as the(GIN, need [34]). These searches supplemented with informafor corrections and the were availability of new evidence. tion obtained directly from guideline development organizations and our own Because involvement Oortwijn identified 25 files. criteria used of to our prioritise health with organizations thatand produce guidelines technology assessments categorised these and into prior four work with gradingburden systems,ofin disease, particularpotential the GRADE sysbroad categories: effects, tem, we had depth knowledge about several systems potential costs,inand uncertainty regarding application of [25,28,29,35,36]. the technology [15].

ualThere studies, predicated on thebetween extent actual to which bias was 3. is evidence of variation and appropriminimized), quantity (magnitude of effect, numbers of ate care. studies, and sample size or power) and consistency (for any given the extent to which available similar findings 4. There aretopic, no existing valid guidelines to use. are reported using similar and different study designs). 5. There is an adequate amount of existing evidence available. More recently, independent work by the Canadian Optimal Prescribing and Utilization Service 6. TheMedication recommendations will be acceptable to the potential (COMPUS) used a detailed process to evaluate and select users. an evidence grading system and expanded the work by AHRQ (while accepting it) until the year 2005 COM7. Implementation of the guideline is feasible, will [38]. not exhausPUS, use which evaluates, promotes and to facilitates tively the identifies, communities' resources, and barriers clinical best practices inhigh drugthat prescribing and among health change are not so they cannot be use overcome. care providers and consumers in Canada [39], is a nationally coordinated program, by Health and While burden of disease is funded commonly used asCanada a criterion delivered the Canadian forofHealth for priorityby setting, it shouldCoordinating be noted thatOffice the use sumTechnology They assemmary burdenAssessment of disease(CCOHTA, measures, [39]). such as disability bled a working group of internal researchers, information adjusted life years (DALYs) has been criticised for focusing specialists, experts, and researchers on disease methodology rather than resource use external and interventions, to update of AHRQ. COMPUS searched in forsuch and because of the the work assumptions about values inherent selected review articlesoffor the period 2000 to 2005. This measures, and because the technical limitations of such resulted in(see more 3,000 citations [17] for selection. Eleven measures forthan example references and [18]). review articles were selected for further analysis based on What processes should be specified used to agree onworking priorities? a priori selection criteria by the group. Batista Hodge in a review conducted 10 years from ago Nearly and 50 evidence grading systems were identified found three articles pertinent priority setting for the 11 only review articles. Canadian andtointernational experts clinical practice guidelines [10]. They helped suggested the folin evidence evaluation methodology identify an lowing framework for priority setting: not included in the additional 10 instruments or systems list of identified grading systems. The identified instru1. Consult endwere users and other stakeholders ments and with systems evaluated using the AHRQbefore evaluselecting topics. ation grids. The highest scoring instruments were the GRADE and the SIGN approach [38]. A second round of 2. Consider feasibility during the consultation. expert consultation and stakeholder input from all interested parties confirmed the selection of these instruments. 3. Document the process used to select guideline topics. The GRADE system was developed through an intensive The IOM suggested the following procedures [14]: international collaboration of methodologists, guideline developers and clinicians and incorporates the factors •identified the use ofinDelphi-like for obtaining the AHRQprocedures review and described expert above judgments [35,36]. or topic rankings through correspondence WHO grade thethat quality of evidence and the •Should the use of questions are specific, explicit and constrength of recommendations? sistent with standard methods for questionnaire construcWe did not identify published studies that compared tion graded with non-graded recommendations. The only eviwe are awarewith of are three unpublished studies. The •dence experimentation more formal procedures to arrive first was judgments conducted by UpToDate®, an electronic textbook, at group that asked a small group of users to compare graded with non-graded recommendations and explore in a focus They also suggested there is a need to define– more nar®, pergroup and setting – reasons for their (UpToDate rowly precisely topics for answers guideline development. sonalargued communication). Theresult second is our efficient own study askThey that this would in more organing a small group ofand the general publicresolution interested of in health ization of panels their work, some care issues (Akl E, et al,more manuscript in preparation). The apparent controversies, responsive guidelines, and third is a study by researchers in Norway who provided easier implementation. patients with back problems with graded evidence of the

In a more recent selective review for the New Zealand Findings We identified one systematic review dealing the evalGuidelines Group, the following criteria werewith identified as uation of that grading systems. In 2002, the US Agency for indicating a topic is suitable for guideline developHealthcare ment [16]: Research and Quality (AHRQ) published a systematic review of existing systems to grade the quality of The evidence strength of recommendations [37]. The 1. topic isand clinically important affecting large numbers of AHRQwith review considered 40 systems until (the the year 2000 people substantial morbidity or mortality burden of that addressed grading the strength of a body of evidence. illness). The important domains and elements for the systems to grade the strength of enough evidence that thedebate authors agreed on 2. The topic is complex to initiate about the recwere quality (the aggregate of quality ratings for individommendations.

Page Page 43 of of 77 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:21 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/21

effects of alternative interventions graded with the GRADE approach. Users of the website intuitively understood the meaning of the quality grades for each outcome (Claire Glenton, personal communication). The findings of these evaluations suggested that users preferred graded over non-graded recommendations.

ings. In addition the GRADE group is supported by an international collaboration [36]. The main limitation and criticism of the GRADE system is its complexity. Work in progress is addressing this limitation including the development of user friendly software to develop evidence profiles (G. Vist, personal communication and [26,29]).

Despite the lack of stronger direct evidence, there is agreement among most guideline developers that grading the quality of evidence has advantages, because health care decisions involve a trade-off between likely benefits on the one hand, and downsides (harms, burden and costs) on the other hand [40]. To integrate these recommendations with their own judgment, guideline users need to understand the basis for the recommendations that guidelines offer them. A systematic approach to grading the strength of recommendations should minimize bias and aid interpretation about benefits and downsides. In addition, a systematic and explicit approach to making judgements about the quality of evidence and the strength of recommendations is likely to help prevent errors, facilitate critical appraisal of these judgements, and can help improve communication of this information [36].

Should WHO use the same grading system for all of its recommendations? We did not identify evidence for or against using a single grading system for all types of recommendations, including clinical, public health and health policy recommendations. The arguments for and against using a single grading system are summarised in Table 3. The most important reasons for using a consistent system are a) minimising confusion amongst users of WHO recommendations; b) the risk of bias if groups can select a system that makes the quality of evidence and the strength of recommendations look better for their preferred interventions; and c) being intellectually honest about recognising the limits of the evidence rather than having a double standard. If an approach can be identified that is suitable across a wide range of interventions and contexts both methodologically and politically, the advantages outweigh the disadvantages.

What criteria should be used to grade evidence and recommendations? In a series of 16 international meetings and correspondence over five years the GRADE Working Group has derived a set of criteria to assess the quality of evidence (Table 1) and the strength of recommendations (Table 2) [25,29,35,36,41,42]. The GRADE system has several advantages over other systems including explicit definitions and sequential judgments during the grading process; a detailed description of the criteria for the quality of evidence for single outcomes and for the overall quality of the evidence; weighing the relative importance of outcomes; consideration of the balance between health benefits versus harms, burdens and cost; and the development of evidence profiles and summaries of find-

Some developers and users of GRADE believe that GRADE can be consistently and usefully applied across clinical and non-clinical interventions, based on conceptual arguments and experience up to now applying this approach to a wide range of interventions, including public health and health system interventions. Others disagree because they believe it is unlikely to be an appropriate approach for some areas for the reasons summarised in table 3. There is not yet an empirical evidence base with which to mediate this disagreement for GRADE or any other grading system. Up to now GRADE has been used mostly for clinical interventions and few examples of its use with public health questions have been published. There is an

Table 1: GRADE quality assessment criteria Quality of evidence

Study design

Lower if *

Higher if *

High

Randomised trial

Study quality: -1 Serious limitations -2 Very serious limitations -1 Important inconsistency Directness: -1 Some uncertainty -2 Major uncertainty -1 Sparse data -1 High probability of Reporting bias

Strong association: +1 Strong, no plausible confounders, consistent and direct evidence** +2 Very strong, no major threats to validity and direct evidence*** +1 Evidence of a Dose response gradient +1 All plausible confounders would have reduced the effect

Moderate Low Very low

Observational study

* 1 = move up or down one grade (for example from high to intermediate) 2 = move up or down two grades (for example from high to low) ** A statistically significant relative risk of >2 (< 0.5), based on consistent evidence from two or more observational studies, with no plausible confounders *** A statistically significant relative risk of > 5 (< 0.2) based on direct evidence with no major threats to validity

Page 4 of 7 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:21

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/21

Table 2: Guidelines Decisions about the strengthNetwork of a recommendation and the International [8]) for existing 3. There is evidence of variation between actual and approprisystematic reviews and relevant methodological research ate care. Factors that can weaken the strength of a recommendation Explanation that address these questions. The answers to the questions are ourquality conclusions based on the available evidence, 4.uncertainty There are nothe existing valid guidelines available to use. Lower evidence Will concreate greater about size of the (relative) effects (benefits and harms) Uncertainty about balance of benefits versus harms and burdens Uncertainty sideration of the what WHO and other organisations are about the baseline risk, prevalence of a problem or health status, which could affect the size of the (absolute) effects doing, andorlogical an adequate amount existing evidence available. Uncertainty differencesarguments. in values Uncertainty about5. theThere relativeisimportance of the benefits and of downsides to those affected, or differences in how important they are to different people, which could affect the balance between the benefits versus harms and burden

For thisnet review searched PubMed using (clinical 6.benefits The recommendations willuncertain) be acceptable to the potential Marginal benefits we or downsides Thepracanticipated net or downsides are small (and tice guidelines or public guidelines) users. Uncertainty about whether the net health benefits are worth the costsand (priority Uncertainty related to lack of information about the cost or whether the resource expenditure is justified by the anticipated benefit setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhausongoing international collaborative effort to apply the the ogy Register using priority or priorities. We reviewed Moreuse recently, however, the WHO rapid advice to guideline tively the communities' resources, and barriers clinical Conference on Priorities GRADE ofapproach to public health and health systems website the 5th International panel are for not thesopharmacological management of human change high that they cannot be overcome. in Health Care and [11]itand references that we had in our files interventions, is possible that modifications may be infection with avian influenza A (H5N1) virus applied [12-15]. needed to ensure its usefulness for non-clinical intervenGRADE successfully [30]isand several WHO are While burden of disease commonly used guidelines as a criterion tions. For example, in one recent review of drug policies under development using be GRADE Hill, for priority setting, it should noted (Sue that the usepersonal of sumthe authors felt that it was important to distinguish communication). general, the evidence recmary burden of In disease measures, suchthat as graded disability Findings What criteria shouldtypes be used to establish priorities? between different of observational studies (interommendations advantages over non-graded recomadjusted life yearshave (DALYs) has been criticised for focusing The US time-series Institute ofanalyses Medicine's Committee on rupted and (IOM) controlled before-after mendations is limited, but thereuse are and strong arguments, on disease rather than resource interventions, Methods for Setting Priorities for Guidelines Developstudies) when making judgements about the quality of including clear and transparent communication of because of the assumptions about values inherent in such ment in its of setting priorities evidence forstudy important outcomes [43].for clinical practice how much confidence can place in recommendameasures, and because ofusers the technical limitations of such guidelines published in 1995 argued that the priority settions and(see the for evidence underlying them. limitameasures example references [17] Another and [18]). ting process should be open and defensible [14] They rection is that both the quality of evidence and the strength Discussion What processes should exist be used agree on priorities? WHO has made a decision to use a gradingburden system of to ommended six general criteria: prevalence, of recommendations on to a continuum. CategorizaBatista Hodge in a categories review conducted 10 years ago grade the of evidence and strength of recommenillness, costquality of managing the problem, variability in praction of and quality into four and recommendations found only three articlesinto pertinent to priority for dations that is and being health used widely, the tice, potential of asensible guideline to is improve outcomes, for or against treatments two grades, strongsetting and weak, clinical practice guidelines [10]. They suggested the folGRADE system WHOtohas been involved the and potential of a[36]. guideline reduce costs. Becauseindata may oversimplify complex health care recommendations, lowing framework for priority development of thisjudgements system from beginning, and sugconused to make these is the often lacking, they but guidelines consumers aresetting: generally likely to benefit sideration has been given to the application gested explicit opportunities for potential importantforunmeasured from this simplification as they are most interested in 1. Consult with end userstoand other stakeholders before of the tosystem to WHOThey guidelines developing the factors be considered. further in suggested separate which recommendations follow. selecting topics. GRADE approach. This might expected to facil"tracks" for considering issueshave for been specific populations, itate the dissemination and adoption of the thisrationales approachfor by conditions or concerns. They argued that Further work We have found a largeduring body of on the development 2. Consider feasibility thework consultation. WHO guideline developers. However, interestclearly; in GRADE designating special tracks should be defined i.e. and evaluation of various grading systems. Problems have workshops at WHO has been limited, there is not a tradiwhy they warrant special consideration. They suggested arisen because grading systems exist. 3. Document the many processdifferent used to select guideline topics. tionupdating of grading quality should of evidence or considered strength of as recthat of the guidelines also be a Future efforts should focus on forging a consensus on ommendations at WHO, and of few resources been separate "track", taking account issues such ashave the need using a sensible andthe uniform approach to grade the qualThe IOM suggested following procedures [14]: invested in supporting use of GRADE for corrections and the the availability of newspecifically, evidence. or in ity of evidence and strength of recommendations, buildsupporting more rigorous guidelines development meththeofwork of the GRADE working Useexpert of the •ing theonuse Delphi-like procedures for group. obtaining ods generally. Oortwijn identified 25 criteria used to prioritise health judgments or topic rankings through correspondence technology assessments and categorised these into four Table 3: Pros and cons of using the same system for grading evidence and formulating recommendations for a wide range of health broad categories: burden of disease, potential effects, care interventions, including clinical and non-clinical interventions • the use of questions that are specific, explicit and conpotential costs, and uncertainty regarding application of Arguments for having a common approach Arguments against having a methods common approach sistent with standard for questionnaire constructhe technology [15]. tion • Having an infeasible system for some kinds of questions might result in less recent demanding systems for some for kindsthe of questions might In• Having a more selective review New Zealand negative conclusions. result in false positive conclusions. • experimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as false • False negative conclusions due to inappropriate evaluation • People with vested interests in particular interventions could choose at group judgments indicating that a topic is suitable for guideline developrequirements may have negative political and health consequences; for the system that makes their intervention look best. ment [16]: • People with vested interests in particular evaluation approaches could example, effective programs that cannot be studied with randomised funding cuts. choose the system that makes their preferred evaluation approach look trials Theymight alsoexperience suggested there is a need to define more nar• Interventions that cannot be studied with randomised trials might not best. rowly and precisely topics for guideline development. 1.• Having The topic is clinically important affecting large numbers of be evaluated. different systems for different types of interventions might be argued would result in more efficient organpeople with substantial morbidity or mortality (the burden of •They A single systemthat may this not discriminate adequately within the range of confusing. appropriate considerwork, for clinical and non-clinical • It is intellectually honest to recognise the limits of evidence where this evidence izationthat of ispanels andto their resolution of some illness). interventions. is appropriate. apparent controversies, more responsive guidelines, and • A system that can adequately address evidence across a wide range of • Admitting the limitations of evidence, if this is appropriate, might easier implementation. 2.promote The topic is complex to initiate debate about the rec- interventions and contexts may be overly complex. more and betterenough research. ommendations.

Page Page 45 of of 77 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

the 2006, use of4:21 research evidence in guideline development Health Research PolicyImproving and Systems http://www.health-policy-systems.com/content/4/1/21

GRADE system by WHO, as is currently recommended by the Guidelines for WHO Guidelines, could help by obtaining more experience, particularly with non-clinical interventions, contribute to improvements in the existing system, contribute to agreeing on a common international approach to grading of recommendations and help to ensure the quality and transparency of the judgements that are made across various groups that make recommendations on behalf of WHO. Development of software and a detailed manual to simplify the use of the GRADE system is underway and should facilitate the use of this system and its further development.

Competing interests

12.

10.

11.

13. 14. 15.

16. 17. 18.

Authors' contributions

19.

HJS prepared the first draft of this review. AF and ADO authors contributed to drafting and revising it. All authors read and approved the final manuscript 20.

21. 22.

References 1. 2.

3. 4. 5. 6. 7.

23.

Anonymous: Canadian Task Force on the Periodic Health Examination. The periodic health examination. CMAJ 1979, 121:1193-1254. Buller HR, Agnelli G, Hull RD, Hyers TM, Prins MH, Raskob GE: Antithrombotic therapy for venous thromboembolic disease: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004, 126(3 Suppl):401S-428S. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include. Health Res Policy Syst 2006. Oxman AD Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 8. Synthesis and presentation of evidence. Health Res Policy Syst 2006. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 14. Reporting guidelines. Health Res Policy Syst 2006. The Russian Federation. Making Pregnancy Safer/Promoting Effective Perinatal Care. National Orientation and Planning Meeting. Moscow, 11-14 February 2003. . Fourth Futures Forum of High-Level Decision-Makers Tools for decision-making in public health. Brussels, Belgium 16– 17 June 2003. .

24. 25. 26. 27. 28.

29.

Global Programme on Evidence for Health Policy. Guidelines for WHO Guidelines. EIP/GPE/EQC/2003.1. Geneva , World Health Organization; 2003. Schünemann HJ Hill S, Kakad M, Bellamy R, Uyeki T, Hayden F, Yazdanpanah Y, Beigel J, Chotpitayasunondh T, Vist G, Del Mar C, Farrar J, Fukuda K, Hien TT, Shindo N, Stockman L, Sugaya N, Tomson G, Nagjdaliyev A, Özbay B, Roth C, Croisier A, Oxman, AD: WHO Rapid Advice Guidelines for the pharmacological management of human infection with avian influenza A (H5N1) virus. Lancet companion paper:. submitted Schünemann HJ Hill S, Kakad M, Vist G, Bellamy R, Stockman L,Uyeki T, Del Mar C, Shindo N, Fukuda K, Hayden F, Yazdanpanah Y,Oxman, AD: WHO Rapid Advice Guidelines: Quick and Transparent. Lancet companion paper:. submitted World Health Organization. Prevention of Recurrent Heart Attacks and Strokes in Low and Middle Income Populations: Evidence-based Recommendations for Policy Makers and Health Professionals. Geneva , World Health Organization; 2003. Guidelines for the Prevention of Malaria . Geneva , World Health Organization; 2006. Oxman AD Lavis J, Fretheim A: The use of research evidence in WHO recommendations. . (submitted) Panisset U: A Review of WHO recommendations published in 2005. Health Res Policy Syst 2006. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, Atkins D: Current methods of the U.S. Preventive Services Task Force: A review of the process. American Journal of Preventive Medicine 2001, 20(3, Supplement 1):21-35. National Instutites for Health and Clinical Excellence [http:/ /www.nice.org.uk/pdf/GDM_Allchapters_0305.pdf] Harbour R Miller J: A new system for grading recommendations in evidence based guidelines. BMJ 2001, 323:334-336. Middleton P TR Salisbury J, Coleman K, Norris S, Grimmer K, Hillier S: Assessing the body of evidence and grading recommendations in evidence-based clincal practice guidelines: Melbourne, Australia. Edited by: Collaboration C. ; 2005:42. Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, Truman BI, Hopkins DP, Mullen PD, Thompson RS, Woolf SH, CarandeKulis VG, Anderson L, Hinman AR, McQueen DV, Teutsch SM, Harris JR: Developing an evidence-based Guide to Community Preventive Services--methods. The Task Force on Community Preventive Services. Am J Prev Med 2000, 18(1 Suppl):35-43. Truman BI, Smith-Akin CK, Hinman AR, Gebbie KM, Brownson R, Novick LF, Lawrence RS, Pappaioanou M, Fielding J, Evans CA Jr., Guerra FA, Vogel-Taylor M, Mahan CS, Fullilove M, Zaza S: Developing the Guide to Community Preventive Services--overview and rationale. The Task Force on Community Preventive Services. Am J Prev Med 2000, 18(1 Suppl):18-26. Sackett DL: Rules of evidence and clinical recommendations on the use of antithrombotic agents. Archives Int Med 1986, 146:464-465. Phillips B Ball C, Sackett D, Badenoch D, Straus S, Haynes B, Dawes M.: Levels of Evidence and Grades of Recommendation. Centre for Evidence-Based Medicine. . Philadelphia Panel evidence-based clinical practice guidelines on selected rehabilitation interventions: overview and methodology. Phys Ther 2001, 81(10):1629-1640. Evidence Based Urology Task Force [http://www.gradework inggroup.org/EBUro/intro.htm] Guyatt G, Vist G, Falck-Ytter Y, Kunz R, Magrini N, Schunemann H: An emerging consensus on grading recommendations? (Editorial). ACP J Club 2006, 144(1):A08. GRADE Working Group [http://www.gradeworkinggroup.org] Researchers test system to custom-tailor guidelines. Clin Resour Manag 2002, 3(1):8-10, 1. Schünemann HJ Jaeschke J, Cook D, Bria WF, El Solh AA, Ernst A, Fahy BF, Gould MK, Horan KL, Krishnan JA, Manthous CA, Maurer J, McNicholas W, Rubenfeld G, Turino GM, Guyatt G (on behalf of the ATS Documents Development and Implementation Committee): An official ATS statement: Grading the quality of evidence and strength of recommendations for guidelines and recommendations. American Journal of Respiratory and Critical Care Medicine 2006, 174(5):605-614. Guyatt G Gutterman D, Baumann M, Addrizzo-Harris D, Hylek E, Phillips B, Raskob G, Lewis S, Schünemann H.: Grading strength of

Page 6 of 7 (page number not for citation purposes)

Health Health Research Research Policy Policy and and Systems Systems 2006, 2006, 4:14 4:21

http://www.health-policy-systems.com/content/4/1/14 http://www.health-policy-systems.com/content/4/1/21

recommendations and quality of evidence[8]) in clinical guideand the Guidelines International Network for existing lines. Chest 2006, 129:174-181. systematic reviews and relevant methodological research 30. WHO Rapid Advice Guidelines on pharmacological managethat address questions. to the questions ment of these humans infected The withanswers avian influenza A (H5N1) virus [http://www.who.int/csr/disease/avian_influenza/guidelines/ are our conclusions based on the available evidence, conen/index.html] sideration what WHO and other organisations are 31. Ebell MH,ofSiwek J, Weiss BD, Woolf SH, Susman J, Ewigman B, Bowman M: logical Strengtharguments. of recommendation taxonomy (SORT): a doing, and

3. There is evidence of variation between actual and appropriate care. 4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available.

patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract 2004, 17(1):59-67. For this review we searched PubMed (clinical prac32. Oxman AD Fretheim A, Schünemann HJ:using Improving the Use of Research Evidence in Guideline Development: Introduction. tice guidelines or public health guidelines) and (priority Health Res Policy Syst 2006. setting or setting priorities) and relatedand articles for selected 33. Agency for Healthcare Research Quality [http:// references [9,10]. We searched the Cochrane Methodolwww.guidelines.gov] 34. Register Guidelines International [http://www.g-i-n.net] ogy using priority Network or priorities. We reviewed the 35. Atkins D, EcclesthM, Flottorp S, Guyatt GH, Henry D, Hill S, Liberati International Conference onH,Priorities website of the 5 A, O'Connell D, Oxman AD, Phillips B, Schunemann Edejer TT, Vist GE, Williams Jr.: references Systems forthat grading the quality eviin Health Care [11]JW and we had in ouroffiles dence and the strength of recommendations I: critical [12-15]. appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res 2004, 4(1):38. 36. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt Findings GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, LibWhaterati criteria should be used to establish priorities? A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, AD, Phillips Schunemann HJ, Edejer TT, Varonen H, on Vist The Oxman US Institute ofB,Medicine's (IOM) Committee GE, Williams JW Jr., Priorities Zaza S: Grading quality of evidence and Methods for Setting for Guidelines Developstrength of recommendations. Bmj 2004, 328(7454):1490. ment in its study of setting clinical practice 37. West S, King V, Carey TS, Lohrpriorities KN, McKoyfor N, Sutton SF, Lux L: Systems topublished Rate the Strength Scientific Evidence. In Evidence guidelines in 1995ofargued that the priority setReport/Technology Assessment Volume AHRQ Publication No. 02-E016. ting process be open and defensible [14] They2002. recRockville, should MD , Agency for Healthcare Research and Quality; ommended general criteria: Prescribing prevalence, of 38. Canadiansix Optimal Medication andburden Utilization Service [https://www.ccohta.ca/compus/compus_pdfs/ illness, cost of managing the problem, variability in pracCOMPUS_Evaluation_Methodology_draft_e.pdf] tice, a guideline Office to improve health outcomes, 39. potential Canadian of Coordinating for Health Technology Assessment and potential of a[https://www.ccohta.ca/] guideline to reduce costs. Because data 40. Schünemann HJ Fretheim A, Oxman, AD: Improving of used to make these judgements is often lacking, the theyUse sugResearch Evidence in Guideline Development: 6. Determingested opportunities for important ingexplicit which outcomes are important. Health Resunmeasured Policy Syst 2006. 41. Atkins PA, Eccles M, Flottorp S, Guyatt GH, Harbour RT, Hill factors to D, beBriss considered. They further suggested separate S, Jaeschke R, Liberati A, Magrini N, Mason J, O'Connell D, Oxman "tracks" for considering issues for specific populations, AD, Phillips B, Schunemann H, Edejer TT, Vist GE, Williams JW Jr.: conditions orfor concerns. They argued that theand rationales for Systems grading the quality of evidence the strength of recommendations II: should pilot study a new system. designating special tracks be ofdefined clearly; BMC i.e. Health Serv Res 2005, 5(1):25. why they warrant special consideration. They suggested 42. Schünemann HJ, Best D, Vist G, Oxman AD: Letters, numbers, symbols and how to communicate of evidence that updating ofwords: guidelines should also begrades considered as a and recommendations. Cmaj 2003, 169(7):677-680. separate "track", taking account of issues such as the need 43. Aaserud M Dahlgren A, Kosters JP, Oxman AD, Ramsay C, Sturm H.: for corrections and the availability ofpricing new evidence. Pharmaceutical policies: effects of and purchasing policies on rational drug use. In The Cochrane Library Chichester, UK , John Wiley & Sons, Ltd; 2006.

In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]:

disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page Page 47 of of 77 (page (pagenumber numbernot notfor forcitation citationpurposes) purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement Holger J Schünemann*1, Atle Fretheim2 and Andrew D Oxman2 Address: 1INFORMA/CLARITY Research Group, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 2Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway Email: Holger J Schünemann* - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@kunnskapssenteret.no ; Andrew D Oxman - oxman@online.no * Corresponding author

Published: 05 December 2006 Health Research Policy and Systems 2006, 4:22

Received: 07 April 2006 Accepted: 05 December 2006

doi:10.1186/1478-4505-4-22

This article is available from: http://www.health-policy-systems.com/content/4/1/22 © 2006 Schünemann et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract Background: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 10th of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. Objectives: We reviewed the literature on integrating values and consumers in guideline development. Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We reviewed the titles of all citations and retrieved abstracts and full text articles if the citations appeared relevant to the topic. We checked the reference lists of articles relevant to the questions and used snowballing as a technique to obtain additional information. We did not conduct a full systematic review ourselves. Our conclusions based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key questions and answers: We did not find a systematic review of methods for integrating values in guidelines, but we found several systematic reviews that dealt with related topics. Whose values should WHO use when making recommendations? • Values, the relative importance or worth of a state or consequences of a decision (outcomes relating to benefits, harms, burden and costs), play a role in every recommendation. Ethical considerations, concepts that determine what is right, also play a role. • The values used in making recommendations should reflect those of the people affected. Judgements should be explicit and should be informed by input from those affected (including citizens, patients, clinicians and policy makers). • When differences in values may lead to different decisions or there is uncertainty about values, this should also be explicit. If differences in values are likely to affect a decision, such that people in different setting would likely make different choices about interventions or actions based on differences in their values, global recommendations should be explicit in terms of which values were applied and allow for adaptation after incorporating local values. How should WHO ensure that appropriate values are integrated in recommendations? • All WHO guideline groups should uniformly apply explicit, transparent and clearly described methods for integrating values. • WHO should consider involving relevant stakeholders if this is feasible and efficient.

Page 1 of 8 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:22

http://www.health-policy-systems.com/content/4/1/22 http://www.health-policy-systems.com/content/4/1/14

and• the WHO Guidelines should develop International a checklist Network for guidelines [8]) for existing panels to help them to ensure that ethical considerations to 3. There is evidence of variation between actualrelevant and approprisystematic recommendations reviews and arerelevant addressed methodological explicitly and transparently. research ate care. that address these questions. The answers to the questions How should users and consumers be involved in generating recommendations? are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. • Including in groups that are making globalare recommendations presents major challenges with respect to the sideration of consumers what WHO and other organisations impossibility of including a representative spectrum of consumers fromisaan variety of cultures settings.evidence Nonetheless, doing, and logical arguments. 5. There adequate amountand of existing available. consideration should be given to including consumers in groups who are able to challenge assumptions that are made the values used for making recommendations, rather than the values of will consumers aroundtothethe world. Forabout this review we searched PubMed using (clinical prac6. represent The recommendations be acceptable potential tice•guidelines or public health guidelines) and (priority users. WHO should establish a network to facilitate involvement of users. setting or setting priorities) and related articles for selected • Draft recommendations should reviewed Methodolby consumers, who should be asked to consider thewill values references [9,10]. We searched thebeCochrane 7. Implementation of explicitly the guideline is feasible, not that exhauswere used. ogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical International Conference on Priorities website the 5thvalues change are not so high that they cannot be overcome. Howof should be presented in recommendations? in Health Care [11] and references that we had in our files • Recommendations should include a description of how decisions were made about the relative importance of the [12-15]. While burden of disease is commonly used as a criterion consequences (benefits, harms and costs) of a decision. for priority setting, it should be noted that the use of sum• Values that influence recommendations should be reported along with the research evidence mary burden of disease measures, such underlying as disability Findings recommendations. What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing The• When US Institute of Medicine's (IOM) on onordisease thanuncertainty resource about use and interventions, differences in values would lead Committee to different decisions there is rather important values that are Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such critical to a decision, this should be flagged and reflected in the strength of the recommendation. ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such • Adaptable guideline that that allowthe forpriority integration values be developed and [17] used and when[18]). guidelines published in templates 1995 argued setof different measures (seeshould for example references differences in values are likely to be critical to a decision. ting process should be open and defensible [14] They recWhat processes should be used to agree on priorities? ommended six general criteria: prevalence, burden of Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracfound only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, clinical guidelines [10]. They suggested foland potential of a guideline to reduce costs. Because data tation ofpractice recommendations. We will refer to these the as ethiBackground lowing framework for priority setting: used to make theseOrganization judgements is(WHO), often lacking, theyother sugThe World Health like many cal considerations. gested explicitaround opportunities for has important unmeasured organisations the world, recognised the need 1. Consult withmethods end users and stakeholders before factors to be rigorous considered. They further suggested separate to use more processes to ensure that health care Several formal exist to other measure values in healthselecting "tracks" for considering issues forbyspecific populations, recommendations are informed the best available care [5,6].topics. The principle methods are based on direct utilconditions or concerns. They the of rationales for a series 16 reviews research evidence. This is the argued 10th of that ity instruments including the standard gamble, time-trade 2. feasibility during the[5,6]. consultation. designating special tracksasshould be defined clearly;from i.e. that have been prepared background for advice offConsider and visual analogue scales Direct preferencewhy they warrant consideration. They suggested the WHO Advisoryspecial Committee on Health Research to based instruments generate a value score for respondents' 3. Document process used to select guideline topics. that updating should also be considered as a WHO on howof toguidelines achieve this. current healththe state or hypothetical states, typically on a separate "track", taking account of issues such as the need 0.0 to 1.0 scale where 0.0 indicates dead and 1.0 indicates The suggested following procedures [14]: for corrections the availabilityorofvalues new evidence. Utilities, healthand state preferences are the desirafull IOM health. Indirectthe methods offer alternatives to direct bility or preference that individuals exhibit for a particular assessments and include multi-attribute utility tools and •transformations the use of Delphi-like expert Oortwijn identified 25 criteria used assign to prioritise health health state [1]. Individuals usually less value to based onprocedures quality of for lifeobtaining assessments [7judgments or topic rankings correspondence technology assessments categorised these intostates four and have less preference and for more impaired health 10]. Multi-attribute utility through tools ask respondents to broad categories: burden of disease, potential effects, (e.g. death or dependency after a stroke) compared to describe their health state, and the value of that health •state theisuse of questions are specific, explicit and conpotential costs, and(e.g. uncertainty regarding application of other health states full health or having a very mild calculated usingthat a developed formula representing sistent with standard methodspopulation. for questionnaire constructhe technology [15]. sequela). In this document we will stroke without serious preferences of the general The Euroquol, tion use the terms "values" to refer to the relative worth or Quality of Well-Being Index, and the Health Utilities In a more recent selective review for the New(benefits, Zealand importance of a health state or consequences Index are examples of this approach [9-11]. Transforma•tions experimentation more formal procedures to arrive Guidelines Group,ofthe followingItcriteria were identified as harms and costs) a decision. is primarily this concept from generic with health related quality of life tools utiat judgments indicating thatwea focus topic on is suitable of values that here. for guideline developlizegroup modelling techniques that transform quality of life ment [16]: scores into values [8]. Ranking scales and qualitative They also compliment suggested there is methods a need to[6]. define more narEthical or moral values also play a role in making health methods these However, the rowly and precisely topicsis for guidelineby development. 1. The topic is clinically important affecting numbers of of care recommendations [2-4]. These referlarge to concepts application of their results complicated the fact that They argued that this would result in moremethods efficientisorganpeople substantial or mortality (the burden of what iswith right based onmorbidity philosophical, humanistic or relithe reproducibility between the different poor ization and their work, resolution of some illness). gious considerations. Ethical values can vary among indiand eachofofpanels the methods has strength and limitations. apparent controversies, more responsive guidelines, and viduals within a society and across societies or culture, easier 2. The topic is complex enough to initiate and debate the recand may influence recommendations theabout implemenValuesimplementation. play a role in every recommendation, either explicommendations. itly or implicitly. For instance patients who suffered an

Page 4 2 of 7 8 (page number not for citation purposes)

the use of 4:22 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/22

idiopathic deep venous thrombosis (DVT) usually receive treatment with adjusted dose warfarin for one year to prevent recurrent DVT and pulmonary embolism [12]. Continuing on standard-intensity warfarin beyond the treatment of one year will reduce his absolute risk for recurrent DVT by more than 7% per year for several years [13]. The burdens of treatment include taking a warfarin pill daily, keeping dietary intake of vitamin K constant, monitoring the intensity of anticoagulation with blood tests, and living with the increased risk of both minor and major bleeding. Patients who are very averse to a recurrent DVT would consider the benefits of avoiding DVT worth the downsides of taking warfarin. Other patients are likely to consider the benefit not worth the harms and burden. Another example refers to the different values patients with atrial fibrillation and their clinicians place on the adverse consequences of stroke and gastrointestinal bleed [14]. A third example relates to a health care recommendation about the combination of chemotherapy and radiotherapy versus radiotherapy alone in unresectable, locally advanced non-small cell lung cancer [15,16]. Compared with radiotherapy alone, the combination of chemotherapy and radiotherapy reduces the risk for death corresponding to a mean gain in life expectancy of a few months [15], but increases harm and burden related to chemotherapy. Thus, considering the values and preferences patients would place on the small survival benefit in view of the harms and burdens, guideline panels may offer a weak recommendation despite the high quality of the available evidence. Generally, there is agreement that the values that are used for comparing the relative benefits and downsides of interventions should be explicit [1720].

guidelines did not yield information about the inclusion of end users or patients in guideline groups or the use of other methods to ensure appropriate integration of values and consumer involvement. The fact that WHO makes global recommendations presents challenges for obtaining appropriate representation of values, because values may differ across different cultures and settings.

What other organisations are doing?

A number of guideline developers have invited individuals who could represent and understand the perspectives of stakeholders, including consumers [25]. However, in a recent survey of organizations and specialty societies that develop guidelines only approximately 25% regularly involved consumers in the process [26]. In another survey of 18 prominent organizations that develop guidelines approximately 50% regularly involved patients [27]. The UK National Institute for Health and Clinical Excellence (NICE) has adopted a very comprehensive approach to involving patients and consumers and has formed a patient involvement unit aiming to involve patients and carers in the development of individual clinical guidelines [20,28]. NICE consumer involvement can be categorized into four broad areas: 1) Stakeholder consultation Organisations can register and comment at any stage during the development process. They can nominate patient groups and participate and comment on the development of a guideline at every stage from the suggestion of guideline topics, drafting of scopes, development and initial drafting of guidelines, to the second consultation draft.

In this paper we addressed the following questions: 2) Direct input • Whose values should WHO use when making recommendations?

• How should users and consumers be involved in generating recommendations?

NICE committees and working groups are expected to include at least two members who play a crucial role by providing a patient/carer perspective to their discussions and decisions. They may be patients, carers or patient advocates. Vacancies are publicised via national patient and carer organisations, on a website or via the national press.

• How should values be presented in recommendations?

3) Indirect input

Questions related to identifying important outcomes, group composition and equity considerations are addressed in other papers in this series [21-23].

Examples include focus groups with patients, patient written testimonials and video-taped interviews with patients that were presented to a technology appraisal committee.

What WHO is doing now

4) Dissemination of NICE guidance to and by patients

• How should WHO ensure that appropriate values are integrated in recommendations?

The Guidelines for WHO Guidelines suggests that end users, and patients specifically, should be represented on guideline panels [24]. However, review of selected WHO

All NICE guidance is produced in versions written for patients, carers and the public. Copies can be downloaded

Page 3 of 8 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:22

http://www.health-policy-systems.com/content/4/1/22 http://www.health-policy-systems.com/content/4/1/14

and the from theGuidelines NICE website International or printedNetwork copies can [8])be forobtained existing by telephoning systematic reviews the and NHSrelevant Response methodological line. Patient organisaresearch tionsaddress that play anthese increasingly questions. significant The answers role in to helping the questions NICE disseminate are our conclusions its guidance basedtoon individual the available patients evidence, and carers conand providing sideration of what feedback. WHO and other organisations are doing, and logical arguments. The Scottish Intercollegiate Guidelines Network (SIGN) involves For this review patients weand searched patientPubMed representatives using (clinical in the guidepracline guidelines tice development or public process.health For each guidelines) guideline, and SIGN (priority aims to search setting or setting for evidence priorities) about and patient related important articles for outcomes selected after having references [9,10]. identified We searched a guideline the topic, Cochrane but Methodolthis is not doneRegister ogy consistently using(R. priority Habour, or priorities. personal communication). We reviewed the International Conference on Priorities SIGN also website of the aims 5thto include patient representatives and in Health Care had during in our files guideline users[11] in and the references process atthat all we stages the [12-15]. development process [17]. The UK National Health System (NHS) Health Technology Assessment program also systematically involves public advocates in their work [29Findings What criteriaspeciality should be societies used to establish 32]. Various describe priorities? that representaThe USconsumers Institute on of guideline Medicine'spanels (IOM) on tion of (at Committee least one memMethods for Setting for Guidelines Developber) is required which Priorities is often described as representation ment in its study of setting for clinical practice by a patient advocacy group priorities [33]. guidelines published in 1995 argued that the priority setting process should be open and defensible [14]systematic They recThe Cochrane Collaboration, which produces ommended six general criteria: prevalence, burden of reviews, but does not make recommendations, has made illness, costinvolvement of managinganthe problem, consumer integral part variability of its workin ofpracprotice, potential of a guideline to improve health outcomes, ducing and disseminating systematic reviews and demonand potential a guidelineofto reduce costs. Because data strated the of feasibility international consumer used to make projects. these judgements is often lacking,two theythirds suginvolvement In 1998 approximately gested explicit opportunitiesreview for important unmeasured of Cochrane Collaboration groups had consumer factors to be considered. suggested separate involvement [34]. There They is alsofurther a Cochrane Consumers "tracks" considering issues specific populations, Network for [35], which helps to for provide consumer input conditions or concerns. They argued that the rationales for into developing Cochrane reviews. designating special tracks should be defined clearly; i.e. why they warrant special consideration. They suggested Methods The methods to prepare this review describedasina that updating used of guidelines should also beare considered the introduction to thisaccount series [36]. Briefly, theaskey separate "track", taking of issues such the quesneed tions addressedand in the thisavailability paper were amongst the for corrections of vetted new evidence. authors and the ACHR Subcommittee on the Use of Research Evidence We used reviewed existing health guideOortwijn identified(SURE). 25 criteria to prioritise lines for guidelines to identify processes these for integrating technology assessments and categorised into four consumer values and consumer involvement. Weeffects, based broad categories: burden of disease, potential the current summary and recommendation on the work potential costs, and uncertainty regarding application of of prominent of guidelines. We also searched the technologydevelopers [15]. PubMed using "consumer" and "involvement" as search terms (MESH headings/keywords) reviews. In a more recent selective review for forsystematic the New Zealand We searched PubMed for systematic reviews on how Guidelines Group, the following criteria were identified as guideline groups integrate values for andguideline preferences using indicating that a topic is suitable developthe terms ment [16]:"guideline" and "values OR preferences OR utilities" (we identified 694 citations labelled as systematic reviews). Weisalso searched the Cochrane Method1. The topic clinically important affecting library, large numbers of ology with registry and database using the keywords "guidepeople substantial morbidity or mortality (the burden of line" and ("values" or "preferences"). We searched illness). databases maintained by the Agency for Healthcare Research and Qualityenough (AHRQ, [37]) and theabout Guidelines 2. The topic is complex to initiate debate the recInternational Network (GIN, [38]), and reviewed inforommendations.

mation various between organizations own 3. Thereobtained is evidencefrom of variation actual and our approprifiles. We did not conduct a full systematic review. The ate care. answers to the questions are our conclusions based on the available evidence, consideration of what WHOtoand 4. There are no existing valid guidelines available use.other organisations are doing and logical arguments. 5. There is an adequate amount of existing evidence available.

Findings

WeThe did recommendations not find a systematic review of methods inte6. will be acceptable to the for potential grating values in guidelines, but we found one systematic users. review that compared whether values differ between the general population patients is [39]. Another systematic 7. Implementation of and the guideline feasible, will not exhausreviewusethat on interventions promote contively the focused communities' resources, and to barriers to clinical sumer involvement developing policy and change are not so highin that they cannothealthcare be overcome. clinical practice guidelines did not find any comparative studiesburden of consumer involvement versus While of disease is commonly usedno as aconsumer criterion involvement or of different of involving consumers for priority setting, it should ways be noted that the use of sum[40]. We identified systematic reviews such dealing indimary burden of disease measures, as with disability rect evidence focusing onhas consumer involvement, for adjusted life years (DALYs) been criticised for focusing example research setting [31].interventions, One review on diseasein rather thanagenda resource use and focused of onthe involving patients in the planning andindevelbecause assumptions about values inherent such opment ofand health careof and another limitations systematic of review measures, because the technical such addressed(see which should be [17] usedand to include measures for methods example references [18]). the views of the public in policy documents [6,41]. We also What be usedvalues to agree on priorities? foundprocesses a review should summarized obtained by rating Batista and Hodge in astandard review conducted years disago scale, time-trade off and gamble for10 different found only[42]. threeInarticles pertinent to priority setting for ease states addition, we identified several articles clinical practice whether guidelines [10]. They theintefolthat addressed patient valuessuggested should be lowing framework for priority setting: grated in clinical practice guidelines [43]. Whose values should use when 1. Consult with end WHO users and other making stakeholders before recommendations? selecting topics. Clinicians' values for health states differ from those of patients andfeasibility among different groups [14,44]. For 2. Consider duringclinician the consultation. example, the values physicians assign to stroke as an outcome and to adverse consequences (e.g., gastrointestinal 3. Document the process used to select guideline topics. bleeding) of treatment to prevent stroke in patients with atrialIOM fibrillation differ from those of patients [14]. The The suggested the following procedures [14]: values used in making recommendations should reflect those of the people affected. While for there is widespread â&#x20AC;˘ the use of Delphi-like procedures obtaining expert belief that or values healththrough states also differ between judgments topic for rankings correspondence patients and the general public, this belief is not supported byof the availablethat evidence. A systematic of â&#x20AC;˘ the use questions are specific, explicit review and con33 studies thatmethods preferences for hypothetical health sistent withfound standard for questionnaire construcstates did not differ between patients and the public [39]. tion

Since guidelines, for themore mostformal part, affect the usetoofarrive limâ&#x20AC;˘ experimentation with procedures ited public resources and, therefore, inevitably affect the at group judgments general public, WHO guidelines should consider societal valuesalso andsuggested recognise there when is there mayto bedefine important, They a need morelegitnarimate differences in values differentdevelopment. cultures and rowly and precisely topics across for guideline settings. Judgements shouldresult be explicit should be They argued that this would in more and efficient organinformed input from (includingof citizens, ization ofbypanels and those their affected work, resolution some patients, clinicians and policy Representation of apparent controversies, more makers). responsive guidelines, and all potentially relevant societies in groups developing gloeasier implementation. bal recommendations is not feasible. If differences in val-

Page 4 of 7 8 (page number not for citation purposes)

the use of 4:22 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/22

ues are likely to affect a decision, such that people in different setting would likely make different choices about interventions or actions based on differences in their values, global recommendations should be explicit in terms of which values were applied and allow for adaptation after incorporating local values. It may also be possible to include consumers or other stakeholders in panels with the primary responsibility of questioning assumptions that are made about values, rather than representing the values of any particular group.

sometimes be useful [45]. Less formal methods include representation on guideline panels and consultation with consumers to inform judgements about the relative importance of the benefits and downsides of interventions based on rating scales. WHO guideline groups should uniformly apply explicit, transparent and clearly described methods for integrating values. How should users and consumers be involved in generating recommendations? Consumers should be part of guideline groups, be able to contribute and be heard. Van Wersch and Eccles evaluated a guideline development program [46]. They described experience with three alternative methods of consumer involvement in guideline development: a) incorporating individual patients; b) a one off meeting with patients; and c) incorporating a consumer advocate in the guideline development group. They concluded that consumers should be involved in all stages of guideline development, while acknowledging that this is not straightforward, that there is no right way to accomplish this, and that more research is required to optimize the process and outcomes.

We found discussions of ethical considerations in guidelines and health technology assessments [2-4], but we did not find a systematic review of processes for addressing ethical considerations systematically and transparently. There is no standard way for doing this, although Hofmann has taken a step towards developing a practical approach by identifying relevant questions that could be asked in the context of health technology assessments [4]. How should WHO ensure that appropriate values are integrated in recommendations? The evidentiary basis for appropriate identification and representation of values in guideline development is limited. We did not identify a systematic review focusing on different ways of including values in clinical practice guidelines. A review by Ryan and colleagues focused on public preferences for healthcare [6]. The authors concluded that there was no single, best method to gain public opinion. The method must be carefully chosen and rigorously carried out in order to accommodate the question being asked. They recommend conjoint-based methods (including ranking, rating and choice-based), willingness to pay, standard gamble and time trade-off as quantitative techniques and one-to-one interviews, focus groups, Delphi technique and citizens' juries as qualitative techniques. There were only a few studies that conducted direct comparisons of methods at that time (up to 2000) and the review requires updating.

Indirect evidence about the involvement of consumers comes from studies evaluating the involvement of consumers in research priority setting. Oliver and colleagues systematically reviewed different methods of consumer involvement in research priority setting [31]. They concluded that "what we know about the advantages and disadvantages of methods for involving consumers in agenda setting rests on weak short-term evidence and almost entirely speculative long-term evidence". Telford and colleagues used a Delphi approach to identify principles and indicators of successful involvement of consumers in research [47]. They identified eight principles for the successful involvement in research that could also be used by WHO to ensure that consumers are adequately represented in guideline development projects (Table 1).

While the evidence about which method should be adopted is inconclusive, all of the methods mentioned above are acceptable. Methods that transform results of generic health related quality of life instruments into value scores can also be used but they have the disadvantage of requiring calculations and are based on mathematical assumptions in developing transformation equations. If data from primary research on societal or patient values are used, guideline panels should make explicit how these values were measured (e.g., they should specify whether a visual analogue scale, standard gamble or other methods were used).

Any involvement of consumers requires a clear understanding of the evidence by consumers. Difficulties with medical terminology or other jargon are an important barrier to involvement. Well-informed and experienced consumers are more likely to interact with the guideline developers than those who are less informed or less familiar with medical terminology or other jargon that is used. A large number of studies have been conducted using a variety of methods to elicit values for direct patient care questions, but this literature is not well summarised and it may be beyond the capacity of most groups developing recommendations to systematically review the literature relevant to the specific questions that they are addressing. The review by Morimoto and Fukui is limited to direct

Decision analysis and economic analyses are approaches to explicitly integrating values into guidelines, which may

Page 5 of 8 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:22

http://www.health-policy-systems.com/content/4/1/22 http://www.health-policy-systems.com/content/4/1/14

Table 1: The principles and indicatorsNetwork of successful involvement in NHS researchof(from Telford et al.) actual [37] and appropriand the Guidelines International [8])consumer for existing 3. There is evidence variation between systematic reviews and relevant methodological research ate care. Principle Indicator(s) that address these questions. The answers to the questions are our conclusions on thethe available con- • The 4. There no existing valid were guidelines available to use. 1 The roles of consumers arebased agreed between researchersevidence, and consumers roles of are consumers in the research documented involved in the research sideration of what WHO and other organisations are 2 Researchers budget appropriately for the costs of consumer involvement in • Researchers applied for funding to involve consumers in the research doing, 5. There were is anreimbursed adequate existing evidence available. research and logical arguments. • Consumers foramount their travelof costs 3 Researchers respect the differing skills, knowledge and experience of consumers

For this review we searched PubMed using (clinical prac4 Consumers are offered training and personalguidelines) support, to enable them(priority to be tice guidelines or public health and involved in research setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodol5 Researchers ensure that they have the necessary skills to involve consumers in the research process using priority or priorities. We reviewed the ogy Register 6 Consumers involved in decisions about how participants areon bothPriorities recruited and Conference website ofare the 5th International kept informed about the progress of the research in Health Care [11] and references that we had in our files [12-15]. 7 Consumer involvement is described in research reports

Findings

8 Research findingsshould are available consumers, in formats and in language they can What criteria betoused to establish priorities? easily understand The US Institute of Medicine's (IOM) Committee on Methods for Setting Priorities for Guidelines Development in its study of setting priorities for clinical practice guidelines published in 1995 argued that the priority setting process should be open and defensible [14] They recommended six general criteria: prevalence, burden of illness, cost of managing the problem, variability in pracpreference instruments and to does not provide values for tice, potential of a guideline improve health outcomes, the full spectrum diseases for which WHO recand potential of aofguideline to reduce costs. develops Because data ommendations [42]. However, isthis andlacking, similarthey reviews used to make these judgements often sugcan provide some guidance because direct elicitation of gested explicit opportunities for important unmeasured values from representative sample of people is rarely feafactors to bea considered. They further suggested separate sible. "tracks" for considering issues for specific populations, conditions or concerns. They argued that the rationales for WHO shouldspecial consider developing a database collabodesignating tracks should be defined or clearly; i.e. ratingthey withwarrant others special to establish a database of suggested evidence why consideration. They about the relative values of common health outcomes that updating of guidelines should also be considered as a across different and settings, which could be need used separate "track",cultures taking account of issues such as the to inform the judgements made by groups recomfor corrections and the availability of new making evidence. mendations. Oortwijn identified 25 criteria used to prioritise health How should assessments values be presented in recommendations? technology and categorised these into four Values categories: should always be considered making recombroad burden of disease,inpotential effects, mendations, although they do not alwaysapplication influence the potential costs, and uncertainty regarding of strength of a recommendation when they are uniformly the technology [15]. shared among patients and society. While values should always be made WHO could its presentaIn a more recentexplicit, selective review forrestrict the New Zealand tion and labelling to those thatwere are most imporGuidelines Group,of thevalues following criteria identified as tant for decision-making. These recommendations could indicating that a topic is suitable for guideline developbe flagged ment [16]: as being strongly influenced by values and include a presentation of whose values they represent. In particular should: 1. The topicrecommendations is clinically important affecting large numbers of people with substantial morbidity or mortality (the burden of • Include a detailed description of how decisions were illness). made about the relative importance of the consequences (benefits and downsides) of to a decision. Thisabout should 2. The topic is complex enough initiate debate therourectinely be included in the methods section of a guideline. ommendations.

• Consumers were reimbursed for their indirect costs (e.g. carer costs) • The contribution of consumers-skills, knowledge and experience were included in 6. The recommendations will be acceptable to the potential research reports and papers • Consumers users. – training needs related to their involvement in the research were agreed between consumers and researchers • Consumers had access to training to facilitate their involvement in the research • Mentors were available to provide personal and technical support to consumers 7. Implementation of the guideline is feasible, will not exhaus• Researchers ensured that their own training needs were met • in relation to involving consumers in the research and barriers to clinical tively use the communities' resources, • Consumers gavenot advice to researchers on how to recruit to the change are so high that they cannot be participants overcome. research • Consumers gave advice to researchers on how to keep participants informed about the progress of the research While burden of disease is commonly used as a criterion • The involvement of consumers in the research reports and publications was acknowledged for priority setting, it should be noted that the use of sum• Details were given in the research reports and publications mary burdenwere ofinvolved disease measures, • of how consumers in the research processsuch as disability • Research findings disseminated to consumers in the research in adjusted lifewere years (DALYs) has beeninvolved criticised for focusing appropriate formats (e.g. large print, translations, audio, Braille) on distribution disease ofrather than resource • The the research findings to relevantuse and interventions, • consumer groups was in appropriate formats and easily understandable language because involved of the inassumptions values in such • Consumers the research gave about their advice on the inherent choice of methods used to distributeand the research findings measures, because of the technical limitations of such

measures (see for example references [17] and [18]). What processes should be used to agree on priorities? Batista and Hodge in a review conducted 10 years ago onlythat three articles pertinent to priority should setting for •found Values influence recommendations be clinical practice guidelines [10]. evidence They suggested the folreported along with the research underlying the lowing framework for priority setting: recommendations.

Consult with endinusers and otherlead stakeholders •1.When differences values would to differentbefore deciselecting topics. sions or there is important uncertainty about values that are critical to a decision, this should be flagged and 2. Consider feasibility consultation. reflected in the strengthduring of thethe recommendation. Documentguideline the process used tothat select guideline topics. •3.Adaptable templates allow for integration of different values should be developed and used when The IOM suggested differences in valuesthe arefollowing likely to procedures be critical to[14]: a decision [48]. • the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence Discussion There is no high quality research informing the choice of whose values guidelinethat panels or methods of • the use of questions are should specific,use explicit and conconsumer NICE has set examples and made sistent withinvolvement. standard methods for questionnaire construcadvancements in involving consumers in guideline develtion opment. Feedback from consumers involved in the NICE process indicates that they value their procedures involvement • experimentation with more formal tohighly arrive [28]. at group judgments Oliveralso andsuggested colleaguesthere identified a number of studies that They is a need to define more narevaluated in rowly and different precisely ways topics offorinvolving guideline consumers development. research priority [31].result Whileinthe results of their sysThey argued that setting this would more efficient organtematic of review are and informative, thereresolution are two important ization panels their work, of some limitations: 1) they focused research guidelines, priority setting apparent controversies, moreon responsive and and 2)implementation. direct comparison of different methods of coneasier sumer involvement were not found. A systematic review

Page 4 6 of 7 8 (page number not for citation purposes)

the use of 4:22 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/22

Authors' contributions

of comparative studies of methods for involving consumers in developing health care policy, research, clinical practice guidelines and patient information found five randomised trials, but none of these were relevant to informing decisions about how best to involve consumers in developing health care recommendations [40].

HJS prepared the first draft of this review. AF and ADO contributed to drafting and revising it. All authors read and approved the final manuscript.

Further work

We have identified a number of unresolved questions that require systematic reviews and additional research. A systematic review to evaluate the differences in values between consumers/patients and clinicians or experts is needed. Experimental work is needed that compares different strategies of consumer involvement in guideline development to evaluate whether more resource intensive approaches that include detailed methods to elicit and include values lead to different recommendations or other important differences.

References 1. 2. 3. 4.

A database of values assigned to specific health states may also facilitate the development of guidelines. Such a database should include information on the methods used and these methods should be explicitly stated when values are included in recommendations. While additional research on acceptable methods for eliciting values for inclusion in guidelines is required, one barrier is the philosophical and personal investment of researchers in particular methods. None of these available methods has demonstrated its superiority to others. Thus, achieving consensus on current best practice, that could be modified when new evidence become available, might be helpful.

7. 8. 9.

Development of an appropriate checklist of questions that address key ethical considerations would help to ensure that these were addressed more systematically and facilitate reporting of important considerations, so that these were made more transparent. Adaptable guideline templates that allow for integration of different values should be developed and used when differences in values are likely to be critical to a decision [49]. Finally, better ways of communicating value-sensitive information need to be investigated.

10. 11. 12.

13. 14.

Competing interests

15.

16. 17. 18.

Torrance GW Feeny, D: Utilities and quality-adjusted life years. Int J Tech Ass Health Care 1989, 5(4):559-575. Cohen J: Are clinical practice guidelines impartial. Int J Technol Health Care 2004, 20:415-420. Giacomini MK Cook DJ, Streiner DL, Anand SS: Using practice guidelines to allocate medical technologies: an ethics framework. Int J Technol Health Care 2000, 16:987-1002. Hofmann B: Toward a procedure for integrating moral issues in health technology assessment. Int J Technol Health Care 2005, 21:312-318. Bennett KJ Torrance, GW: Measuring health state preferences and utilities: rating scale, time trade-off, and standard gamble techniques. In Quality of life and pharmacoeconomics in clinical trials second edition. Edited by: Spilker B. Philadelphia , LippincottRaven; 1996:259. Ryan M, Scott DA, Reeves C, Bate A, van Teijlingen ER, Russell EM, Napper M, Robb CM: Eliciting public preferences for healthcare: a systematic review of techniques. Health Technol Assess 2001, 5(5):1-186. Froberg DG, Kane RL: Methodology for measuring health-state preferences--I: Measurement strategies. Journal of Clinical Epidemiology 1989, 42(4):345-354. Brazier J Usherwood T, Harper R, Thomas K: Deriving a preference-based single index from the UK SF-36 Health Survey. J Clin Epidemiol 1998, 51:1115-1128. Torrance GW Furlong W, Feeny DH, Boyle M: Multiattribute preferences functions: Health Utilities Index. Pharmacoeconomics 1995, 6:503-520. Kaplan RM Bush JW, Berry CC: Health Status: Types of Validity and the Index of Well-being. Health Services Res 1976, 11:478-507. EuroQol-Group: EuroQol - a new facility for the measurement of health-related quality of life. Health Pol 1990, 16:199-208. Guyatt G Gutterman D, Baumann M, Addrizzo-Harris D, Hylek E, Phillips B, Raskob G, Lewis S, SchĂźnemann H.: Grading strength of recommendations and quality of evidence in clinical guidelines. Chest 2006, 129:174-181. Buller HRA G. Hull, RD. Hyers, TM. Prins, MH. Raskob GE.: Antithrombotic Therapy for Venous Thromboembolic Disease. CHEST 2004, 126:401Sâ&#x20AC;&#x201C; 428S. Devereaux PJ, Anderson DR, Gardner MJ, Putnam W, Flowerdew GJ, Brownell BF, Nagpal S, Cox JL: Differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation: observational study. Bmj 2001, 323(7323):1218-1222. Pritchard RS, Anthony SP: Chemotherapy plus Radiotherapy Compared with Radiotherapy Alone in the Treatment of Locally Advanced, Unresectable, Non-Small-Cell Lung Cancer: A Meta-Analysis. Ann Intern Med 1996, 125(9):723-729. Robinson LA, Wagner H Jr., Ruckdeschel JC: Treatment of Stage IIIA Non-small Cell Lung Cancer. Chest 2003, 123(90010):202S-220. Methodology of guideline development. Neurosurgery 2002, 50(3 Suppl):S2-6. AGREE, Collaboration: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12(1):18-23.

Page 7 of 8 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:22

http://www.health-policy-systems.com/content/4/1/22 http://www.health-policy-systems.com/content/4/1/14

19. RS WMCInternational Tunis SR, Bass EB, Rubin HR,[8]) Haynes More and Hayward the Guidelines Network for RB: existing informative abstracts of articles describing clinical practice systematic reviews and relevant methodological research guidelines. Ann Intern Med 1993, 118:731-737. that Rawlins addressMD, these questions. The Institute answers for to the questions 20. Culyer AJ: National Clinical Excellence and its valuebased judgments. 2004, 329(7459):224-227. are our conclusions on theBmj available evidence, con21. Fretheim A Schünemann HJ, Oxman AD: Improving the Use of sideration of Evidence what WHO and other organisations are Research in Guideline Development: 3. Group composition. Health Res Policy Syst 2006. doing, and logical arguments.

22.

blockade in patients receivingbetween mechanical ventilation. Am J 3. There is evidence of variation actual and appropriCrit Care 2000, 9(6):412-418. ate care. 45. Guyatt G Strauss S, McAlister F, Haynes B, Sinclair J, Devereaux PJ, Lachetti C: Moving from Evidence to Action. Incorporating

patient In Users' Guide to the Medical Literature: Manual for 4. There arevalues. no existing valid guidelines available toAuse.

Evidence-Based Clinical Practice Edited by: Guyatt G RD. Chicago , AMA Press; 2002:567-582. 46. van Wersch A, Ecclesamount M: Involvement consumers in the 5. There is an adequate of existingofevidence available. development of evidence based clinical guidelines: practical experiences from the North of England evidence based 6. The recommendations be acceptable the Care potential guideline development will programme. Qual SaftoHealth 2001, users.10(1):10-16. 47. Telford R, Boote JD, Cooper CL: What does it mean to involve consumers successfully in NHS research? A consensus study. 7. Implementation of 7(3):209-220. the guideline is feasible, will not exhausHealth Expect 2004, 48. HJ, Fretheim A,resources, Oxman AD:and Templates, tivelySchünemann use the communities' barriers adaptation to clinical and the role of international guidelines. . (submitted as part of change are not so high that they cannot be overcome. this series) 49. Schünemann HJ Oxman AD, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 13. AdaptaWhile burden of disease is commonly used as a criterion tion, applicability and transferability. Health Res Policy Syst 2006.

Oxman AD Schünemann HJ, Fretheim A.: Improving the Use of Research Evidence in Guideline Development: 12. Incorpo-

For this review we searched PubMed (clinical rating considerations of equity. Healthusing Res Policy Syst 2006.prac23. Schünemann HJ Oxman, AD: Improving Use of tice guidelines orFretheim public A, health guidelines) andthe (priority Research Evidence in Guideline Development: 6. Determinsetting or setting priorities) and related articles for selected ing which outcomes are important. Health Res Policy Syst 2006. references We searched theforCochrane Methodol24. Global [9,10]. Programme on Evidence Health Policy. Guidefor WHO Geneva ogy lines Register using Guidelines. priority orEIP/GPE/EQC/2003.1. priorities. We reviewed the, World Health Organization; 2003. on Priorities website theconsensus 5th International 25. The of 2000 conferenceConference on clinical practice guidelines. Toronto. Health Policy Forum; in Health Care [11] and references that2000 we Nov had 2. in .our files 26. Moynihan R Paulsen, E, Lavis, J, Oxman, AD: A survey of producers [12-15]. of clinical practice guidelines and health technology assess-

for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]).

ments. 2005. 27. Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards eviFindings dence-based clinical practice: an international survey of 18 What criteria should be used to establish clinical guideline programs. Int J Qualpriorities? Health Care 2003, The 15(1):31-45. US Institute of Medicine's (IOM) Committee on 28. Kelson for MC: Setting The NICE Patient for Involvement Unit.DevelopEvidenceMethods Priorities Guidelines Based Healthcare & Public Health 2005, 9:304–307. mentRoyle in its study of settinginvolvement priorities for clinical 29. J, Oliver S: Consumer in the healthpractice technology assessment program. J Technolthat Assessthe Health Care 2004, guidelines published in 1995Intargued priority set20(4):493-497. ting Oliver process be openofand rec30. S: should The progress lay defensible involvement[14] in They the NHS ommended six Development general criteria: prevalence, burden of Research and Programme. J Eval Clin Pract 1996, 2(4):273-280. illness, cost of managing the problem, variability in prac31. Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte tice, G, potential a guideline to improve health outcomes, Oakley A,ofStein K: Involving consumers in research and settingtofor the NHS: developing eviand development potential of agenda a guideline reduce costs. Becauseandata Health Technol Assess 2004, 8(15):1-148, approach. useddence-based to make these judgements is often lacking, they sugIII-IV. gested explicit for important 32. Oliver S, Milneopportunities R, Bradburn J, Buchanan P, Kerridge L,unmeasured Walley T, Gabbay to J: Involving consumers a needs-led research profactors be considered. Theyinfurther suggested separate gramme: a pilot project. Health Expect 2001, 4(1):18-28. "tracks" for considering issues for specific populations, 33. Brainin M, Barnes M, Baron JC, Gilhus NE, Hughes R, Selmaj K, Walconditions concerns. argued that rationales for demar G:or Guidance forThey the preparation ofthe neurological management special guidelines by EFNS scientific task forces - revised designating tracks should be defined clearly; i.e. recommendations 2004. European Journal of Neurology 2004, why 11(9):577-581. they warrant special consideration. They suggested 34. MC:ofConsumer collaboration, outthat Kelson updating guidelines should also patient-defined be considered as a comes and the preparation of Cochrane Reviews. Health separate "track", taking account of issues such as the need Expectations 1999, 2(2):129-135. for corrections and the inavailability new evidence. 35. CCNET: Consumers Cochrane. of[http://www.cochrane.org/ consumers/homepage.htm]. 36. Oxman AD Fretheim A, Schünemann HJ: Improving the Use of Oortwijn identified criteria used to prioritise health Research Evidence25 in Guideline Development: Introduction. Health Resassessments Policy Syst 2006. and categorised these into four technology 37. Agency for Healthcare andpotential Quality effects, [http:// broad categories: burden Research of disease, www.guidelines.gov] potential costs, and uncertainty regarding application of 38. Guidelines International Network [http://www.g-i-n.net] 39. Dolders MGT,[15]. Zeegers MPA, Groot W, Ament A: A Meta-analysis the technology demonstrates no significant differences between Patient and Population Preferences. Journal of Clinical Epidemiology in press. In aNilsen moreESrecent review for the New AD: Zealand 40. Myrhaug,selective HT, Johansen, M, Oliver, S, Oxman, InterventionsGroup, for promoting consumer involvement in developing Guidelines the following criteria were identified as healthcare policy and research, clinical practice guidelines indicating that a topic is suitable for guideline developand patient information material. Cochrane Library . submitted 41. mentCrawford [16]: MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, Tyrer P: Systematic review of involving patients in the planning and development of health care. Bmj 2002, 325(7375):1263. 1. The topic isT,clinically important affecting of 42. Morimoto Fukui T: Utilities measured by large ratingnumbers scale, time trade-off, and standard gamble: and(the reference people with substantial morbidity or review mortality burdenfor of health care professionals. J Epidemiol 2002, 12(2):160-178. illness). 43. Owens DK: Spine update. Patient preferences and the development of practice guidelines. Spine 1998, 23(9):1073-1079. 44. Slomka Hoffman-Hogg L, MiontoLC, Bair N,debate Bobek MB, Arroliga AC: 2. The topicJ, is complex enough initiate about the recInfluence of clinicians' values and perceptions on use of clinommendations. ical practice guidelines for sedation and neuromuscular

1. Consult with end users and other stakeholders before selecting topics. 2. Consider feasibility during the consultation. 3. Document the process used to select guideline topics. The IOM suggested the following procedures [14]: • the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence • the use of questions that are specific, explicit and consistent with standard methods for questionnaire construction Publish with Bio Med Central and every scientist can read your work free of charge • experimentation with more formal procedures to arrive "BioMed Central will be the most significant development for at group judgments disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation. BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 4 8 of 7 8 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Review

Improving the use of research evidence in guideline development: 11. Incorporating considerations of costeffectiveness, affordability and resource implications

Open Access

Tessa Tan-Torres Edejer* Address: Health Systems Financing, World Health Organization, CH-1211 Geneva 27, Switzerland Email: Tessa Tan-Torres Edejer* - tantorrest@who.int * Corresponding author Published: 05 December 2006 Health Research Policy and Systems 2006, 4:23

Received: 07 April 2006 Accepted: 05 December 2006

doi:10.1186/1478-4505-4-23

This article is available from: http://www.health-policy-systems.com/content/4/1/23 © 2006 Edejer; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract Background: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 11th of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. Objectives: We reviewed the literature on incorporating considerations of cost-effectiveness, affordability and resource implications in guidelines and recommendations. Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key questions and answers: When is it important to incorporate cost-effectiveness, resource implications and affordability considerations in WHO guidelines (which topics)? • For cost-effectiveness: The need for cost/effectiveness information should be dictated by the specific question, of which several may be addressed in a single guideline. It is proposed that the indications for undertaking a cost-effectiveness analysis (CEA) could be a starting point for determining which recommendation(s) in the guideline would benefit from such analysis. • For resource implications/affordability: The resource implications of each individual recommendation need to be considered when implementation issues are being discussed. How can cost-effectiveness, resource implications and affordability be explicitly taken into account in WHO guidelines? • For cost-effectiveness: � If data are available, the ideal time to consider cost-effectiveness is during the evidence gathering and synthesizing stage. However, because of the inconsistent availability of CEAs and the procedural difficulty associated with adjusting results from different CEAs to make them comparable, it is also possible for costeffectiveness to be considered during the stage of developing recommendations. � Depending on the quantity and quality and relevance of the data available, such data can be considered in a qualitative way or in a quantitative way, ranging from a listing of the costs to a modelling exercise. At the very least, a qualitative approach like a commentary outlining the economic issues that need to be considered is necessary. If a quantitative approach is to be used, the full model should be transparent and comprehensive.

Page 1 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:23

http://www.health-policy-systems.com/content/4/1/23 http://www.health-policy-systems.com/content/4/1/14

and the• Guidelines International Network [8]) for existing 3. There is evidence of variation between actual and appropriFor resource implications/affordability: systematic reviews and relevant methodological research ate each care. recommendation in a WHO guideline should � Resource implications, including health system changes, for that address these questions. The answers to the questions be explored. At the minimum, a qualitative description that can serve as a gross indicator of the amount of are ourresources conclusions basedrelative on theto available con- be4.provided. There are no existing valid guidelines available to use. needed, current evidence, practice, should sideration of what WHO and other organisations are does arguments. one provide guidance in contextualizing guideline at theevidence country doing, How and logical 5. There is an recommendations adequate amount of existing available. level based on considerations of cost-effectiveness, resource implications and affordability?

All models should bePubMed made available and ideally are designed allow for analysts changes For this• review we searched using (clinical prac6. The to recommendations willtobemake acceptable to inthekey potential parameters and reapply results in their and own(priority country. tice guidelines or public health guidelines) users. setting •orInsetting priorities) and scenarios related articles for selected the global guidelines, and extensive sensitivity/uncertainty analysis can be applied. references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhausResource implications for WHO ogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical it is clearConference that guidelines development change groupsare willnot need a health is need International on Priorities website• From of thethe 5thabove, so high that economist. they cannot There be overcome. to ensure that this is included in the budget for guidelines and that there is in-house support for this as well. in Health Care [11] and references that we had in our files [12-15]. While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability Findings tions andlife that the(DALYs) recommendations could be What criteria should be used to establish priorities? Background adjusted years has been criticised forexpressed focusing Health Organization many other in terms of rather scenarios countries versus The World US Institute of Medicine's(WHO), (IOM) like Committee on on disease than(for resource use with and limited interventions, organisations around Priorities the world,for hasGuidelines recognised Developthe need some resources). It also recommends that 1–2 experts in Methods for Setting because of the assumptions about values inherent in such to usein more rigorous processes to ensure that health care cost-effectiveness analysis (CEA) be involved in the ment its study of setting priorities for clinical practice measures, and because of the technical limitations of techsuch recommendations arein informed by that the the best available nical guidelines group. [17] Finally, the third guidelines published 1995 argued priority setmeasures (see fordevelopment example references andin[18]). th of a series of 16 reviews research evidence. This is the 11 stage or the localization phase in-country, it suggests that ting process should be open and defensible [14] They recthat have been as background for advice from WHO provide technical for on countries to use What processes should be assistance used to agree priorities? ommended six prepared general criteria: prevalence, burden of the WHO Committee on Health Research to their own regional information Batista andorHodge in cost-effectiveness a review conducted 10 years and ago illness, costAdvisory of managing the problem, variability in pracWHO on how of to aachieve this.toInimprove this paper we address the that WHO provide financial costsfor of found onlyalso three articlesinformation pertinent toonpriority setting tice, potential guideline health outcomes, following questions: implementing recommendations. clinical practicethe guidelines [10]. They suggested the foland potential of a guideline to reduce costs. Because data lowing framework for priority setting: used to make these judgements is often lacking, they sug• Whenexplicit is it important to incorporate cost-effectiveness, There has been some mention of cost-effectiveness in a gested opportunities for important unmeasured resourcetoimplications andThey affordability considerations in few guidelines issued and thatbefore claim 1. Consult withthat endwere users and since other 2003 stakeholders factors be considered. further suggested separate WHO guidelines (which topics)? to have used the Guidelines for WHO Guidelines. The selecting topics. "tracks" for considering issues for specific populations, 2003 WHO/ISH guidelines [2] had general sections on conditions or concerns. They argued that the rationales for • How can cost-effectiveness, resource implications cost-effectiveness and resource implications of the recom2. Consider feasibility during the consultation. designating special tracks should be defined clearly;and i.e. affordability be explicitly taken into account in WHO mendations. More recently, the WHO "guidelines" on why they warrant special consideration. They suggested guidelines? hand hygiene the in health care [3]to also included information 3. Document process used select guideline topics. that updating of guidelines should also be considered as a on economic burden of poor hygiene, cost-effectiveness separate "track", taking account of issues such as the need • How does one guidance in contextualizing and resource implications of recommendations. The IOM suggested the following procedures [14]:Another for corrections and provide the availability of new evidence. guideline recommendations at the country level based on example, the malaria treatment guidelines [4] specifically considerations of cost-effectiveness, implications state studies were not included for • the that use cost-effectiveness of Delphi-like procedures for obtaining expert Oortwijn identified 25 criteria usedresource to prioritise health and affordability? consideration because therethrough were very few completed judgments or topic rankings correspondence technology assessments and categorised these into four studies relating to the interventions being considered, and broad categories: burden of disease, potential effects, • What are the and resource implications of the answers to that of the anti-malarial drugsexplicit were fluid, • thethe useprices of questions that are specific, and "renconpotential costs, uncertainty regarding application of these questions [15]. for WHO? dering such standard studies unreliable." sistent with methods for questionnaire constructhe technology tion In a more recent doing selective review for the New Zealand What is WHO now? What other organizations are doing? In 2003, the Guidelines for WHO Guidelines [1] recomIt experimentation is clear that cost-effectiveness and/or cost implications • with more formal procedures to arrive Guidelines Group, the following criteria were identified as mended that both cost-effectiveness and resourcedevelopimplicaof group recommendations at judgments are recognized in guideline developindicating that a topic is suitable for guideline tions of guideline recommendations be considered when ment. The US Preventive Services Task Force [5] lists sevment [16]: developing WHO guidelines. With the primary audience eral reasons why CEAthere is useful guideline development: They also suggested is a in need to define more narbeing of Health officials with large a mandate rowly and precisely topics for guideline development. 1. The Ministry topic is clinically important affecting numbers to of improve population rather the health of indi1. Quantifying the differences between twoefficient or moreorganeffecThey argued that this would result in more people with substantialhealth morbidity or than mortality (the burden of viduals, such concerns were considered to be appropriate. tive services for the and sametheir condition ization of panels work, resolution of some illness). The guidelines (see Additional file 1 for relevant portions) apparent controversies, more responsive guidelines, and states thatenough cost-effectiveness is toabout be consideasier implementation. 2. The very topic briefly is complex to initiate debate the recered during the stage of formulation of the recommendaommendations.

Page 4 2 of 7 6 (page number not for citation purposes)

the use of 4:23 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/23

2. Illustrating the impact of delivering a given intervention at different intervals, different ages, or to different risk groups

organizations in incorporating costs/cost-effectiveness information during their guideline development process (e.g. lessons learned) [11] and an assessment of whether economic evaluations have provided added value to their guidelines [12].

3. Evaluating the potential role of new technologies

Methods

4. Identifying key conditions that must be met to achieve the intended benefit of an intervention

The methods used to prepare this review are described in the introduction to this series [13]. Briefly, the key questions addressed in this paper were vetted amongst the authors of the series of articles and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct full systematic reviews. We searched PubMed and three databases of methodological studies (the Cochrane Methodology Register, the US National Guideline Clearinghouse, and the Guidelines International Network for existing systematic reviews and relevant methodological research that address these questions. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

5. Incorporating preferences for intervention outcomes 6. Developing a ranking of services in order of their costs and expected benefits The AGREE guidelines appraisal instrument [6], on which the WHO checklist was loosely based, includes the cost impact of guideline recommendations under their applicability criteria. To actually gauge what other organizations are doing with respect to cost-effectiveness/costs in guideline development, one can review the guidelines that have been issued by these organizations and/or review their documented methods for guideline development. It is hoped that there is consistency between the two. In 1999, 279 guidelines that were published in peer-reviewed literature were reviewed [7]. Only 41.6% made any mention of projected effects on health care costs, and only 14.3% quantified these estimates in any way. A 2002 study on the incorporation of published cost-effectiveness analysis in published clinical guidelines showed that, using guidelines as the unit of analysis, 9 of 35 (26%) incorporated at least 1 economic analysis of above-average quality in the text and 11 of 35 (31%) incorporated at least 1 in the references [8]. Finally, a search of the database of the National Guideline Clearinghouse [9] showed that of 1616 guidelines published between 2000â&#x20AC;&#x201C;2005, only 369 or 23% had a formal cost analysis.

For this review PubMed was searched using the following text word searches: costs and generalizability, practice guidelines and cost-effectiveness analysis and combinations thereof. Using the same search words, the Internet search machine, Google, was also used to search for unpublished documents. Websites of known repositories of guidelines, of organizations of guideline agencies, and of pioneer/well-known guideline development agencies were visited. References in key documents that had titles which could be probably relevant were also pursued. A few times, when it was evident that an author had this as his/her special area of interest, a search using his/her name in connection with guidelines was also done. Papers were included if they described guidelines that included cost-effectiveness information, or described methods of guideline agencies to incorporate costs/resource implications/CEA information.

For a review of methods, a 2003 survey [10] of 18 clinical practice guideline development agencies showed that six included costs/cost containment/cost-effectiveness in their objectives but only three routinely included health economists in their guideline development groups (NHRMC in Australia, SBU in Sweden, and North of England). The AGREE prototype electronic library shows some illustrative excerpts specifically mentioning costs from the methodologies of five national agencies. More detailed information on the use of costs/cost-effectiveness information by guidelines agencies is available (see Additional file 2).

The information collected during the review was then synthesized where relevant for each question and was used as the basis to draw the implications for WHO guidelines.

Findings

When is it important to incorporate considerations of costeffectiveness and resource implications of recommendations in WHO guidelines (which topics)? The need for cost-effectiveness information should be dictated by the specific question of which several may be addressed in one guideline alone. It is proposed that the indications for undertaking a cost-effectiveness analysis could be a starting point for determining which recommendation(s) in the guideline would benefit from such analysis [14]. From the review, NHRMC states explicitly

To what extent these organizations routinely and explicitly use costs/CEA in their guidelines is not clear. More information is needed on the actual experience of these

Page 3 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:23

http://www.health-policy-systems.com/content/4/1/23 http://www.health-policy-systems.com/content/4/1/14

and the that "theGuidelines challenge International is to focus onNetwork the decision [8]) for points existing that are of key importance systematic reviews andinrelevant an economic methodological sense and pinpoint research the nature that addressof these economic questions. information The answers needed to theto questions address these are ourquestions. conclusions Thebased key on decisions the available concern evidence, health concare that contributes sideration of what significantly WHO and to the other totalorganisations cost of an option, are optionsand doing, with logical very arguments. different costs or care that contributes significantly to health outcomes. On the other hand, decisions For this arereview unimportant we searched if they PubMed concernusing health (clinical care that pracis uncontroversial, tice guidelines oroptions public that health areguidelines) not economically and (priority viable or options setting or setting for which priorities) there and are no related largearticles resource forimplicaselected tions." [15][9,10]. We searched the Cochrane Methodolreferences ogy Register using priority or priorities. We reviewed the International Conference on Priorities The UK of National for Health and Clinical Excelwebsite the 5th Institute in Health Careextends [11] and references we had in files lence (NICE) this concept that by adopting a our value of [12-15]. approach where aside from "the overall 'imporanalysis tance' of the recommendation (which is a function of the number of patients affected and the potential impact on Findings What criteria should be used establish they priorities? costs and health outcomes pertopatient)", also suggest The US Institute of Medicine's Committee on evaluating "the current extent of (IOM) uncertainty over costMethods for and Setting Priorities for Developeffectiveness the likelihood thatGuidelines analysis will reduce ment in its study[16]. of setting priorities for clinical practice this uncertainty" guidelines published in 1995 argued that the priority setImplications WHO guidelines ting processforshould be open and defensible [14] They recAt the scoping assistance of an experienced ommended six stage, generalthecriteria: prevalence, burden of health economist who is familiar with the area of in interest illness, cost of managing the problem, variability pracwould be needed and a selective identification of the tice, potential of a guideline to improve health outcomes, issues needingofCEA could betodone. If CEA not initially and potential a guideline reduce costs.isBecause data identified during scoping stage as alacking, clear need, this used to make thesethe judgements is often they sugissue needs to be revisited again the evidence, recomgested explicit opportunities for at important unmeasured mendations peer review stage. factors to beand considered. They further suggested separate "tracks" for considering issues for specific populations, On the other the resource implications of each indiconditions orhand, concerns. They argued that the rationales for vidual recommendation considered when designating special tracks needs shouldtobebedefined clearly; i.e. implementation areconsideration. being discussed. This may be why they warrantissues special They suggested doneupdating at the global guidelineshould level, through the use of scethat of guidelines also be considered as a narios, and at thetaking local adaptation or country separate "track", account of issues such level. as the need for corrections and the availability of new evidence. How can cost-effectiveness and affordability be explicitly taken intoidentified account in25 WHO guidelines? Oortwijn criteria used to prioritise health There are different pointsand in categorised the guideline development technology assessments these into four processcategories: that cost-effectiveness resource implications broad burden of and disease, potential effects, of guidelines can be considered. potential costs,recommendations and uncertainty regarding applicationThe of firsttechnology phase is the evidence phase and to the extent possithe [15]. ble, this is where cost-effectiveness information should be considered. In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as As in questions on effectiveness, theguideline questiondevelopcan be indicating that a topic is suitable for raised[16]: as to why conduct a review of CEAs rather than ment identify a single study that addresses the question? "CEAs vary widely their methods and affecting assumptions. 1. The topic in is clinically important large Because numbers of of this variation, systematically reviewing CEAs(the provides sevpeople with substantial morbidity or mortality burden of eral benefits. First, because CEAs draw on a variety of cost illness). and effectiveness data sources to develop input parameters, systematic review cantoidentify analyses use 2. Thea topic is complex enough initiate which debate about the recthe best available evidence for key inputs and are therefore ommendations.

theThere mostisevidence based. Second, because 3. evidence of variation between actualthe andcredibility appropriof CEAs ate care. rests on their quality, a critical review of CEAs and a rating of the quality of each allow for identifying the most methodologically rigorous studies. Third,toa use. compre4. There are no existing valid guidelines available hensive review can identify the studies that best address theThere question asked. Fourth, comparatively 5. is an being adequate amount of existing evidence assessing available. CEAs can help to identify variables and methods that significantly influence the estimated benefits to and effec6. The recommendations will be acceptable thecost potential tiveness of an intervention. For instance, some CEAs users. might assume no harms from a given intervention, while others might assume that the intervention has not significant 7. Implementation of the guideline is feasible, will exhausharms. these studies sideand by side maytoprovide tively useComparing the communities' resources, barriers clinical insight are intonot how the assumption or lack of assumption of change so high that they cannot be overcome. harm affects the estimated benefit of the intervention. assumptions varied within study While some burden of disease isare commonly usedaassingle a criterion using sensitivity analysis, CEAsthat provide a of limited for priority setting, it shouldmost be noted the use sumnumber of sensitivity mary burden of diseaseanalyses. measures,Thus, such systematically as disability reviewinglife CEAs may help identify, a side-by-side adjusted years (DALYs) has beenthrough criticised for focusing comparison that amounts to a "virtual sensitivity analyon disease rather than resource use and interventions, sis," the of impact of different about assumptions on the benefits because the assumptions values inherent in such of a givenand intervention. Finally, the more high-quality, measures, because of the technical limitations of such independently conducted there are a given measures (see for exampleCEAs references [17]for and [18]).intervention, the more convincing the evidence." [17] What processes should be used to agree on priorities? Batista Hodge inreview a review conducted 10 years ago In doingand a systematic of economic evaluations, the found only three articles pertinent to There priority for first step is to search for the literature. aresetting available clinical practice [10].list They suggested folresources on the guidelines internet which databases and the compilowing framework forevaluations priority setting: lations of economic [18]. In addition, there has been a systematic evaluation in terms of sensitivity 1. Consult withofend users and other stakeholders before and specificity different search strategies for economic selecting topics. evaluations [19]. 2. Consider feasibility during the consultation. In extracting data, considerable progress has been reported with development of a systematic process of 3. Document of theresults processfrom used different to select guideline topics. adjustments studies to make them comparable. The Task Force on Community PrevenThe IOM suggested following procedures [14]: tive Services admitsthe that "no process of adjustment or other means of reviewing existing economic evaluations is â&#x20AC;˘ the use of procedures for to obtaining expert flawless." ButDelphi-like it makes the point that adjust data to judgments or topic rankings through correspondence make it comparable is better than to "(1) ignore economic information entirely; (2) attempt to use non-comparable â&#x20AC;˘ the use of questions specific, explicit and condata; or (3) adjust in that waysarethat are not systematic or sistent with standard methods for questionnaire construcexplicit" [20]. tion Some work has also been done in terms of very simple visâ&#x20AC;˘ experimentation with more formal of procedures to arrive ual methods to present summaries cost-effectiveness at group [21]. judgments analysis Despite all of these advances methodologically, the current situation shows however, that there is They also suggestedand there is a need to define moreCEAs narlimited availability variable quality of relevant rowly and precisely topics for guideline development. [22]. They argued that this would result in more efficient organImplications WHO and guidelines ization of for panels their work, resolution of some If data is available, the ideal to consider CEA is durapparent controversies, moretime responsive guidelines, and ing theimplementation. evidence gathering and synthesizing stage. Howeasier ever, because of the inconsistent availability of CEAs and

Page 4 of 7 6 (page number not for citation purposes)

the use of 4:23 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/23

Overall assessment of need for health economics expertise In summary, the role of the health economist in a guideline development group is to:

the procedural difficulty associated with adjusting results from different CEAs to make them comparable [20] it is also possible for CEA to be considered during the stage of developing recommendations. This is also consistent with the GRADE approach [23]. At this stage, the information of the resource implications and outcomes of the recommended interventions can be considered simultaneously. Depending on the quantity and quality and relevance of the data available, such data can be considered in a qualitative way or in a quantitative way, ranging from a listing of the costs to a modelling exercise [24,25]. At the very least, a qualitative approach like a commentary outlining the economic issues that need to be considered is necessary [26]. If a quantitative approach is to be used, the full model should be transparent, be made available and extensive uncertainty/sensitivity analysis built-in so as to allow analysts to selectively reapply results in their own country, as in the WHO-CHOICE contextualization tool [27].

• help to identify the clinical issues or questions for economic analysis • review economic literature • carry out or commission cost-effectiveness analyses • estimate the cost and resource implications of the recommendations. "The relative weight given to each role will vary from guideline to guideline. There may be large differences between guidelines in respect of the literature available to review: the size of the relevant economics literature, its relevance, its quality, its timeliness, its generalisability. In some areas there may be good-quality data that can be used in economic models, whereas other areas may have a dearth of such data." [16] Additionally, all throughout the process, the health economist can educate the other guideline development group about CEA and vice-versa, through interaction with the guideline development group members, s/he will be better able to work with an improved understanding of the health issues being considered.

How does one provide guidance in contextualizing guideline recommendations at the country level based on considerations of cost-effectiveness and affordability? For cost-effectiveness, there are concerns about the generalizability of results from a single CEA or even a systematic review of a CEA. A review of sources of variability frequently mentions volume and costs of resources consumed as a source of variability [28]. Not as much work has been done on variability of outcomes. Very recently, a checklist was developed for assessing variability or generalizability to be able to translate information from one developed country to another [29]. There is a need to pilot test this, a revision or another instrument in developing countries. For costs, more specifically prices, general principles for adaptation are available [30].

Implications for WHO From the above, it is clear that guidelines development groups will need access to a health economist. There is need to ensure that this is budgeted for in the resources and that there is in-house support for this as well.

Further work

There is a need to: 1) get more information on the actual experience of guideline agencies in incorporating CEA in guidelines; 2) assess the added value of economic evaluations in guidelines by comparing recommendations with and without CEA; 3) further expand the section on CEA and resource implications in guidelines, including specification of the minimum information that should be provided, in collaboration with health economists and experienced guideline developers.

Affordability or resource implications can be considered in the global guidelines if it gives guidance by provision of basic information that will allow guideline users to work out the cost implications for their own service [25,15]. A scenario approach can be used. Also, in this context, WHO-CHOICE data and methods are useful for contextualization [31]. Note that this exercise will also need to include the health system implications of the recommendations, from training, changes in supervision, monitoring and evaluation, advocacy, etc. as seen in some recent examples [32,33].

Competing interests

The author helped to develop the current Guidelines for WHO Guidelines.

Implications for WHO Resource implications, including health system changes, for each recommendation in WHO guidelines should be explored. At the minimum, a qualitative description that can serve as a gross indicator of the amount of resources needed, relative to current practice, should be provided.

Authors' contributions

TTTE prepared the first draft of the review and was responsible for the subsequent revisions.

Page 5 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:23

http://www.health-policy-systems.com/content/4/1/23 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing Additional material systematic reviews and relevant methodological research that address these questions. The answers to the questions Additional file 1 are our conclusions based on the available evidence, conCost-effectiveness and affordability in the Guidelines for WHO Guidesideration of what WHO and other organisations are lines. Excerpts from the document "Guidelines for WHO Guidelines" doing, andtological arguments. which refer the concepts of cost-effectiveness and affordability.

13. There Oxman AD, Fretheim A, Schünemann HJ: Improving use of 3. is evidence of variation between actual andthe appropriresearch evidence in guideline development: introduction. ate care. Health Res Policy Syst 2006.

14.

grammes Second edition. Oxford University Press; 1997. NHRMC: How to compare the costs and benefits: evaluation of the economic evidence on available. preparing 5. There is an adequate amountHandbook of existingseries evidence clinical practice guidelines; Endorsed July 2001. [http:// www7.health.gov.au/nhmrc/publications/synopses/cp73syn.htm]. 6. The recommendations will be acceptable to the potential 16. National Institute of Clinical Excellence: The Guidelines Manual. users.Incorporating health economics in guidelines and assessing resource impact Chapter 8: [http://www.nice.org.uk/page.aspx?o=308777]. 25 April 2006. 7. Implementation of the CK, guideline feasible, exhaus17. Truman BI, Smith-Akin HinmanisAR, Gebbie will KM, not Brownson R, LF, communities' Lawrence RS, Pappaioanou M, Fielding J, Evans CA Jr, tivelyNovick use the resources, and barriers to clinical Guerra FA, Vogel-Taylor M, Mahan CS, Fullilove M, Zaza S: Developchange so to high that they Preventive cannot be overcome. ing are the not Guide Community Services – overview and rationale. The Task Force on Community Preventive Services. Am J Prev Med 2000, 18(1 Suppl):18-26. While burden of disease is commonly used as a criterion 18. NHS Economic Evaluation Database [http://www.york.ac.uk/ for priority setting, it should be noted that the use of suminst/crd/econ4.htm] 19. F, Archard Searching literature for marySassi burden ofL, McDaid diseaseD: measures, such asdatabases disability health care economic evaluations: how systematic can we adjusted yearsMed (DALYs) has been criticised for focusing affordlife to be? Care 2002, 40(5):387-94. 20. Carande-Kulis VG, than Maciosek MV, Brissuse PA, Teutsch SM, Zaza S, Truon disease rather resource and interventions, man BI, ML, Pappaioanou Harrisinherent JR, Fielding in J: Methbecause of Messonnier the assumptions aboutM, values such ods for systematic reviews of economic evaluations for the measures, because ofPreventive the technical limitations of such Guide and to Community Services. Task Force on Community Preventive J Prev Med[18]). 2000, 18(1 measures (see for exampleServices. referencesAm[17] and Suppl):75-91. 21. Nixon J, Khan KS, Kleijnen J: Summarising economic evaluations What should be usedato agree priorities? BMJ in processes systematic reviews: new on approach. 322(7302):1596-8. 30 Batista and Hodge 2001 in aJun review conducted 10 years ago 22. Jefferson T, Demicheli V, Vale L: Quality of systematic reviews of found only three articlesinpertinent to JAMA priority setting for economic evaluations health care. 287(21):2809-12. 2002practice Jun 5 clinical guidelines [10]. They suggested the fol23. Atkins D, Best D, Briss Eccles M, Falck-Ytter Y, Flottorp S, Guyatt lowing framework for PA, priority setting: GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips Schunemann HJ, Edejer TT, Varonenbefore H, Vist 1. Consult with endB,users and other stakeholders GE, Williams JW Jr, Zaza S, GRADE Working Group: Grading qualselecting topics. ity of evidence and strength of recommendations. BMJ 328(7454):1490. 2004 Jun 19 24. Eccles M, feasibility Mason J: How to develop cost-conscious guidelines. 2. Consider during the consultation. Health Technol Assess 2001, 5(16):1-69. 25. Philips Z, Ginelly L, Sculpher M, et al.: Review of good practice in decision-analytic modelling in health technology assessment. 3. Document the process used to select guideline topics. Health Technology Assessment 2004, 8(36):. 26. SIGN 50: A guideline developers' handbook [http:// The www.sign.ac.uk/guidelines/fulltext/50/section6.html] IOM suggested the following procedures [14]: 27. CHOosing Interventions that are Cost Effective (WHOCHOICE) [http://www.who.int/choice] • theSculpher use ofMJ,Delphi-like procedures for expert 28. Pang FS, Manca A, Drummond MF,obtaining Golder S, Urdahl H, Davies LM, A: Generalisability in economic evaluajudgments or Eastwood topic rankings through correspondence tion studies in healthcare: a review and case studies. Health Technol Assess 2004, 8(49):iii-iv. 1–192. • theBoulenger use of S, questions that are M, specific, explicit and con29. Nixon J, Drummond Ulmann P, Rice S, de Pouvourvillewith G: Can economic evaluations be made more constructransferasistent standard methods for questionnaire ble? Eur J Health Econ in press. 2005 Oct 26 tion Hutton G, Baltussen R: Cost valuation in resource-poor set30. tings. Health Policy Plan 2005, 20(4):252-9. 31. Hutubessy R, Chisholm Edejer formal TT: Generalized cost-effective• experimentation withD,more procedures to arrive ness analysis for national-level priority-setting in the health at group judgments sector. Cost Eff Resour Alloc 1(1):8. 2003 Dec 19 32. Deen JL, Weber M, Qazi S, Fontaine O: An extended field test of WHO severe malnutrition guidelines in hospitals in Theythe also suggested there is a need to define more nardeveloping countries: preliminary observations. Forum Nutr rowly and precisely topics for guideline development. 2003, 56:183-4. 33. G, Mendis S, would Abegunderesult D, Asmar R, Mieke S, Murray A, TheyParati argued that this in more efficient organShengelia B, Steenvoorden G, Van Montfrans G, O'Brien E: Recomization of panels and pressure their work, resolution mendations for blood measuring devices of for some office/ clinic controversies, use in low resource Bloodguidelines, Press Monit 2005, apparent moresettings. responsive and 10(1):3-10. easier implementation. 15.

Click here for file [http://www.biomedcentral.com/content/supplementary/1478For this review we searched PubMed using (clinical prac4505-4-23-S1.doc] tice guidelines or public health guidelines) and (priority setting or setting priorities) and related articles for selected Additional file We 2 searched the Cochrane Methodolreferences [9,10]. Considerations of costs/cost-effectiveness information by other agencies. ogy Register using priority or priorities. We reviewed the Excerpts from documents from other agencies that issue guidelines which Conference on Priorities website ofconcepts the 5thof International refer to the costs/cost-effectiveness inClick Health here Care for file[11] and references that we had in our files [http://www.biomedcentral.com/content/supplementary/1478[12-15]. 4505-4-23-S2.doc]

Findings

What criteria should be used to establish priorities? The US Institute of Medicine's (IOM) Committee on Acknowledgements Methods for Setting Priorities for Guidelines DevelopI gratefully acknowledge the members of the ACHR SURE, and all of those ment in its study of setting priorities for clinical practice who have provided feedback on earlier versions of this paper. guidelines published in 1995 argued that the priority setting process should be open and defensible [14] They recReferences ommended sixWHO general criteria: burden of 1. Guidelines for guidelines: World prevalence, Health Organization. Global Proillness, costonofEvidence managing the problem, variability in pracgramme for Health Policy [http://whqlibdoc.who.int/hq/ tice, 2003/EIP_GPE_EQC_2003_1.pdf]. potential of a guideline to improve health outcomes, 2. 2003 World Health Organization (WHO)/International and Society potential of a guideline to guidelines reduce costs. Because data of Hypertension (ISH) statement on manof hypertension 2003 lacking, [http://www.who.int/ usedagement to make these judgements is often they sugcardiovascular_diseases/guidelines/hypertension_guidelines.pdf]. Lipgested explicit opportunities for important unmeasured pincott, Williams and Wilkins factors to be considered. They further suggested separate 3. WHO guidelines on hand hygiene in health care (advanced draft)for : a summary : clean handsfor arespecific safer hands 2005 [http:// "tracks" considering issues populations, whqlibdoc.who.int/hq/2005/WHO_EIP_SPO_QPS_05.2.pdf]. World conditions or concerns. They argued thatPolicies the rationales for Health Organization. Dept. of Health System and Operations 4. World Health Organization: Guidelinesbe for defined the Treatment of Malaria designating special tracks should clearly; i.e. Geneva, World Health Organization; 2006. why Saha they warrant special consideration. They suggested 5. S, Hoerger TJ, Pignone MP, Teutsch SM, Helfand M, Mandelblatt that JS, updating ofGroup, guidelines should alsoServices be considered a Cost Work Third US Preventive Task Force: as The art and science of incorporating effectiveness into eviseparate "track", taking account ofcost issues such as the need dence-based recommendations for clinical preventive servfor corrections and availability of new evidence. ices. Am J Prev Medthe 2001, 20(3 Suppl):36-43. 6.

The AGREE Collaboration: Appraisal of Guidelines for Research

& Evaluation (AGREE) Instrument. [http://www.agreecol Oortwijn identified 25 criteria used2001 to prioritise health laboration.org/pdf/agreeinstrumentfinal.pdf]. technology assessments and categorised these into four 7. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of effects, clinical broad categories: burden of disease, potential practice guidelines in the peer-reviewed medical literature. potential costs, and uncertainty regarding application of JAMA 281(20):1900-5. 1999 May 26 the technology [15]. SR, Chiou CF, Henning JM, Hohlbauch AA, 8. Wallace JF, Weingarten

Richards MS, Herzog NS, Lewensztain LS, Ofman JJ: The limited incorporation of economic analyses in clinical practice guide-

In alines. moreJ Gen recent review for the New Zealand Intern selective Med 2002, 17(3):210-20. 9. NationalGroup, Guidelines [http://www.guideline.gov] Guidelines the Clearinghouse following criteria were identified as 10. Burgers JS, Grola R, Klazinga Makela for M, Zaat J, AGREEdevelopCollaboraindicating that topic is NS, suitable guideline tion: Towards evidence-based clinical practice: an internamenttional [16]:survey of 18 clinical guideline programs. Int J Qual Health Care 2003, 15(1):31-45.

11. Louis W: Niessen, Els Grijseels, Marc Koopmanschap and 1. The topic is clinically important large numbers of Frans Rutten. Economic analysisaffecting for clinical practice – the people with substantial morbidityguidelines or mortality (the burden of case of 31 national consensus in the Netherlands. Journal of Evaluation in Clinical Practice (forthcoming) . illness). 12.

Drummond Michael F, O'Brien Bernie, Stoddart Greg L, Torrance

George the Economic Evaluation of Health Care Pro4. There areW: noMethods existingforvalid guidelines available to use.

Granata AV, Hillman AL: Competing practice guidelines: using cost-effectiveness analysis to make optimal decisions. Ann Intern Medis128(1):56-63. 1998 Jan 1 The topic complex enough to initiate debate about the rec-

2. ommendations.

Page 4 6 of 7 6 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 12. Incorporating considerations of equity Andrew D Oxman*1, Holger J Schünemann2 and Atle Fretheim1 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway and 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy Email: Andrew D Oxman* - oxman@online.no; Holger J Schünemann - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no * Corresponding author

Published: 05 December 2006 Health Research Policy and Systems 2006, 4:24

Received: 07 April 2006 Accepted: 05 December 2006

doi:10.1186/1478-4505-4-24

This article is available from: http://www.health-policy-systems.com/content/4/1/24 © 2006 Oxman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract Background: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 12th of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. Objectives: We reviewed the literature on incorporating considerations of equity in guidelines and recommendations. Methods: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key questions and answers: We found few directly relevant empirical methodological studies. These answers are based largely on logical arguments. When and how should inequities be addressed in systematic reviews that are used as background documents for recommendations? • The following question should routinely be considered: Are there plausible reasons for anticipating differential relative effects across disadvantaged and advantaged populations? • If there are plausible reasons for anticipating differential effects, additional evidence should be included in a review to inform judgments about the likelihood of differential effects. What questions about equity should routinely be addressed by those making recommendations on behalf of WHO? • The following additional questions should routinely be considered: • How likely is it that the results of available research are applicable to disadvantaged populations and settings? • How likely are differences in baseline risk that would result in differential absolute effects across disadvantaged and advantaged populations? • How likely is it that there are important differences in trade-offs between the expected benefits and harms across disadvantaged and advantaged populations? • Are there different implications for disadvantaged and advantaged populations, or implications for addressing inequities? What context specific information is needed to inform adaptation and decision making in a specific setting with regard to impacts on equity? • Those making recommendations on behalf of WHO should routinely consider and offer advice about the importance of the following types of context specific data that might be needed to inform adaptation and decision making in a specific setting: • Effect modifiers for disadvantaged populations and for the likelihood of differential effects • Baseline risk in relationship to social and economic status • Utilization and access to care in relationship to social and economic status • Costs in relationship to social and economic status

Page 1 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:24

http://www.health-policy-systems.com/content/4/1/24 http://www.health-policy-systems.com/content/4/1/14

Ethics and laws thatInternational may impact on strategies for[8]) addressing inequities 3. There is evidence of variation between actual and appropriand• the Guidelines Network for existing systematic reviews and to relevant methodological research ate care. • Availability of resources address inequities thatWhat address these questions. The answers to the questions implementation strategies are likely be needed to ensure that recommendations are implemented equitably? are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. • Organisational changes are likely to be important to address inequities. While it may only be possible to consider these in relationship to specific sideration of what WHO and other organisations are settings, consideration should be given to how best to provide support for identifying and addressing needs for organisational changes. In countries doing, logical arguments. is an adequate amount of existing evidence available. with and pervasive inequities institutional, cultural and political changes may first5.beThere needed. • Appropriate indicators of social and economic status should be used to monitor the effects of implementing recommendations on disadvantaged

For this review we searched PubMed using (clinical pracpopulations and on changes in social and economic status. tice guidelines or public health guidelines) and (priority setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodology Register using priority or priorities. We reviewed the Background website of the 5th International Conference on Priorities The WorldCare Health (WHO), in Health [11]Organization and references that welike hadmany in ourother files organisations around the world, has recognised the need [12-15]. to use more rigorous processes to ensure that health care recommendations are informed by the best available Findings th of a series of 16 reviews research evidence. This the to 12establish What criteria should be is used priorities? that have been prepared as background advice from The US Institute of Medicine's (IOM) for Committee on the WHO Advisory Committee on Health Research to Methods for Setting Priorities for Guidelines DevelopWHO on how to achieve this. ment in its study of setting priorities for clinical practice

6. The recommendations will be acceptable to the potential users. 7. Implementation of the guideline is feasible, will not exhaustively use the communities' resources, and barriers to clinical • Whatarequestions equity routinely be change not so highabout that they cannotshould be overcome. addressed by those making recommendations on behalf of WHO? While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of sum• Whatburden contextof specific information needed inform mary disease measures,issuch as to disability adaptation and decision making in a specific setting with adjusted life years (DALYs) has been criticised for focusing regard to impacts on equity? on disease rather than resource use and interventions, because of the assumptions about values inherent in such Related questions about adaptation, applicability and measures, and because of the technical limitations of such transferability are addressed in another paper in this series measures (see for example references [17] and [18]). [10].

guidelines published in 1995 argued that the priority setBraveman Gruskin define as "the absence of ting processand should be open andequity defensible [14] They recdisparities in six health that are systematically associated ommended general criteria: prevalence, burdenwith of social advantage or disadvantage" [1]. The message is illness, cost of managing the problem, variability in pracmade clearer by Margaret Whitehead's definition of ineqtice, potential of a guideline to improve health outcomes, uity:potential "differences health which are costs. not only unnecesand of ain guideline to reduce Because data sary and avoidable but, in addition, are considered unfair used to make these judgements is often lacking, they sugand unjust" [2].opportunities Inequities in for health and health care are gested explicit important unmeasured well documented in relationship to social and economic factors to be considered. They further suggested separate factors, including Place of residence (e.g. urban, "tracks" for considering issues for specific rural, populations, inner city, Race/ethnicity/culture, Occupation, Gender, conditions or concerns. They argued that the rationales for Religion, Educational level, Socioeconomic status and designating special tracks should be defined clearly; i.e. Social capital (availability of neighbourhood support, why they warrant special consideration. They suggested social stigma, of civic society) should (PROGRESS) that updating guidelines also be[3]. considered as a separate "track", taking account of issues such as the need Disadvantaged populations almostof always have poorer for corrections and the availability new evidence. health [4], poorer access to health care [5], and receive poorer quality health25care [6]. To theto extent that recomOortwijn identified criteria used prioritise health mendations assessments influence what done, they can improve the technology andis categorised these into four overall categories: health of the population but have no impact on broad burden of disease, potential effects, inequities, reduce inequities or exacerbate them regardless potential costs, and uncertainty regarding application of of the overall effects the technology [15].on population health. There has been a growing interest in taking equity into consideration in clinical practice guidelines However, In a more recent selective [7,8]. review for the consideration New Zealand of inequities has generally been lacking [7]. For example, Guidelines Group, the following criteria were identified as AGREE and other instruments for assessing the quality of indicating that a topic is suitable for guideline developguidelines ment [16]: do not include items on equity or the fairness of the recommendations [9].

What processes should be used to agree on priorities? What is doing now conducted 10 years ago Batista WHO and Hodge in a review "WHO has embraced the elimination health setting inequities found only three articles pertinent toof priority for as an important target and supports the dual of clinical practice guidelines [10]. They suggested goals the folequity and efficiency for health services. WHO's data gathlowing framework for priority setting: ering on inequalities in health status and access to services is andend in turn informs its advocacy and norma1. shaped Consultbywith users and other stakeholders before tive activities that aim to reduce health inequities. Besides selecting topics. collecting relevant data broken down by group, WHO attempts both to relateduring these data to health determinants 2. Consider feasibility the consultation. (e.g., membership in less privileged social groups and exposure to various hazards) to develop and topics. dissem3. Document the process usedand to select guideline inate interventions to improve conditions for members of such groups" [11]. the following procedures [14]: The IOM suggested

Nonetheless, we are not aware of anyfor specific documents • the use of Delphi-like procedures obtaining expert that provide as to how equity should be taken judgments orguidance topic rankings through correspondence into account in WHO guidelines or recommendations or of anyuse studies or descriptions of current practice. The • the of questions that are specific, explicit and conWHO guidelines for guidelines do not currently provide sistent with standard methods for questionnaire construcany tion explicit advice regarding how to take account of equity. • experimentation with more formal procedures to arrive What organisations are doing at groupother judgments Clinical practice guidelines typically focus on the effectiveness ofsuggested interventions to a recommenThey also there(Will is a adherence need to define more nardation do more good than harm?), occasionally on costrowly and precisely topics for guideline development. effectiveness (Are the net benefits worth the costs?), and They argued that this would result in more efficient organrarely on equity (Are the recommendations fair?) [7]. ization of panels and their work, resolution of some More recently, several guideline developers have begun to apparent controversies, more responsive guidelines, and consider equity explicitly and systematically, including, easier implementation. for example, the Australian NHMRC [7], INCLEN [8], the GRADE Working Group, and the National Institute for

1. The topic is clinically important affecting large numbers of In thiswith paper we address the following questions: people substantial morbidity or mortality (the burden of illness). • When and how should inequities be addressed in systematic reviews that are usedto as background documents 2. The topic is complex enough initiate debate about the recfor recommendations? ommendations.

Page 4 2 of 7 6 (page number not for citation purposes)

the use of 4:24 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/24

Health and Clinical Excellence (NICE) in the UK, which now has an extended mandate including public health guidance and reducing health inequalities, after the Health Development Agency (HDA) became part of NICE in 2005 [12]. The HDA was established in 2000 to develop the evidence base to improve health and reduce health inequalities. It worked in partnership with professionals and practitioners across a range of sectors to translate that evidence into practice. Other countries that have had a major political commitment to reducing inequities in health include the Netherlands [13], Thailand, and Chile [14].

very much smaller one describing interventions that could reduce inequalities [24]. When and how should inequities be addressed in systematic reviews that are used as background documents for recommendations? Evidence of the effects of interventions on inequities is sparse and difficult to search for [25]. For example, Tsikata and colleagues found that only 10% of controlled trials assessed the efficacy of the intervention across socioeconomic subgroups [26]. Similarly, Ogilvie and colleagues found that in Cochrane reviews of controlled studies of tobacco control both the reviews and the primary studies in those reviews rarely assessed the impact of the intervention across socioeconomic factors [27]. Systematic reviews tend not to provide evidence on differential effectiveness [27-33]. Searches of electronic databases in many fields, particularly for social interventions and more upstream interventions, may miss much relevant evidence [31-33]. Publication bias may be a problem [25]. Because there is limited direct evidence of differential effects of interventions across socioeconomic groups, it will generally be necessary to search for and include a wider scope of evidence to support or refute plausible hypotheses of differential effects, or the effects of interventions on reducing inequities.

Methods

The methods used to prepare this review are described in the introduction to this series [15]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct a full systematic review. We searched PubMed and three databases of methodological studies (the Cochrane Methodology Register [16], the US National Guideline Clearinghouse [17], and the Guidelines International Network [18]) for existing systematic reviews and relevant methodological research that address these questions. We did not conduct systematic reviews ourselves. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

Although there are clear arguments for exploring moderator effects in systematic reviews, subgroup analyses can be misleading both because of inadequate power (resulting in false negative conclusions) and multiple testing (resulting in false positive conclusions) [34-38]. The results observed in subgroups may differ by chance from the overall effect identified by the meta-analysis, and the subgroup findings may not be confirmed by subsequent large trials [36,39]. Paradoxically, the best estimate of the outcome of the intervention in a sub-group may come from discounting the results of the sub-group analysis and using the overall results (Stein's paradox) [36,40]. General guidelines for interpreting subgroup analyses can be applied to subgroup analyses based on socioeconomic factors [40,41].

This paper is based in large part on a workshop on addressing inequities held in Oslo August 31 to September 1, 2005 [19], background documentation for that workshop [20-23], and a reference list generated during and subsequent to the workshop. We searched PubMed using (clinical practice guidelines or public health guidelines) and (equity or equality) and related articles for references [7] and [23]. We searched the Cochrane Methodology Register using equity or equality.

Findings

Our database searches yielded few references and we found few directly relevant empirical methodological studies, consistent with the findings of other reviews [22,23]. For example, the literature search and correspondence with guideline developers worldwide by the NHMRC located no examples of where clinical practice guideline developers explicitly incorporated evidence on socioeconomic position and health into generic guidelines, except for when guidelines were developed for specific disadvantaged sub-populations [22]. This is consistent with the findings of the Health Development Agency in England. They observed that there is a very large literature that describes the problem of inequalities and a

What questions about equity should routinely be addressed by those making recommendations on behalf of WHO? Additional questions that should be considered in relationship to equity include questions about the applicability of the evidence to disadvantaged populations, differences in values, and the implications of these differences. General guidelines for considering the applicability of evidence can be applied to considering the applicability of evidence to disadvantaged populations [42], including differences in absolute effects due to differences in baseline risk. The trade-offs between the benefits and harms of

Page 3 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:24

http://www.health-policy-systems.com/content/4/1/24 http://www.health-policy-systems.com/content/4/1/14

andintervention an the Guidelines mayInternational be different Network because of [8]) differences for existing in the relativereviews systematic or absolute and relevant effectsmethodological of an intervention research or because that address of differences these questions. in values The [8]. answers For example, to the questions if an outcome, are oursuch conclusions as the ability based on to quickly the available returnevidence, to or stay conat work, is more sideration of what important WHOtoand disadvantaged other organisations populations, are this might doing, and logical tip the arguments. balance between the benefits, harms and costs of an intervention (for example antiretrovirals for AIDS) For this review in favour we searched of intervening. PubMedDifferences using (clinical in any pracof theseguidelines tice factors can or result publicinhealth different guidelines) implications and (priority and recommendations setting or settingfor priorities) disadvantaged and related populations articles for or selected specific recommendations references [9,10]. We for addressing searched the inequities Cochrane [8].Methodology Register using priority or priorities. We reviewed the What context information is needed to Conference oninform Priorities website of thespecific 5th International adaptation and[11] decision making in that a specific setting in Health Care and references we had in ourwith files regard [12-15].to impacts on equity? While evidence about the effects of interventions generally comes from global research, it is necessary to take into Findings What criteria should used to establish priorities? account factors in a be specific setting to inform decisions The Institute of Medicine's (IOM) Committee on aboutUS what to do. These factors include each of the followMethods for Setting Priorities for Guidelines ing in relationship to socioeconomic factors: theDeveloppresence ment in modifiers its study of setting for clinical of effect that have priorities been identified in thepractice global guidelines published 1995 argued the to priority setresearch, baseline risk,inutilization andthat access care, and ting shoulditbe open and to defensible Theyrelereccosts.process In addition, is necessary take into[14] account ommended six general criteria:inprevalence, burdenand of vant ethical and legal standards a specific setting, illness, cost of managing the problem, variability in practhe availability of resources to address inequities. tice, potential of a guidelineis to improve outcomes, Although this information beyond thehealth scope of a review and potential of guidelines a guidelineortorecommendations, reduce costs. Because data or international internaused make these judgements is often lacking, tionaltogroups can systematically consider the they needsugfor gested explicit opportunities for important unmeasured these different types of information in specific settings factors to beguidance considered. They further suggestedofseparate and provide regarding the importance obtain"tracks" considering for strategies specific populations, ing such for information andissues practical for doing so conditions or concerns. argued that the rationales for and integrating contextThey specific information into decidesignating tracks should be defined clearly; i.e. sion-makingspecial processes. why they warrant special consideration. They suggested Whatupdating implementation strategies arealso likely needed to that of guidelines should be be considered as a ensure that recommendations are separate "track", taking account ofimplemented issues such asequitably? the need Because disadvantaged have poorer for corrections and the populations availability ofgenerally new evidence. access to care and often receive poorer quality care, organisational changes are25 likely to beused needed to addresshealth ineqOortwijn identified criteria to prioritise uities in health care. Organisational are four also technology assessments and categorisedchanges these into likely tocategories: be necessary to implement interventions broad burden of disease, potential targeted effects, at social costs, determinants of health. Identifying necessary potential and uncertainty regarding application of organisational and barriers and facilitators of the technology changes, [15]. implementing change requires context specific knowledge andadecisions. Nonetheless, general for guidance and Zealand support In more recent selective review the New for what information to consider, possible strategies to Guidelines Group, the following criteria were identified as address common barriers and facilitators, anddevelopgeneral indicating that a topic is suitable for guideline frameworks ment [16]: for planning organisational changes and implementation strategies can be provided internationally. countries with important pervasive affecting inequities institutional, 1. TheIntopic is clinically large numbers of cultural andsubstantial political changes first be needed. people with morbiditymay or mortality (the burden of illness). Similarly, although local data are needed to monitor the effects of implementing recommendations, 2. The topic is complex enough to initiate debateguidance about the can recbe provided regarding appropriate indicators of social graommendations.

dients and measuresofof change between (e.g. in the ratio ofappropriquintile 3. There is evidence variation actual and 1 tocare. 5, or concentration indices) to use in order to moniate tor the effects of implementing recommendations on disadvantaged on available changes toinuse.social 4. There are nopopulations existing valid and guidelines gradients. Because the evidence for interventions to reduce will amount commonly be weak, it is generally 5. Thereinequities is an adequate of existing evidence available. important to ensure that monitoring and evaluations are as rigorous as possible towill ensure that intended are 6. The recommendations be acceptable to theeffects potential achieved and unintended adverse effects are avoided. users. What 'maps' are available of the isdifferent of 7. Implementation of the guideline feasible,dimensions will not exhausinequity tively uselocally? the communities' resources, and barriers to clinical Equity are andnot inequity are not change so high that they one-dimensional cannot be overcome.phenomena. They consist of a number of dimensions that include economic status, occupation, gender,used ethnicity, class, While burden of disease is commonly as a criterion caste, religion, status disability, resfor priority setting, it grouping, should beage, noted that the place use ofofsumidence,burden geographical location, and manifest orienmary of disease measures, such sexual as disability tation. These different dimensions of varying salience adjusted life years (DALYs) has beenare criticised for focusing in any givenrather social context. For example caste and religion on disease than resource use and interventions, are moreoffrequently significant invalues pre industrial because the assumptions about inherent systems in such while occupation tendsoftothe betechnical dominant in industrial sysmeasures, and because limitations of such tems. It is(see alsofor important note that[17] theand importance measures exampleto references [18]). of these various dimensions relative to each other also varies, What should be used agree on priorities? as theprocesses dimensions overlap andtooverlay each other. The Batista and Hodge in a review ago health effects of inequities are a conducted product of 10 the years interplay found three dimensions. articles pertinent to priority setting for of theseonly different It is therefore important to clinical guidelines They suggested the foldescribepractice systematically the [10]. dimensions, and if possible lowing framework for in priority setting: their relative salience, any given social arrangement.

1. Consult with end users and other stakeholders before Discussion Inequitiestopics. are rarely addressed in clinical practice guideselecting lines. Evidence of the effects of public health and health policy interventions reducing inequities is generally 2. Consider feasibilityon during the consultation. weak or lacking [43]. As a consequence, advice regarding how to address in recommendations to a 3. Document theinequities process used to select guidelinemust topics. large extent rely on the application of general methodological studies and principles, for example in relationship The IOM suggested the following procedures [14]: to subgroup analyses and applicability. While addressing inequities is aDelphi-like fundamental concern atfor theobtaining heart of WHO's â&#x20AC;˘ the use of procedures expert mission, ator present there appears to be inadequate guidjudgments topic rankings through correspondence ance on how best to do this in developing and implementing â&#x20AC;˘ the userecommendations. of questions that are specific, explicit and consistent with standard methods for questionnaire construcAlthough we have not found empirical descriptions of tion WHO's current practices, it is reasonable to assume that inequities are notwith being addressed systematically and â&#x20AC;˘ experimentation more formal procedures to arrive transparently. This assumption rests in part on documenat group judgments tation that WHO guidelines generally have not adhered to standards such as AGREE maymore be more They also suggested there is[44,45]. a need WHO to define narlikely to inequities thanguideline many other organisarowly andaddress precisely topics for development. tions,argued given its However, available evidence They thatmission. this would result inthe more efficient organsuggests of thatpanels inequities generally well addressed in ization and are their work, not resolution of some most systematic reviews and practice guidelines. It apparent controversies, moreclinical responsive guidelines, and is onlyimplementation. recently that attention has been given to the metheasier

Page 4 of 7 6 (page number not for citation purposes)

the use of 4:24 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/24

ods used to address inequities, both for clinical and public health interventions [7,21,46].

Further work

Although we have not conducted a systematic review of the relevant literature, a more systematic review is not likely to have results or implications that are substantially different, given the sparseness of methodological research in this area. This assumption is supported by the NHMRC review [7] and a NHS HTA review of addressing equity in economic analyses [23]. However, growing attention is being paid to this area and there are areas of research that can further inform specific issues, such as the selection of indicators of socioeconomic status in relationship to specific interventions or conditions. Thus, while we do not believe that WHO should undertake further work at this time, it would be valuable for WHO or others to undertake and keep up-to-date systematic methodology reviews that address specific aspects of how to address inequities in systematic reviews, guidelines and recommendations.

7. 8. 9. 10. 11. 12. 13. 14.

Competing interests

15.

16. 17. 18. 19.

20.

Authors' contributions

ADO prepared the first draft of this review. HJS and AF contributed to drafting and revising it.

21.

Acknowledgements

22.

We would like to acknowledge the participants in the workshop on addressing inequities held in Oslo, August 31 to September 1, 2005, who contributed to much of the thinking reflected in this review, and the Norwegian Directorate for Health and Social Welfare, which funded the workshop. Participants in the workshop included Inday Dans, Tony Dans, Mike Kelly, Sally McIntyre, Trevor Sheldon, Peter Tugwell. We would also like to thank Mark Petticrew and Elizabeth Waters for their contributions. We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper and the others in this series.

23. 24.

25.

26.

27.

References 1. 2.

28.

Braveman PA, Gruskin S: Defining equity in health. J Epidemiol Community Health 2003, 57:254-8. Whitehead M: The concepts and principles of equity in health. Int J Health Serv 1992, 22:429-45.

29.

World Health Organization: World Report on Knowledge for Better Health. Geneva: WHO; 2004:114. Marmot M: Social determinants of health inequalities. Lancet 2005, 365:1099-104. Wilkinson R, Marmot M, ed: Social determinants of health. In The solid facts Copenhagen:WHO; 1998:308. Kahn Katherine L, Marjorie Pearson, Harrison Ellen R, Rogers William H, Brook Robert H, Katherine Desmond, Keeler Emmett B: Analysis of Quality of Care for Patients Who Are Black or Poor in Rural and Urban Settings. RAND Document 1993, xviii:139. Aldrich R, Kemp L, Williams JS, Harris E, Simpson S, Wilson A, McGill K, Byles J, Lowe J, Jackson T: Using socioeconomic evidence in clinical practice guidelines. BMJ 2003, 327:1283-5. Dans AM, Dans LF, Oxman AD, Robinson V, Acuin J, Tugwell P, Dennis R, Kang D: Addressing inequities in clinical practice guidelines. 2003 in press. The AGREE Collaboration [http://www.agreecollaboration.org/] Sch端nemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 13. Adaptation, applicability and transferability. Health Res Policy Syst 2006. World Health Organisation. Equity [http://www.who.int/trade/ glossary/story024/en/print.html] Littlejohns P, Kelly M: The changing face of NICE: the same but different. Lancet 2005, 366:791-4. Mackenbach JP, Stronks K: A strategy for tackling health inequalities in the Netherlands. BMJ 2002, 325:1029-32. Moynihan R: Using Health Research in Policy and Practice: Case Studies from Nine Countries. New York: Milbank Memorial Fund; 2004. Oxman AD, Sch端nemann HJ, Fretheim A: Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst 2006. Cochrane Methodology Register [http://www.cochrane.org/ access_data/cmr/accessDB_cmr.asp] National Guideline Clearinghouse [http://www.guidelines.gov] Guidelines International Network (GIN) [http://www.g-in.net] Dans AM, Dans LF, Kelly M, McIntyre S, Oxman AD, Sheldon T, Tugwell P: Oslo workshop on addressing equity in systematic reviews, health technology Assessments, Clinical Practice Guidelines and policy briefs. Unpublished report. . Methodological Problems in Constructing the Evidence Base in Public Health (Mike Kelly, Catherine Swann, Amanda Killoran, Bhash Naidoo, Elaine Barnett-Paige, & Antony Morgan. Health Development Agency Public Health Evidence Steering Group Methodology Reference Group. Starter Paper. 9 August 2002. . Systematic Reviews of Health Promotion and Public Health Interventions (Nicki Jackson, Elizabeth Waters et al. The Cochrane Health Promotion and Public Health Field. February 2005.). . Using socioeconomic evidence in clinical practice guidelines (NHMRC 2002) [http://www.nhmrc.gov.au/publications/synopses/ cp89syn.htm] Sassi F, Archard L, Le Grand J: Equity and the economic evaluation of healthcare. Health Technol Assess 2001, 5(3):1-138. Millward L, Kelly MP, Nutbeam D: Public Health Interventions Research: The Evidence. 2003 [http://www.publi chealth.nice.org.uk/page.aspx?o=502583]. London: Health Development Agency Howes F, Doyle J, Jackson N, Waters E: Evidence-based public health: The importance of finding 'difficult to locate' public health and health promotion intervention studies for systematic reviews. J Epidemiol Community Health 2004, 26:101-4. Tsikata S, Robinson V, Petticrew M, Kristjansson E, Moher D, McGowan J, et al.: Is health equity considered in systematic reviews of the Cochrane Collaboration? 11th Cochrane Colloquium, Barcelona, Spain; 2003. Ogilvie D, Petticrew M: Smoking policies and health inequalities. Tobacco Control 2004, 13:129-31. Mackenbach J: Tackling inequalities in health: the need for building a systematic evidence base. Journal of Epidemiology and Community Health 2003, 57:162. Gruen R, Bailie R, McDonald E, Weeramanthri T, Knight S: The potential of systematic reviews to identify diversity and ineq-

Page 5 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:24

http://www.health-policy-systems.com/content/4/1/24 http://www.health-policy-systems.com/content/4/1/14

in health care interventions.Network XI Cochrane Colloquium: Eviand uity the Guidelines International [8]) for existing dence, health care and culture; Barcelona; 2003. systematic reviews and relevant methodological research 30. Tsikata S, Robinson V, Petticrew M, Kristjansson B, Moher D, that McGowan address these The systematic answers toreviews the questions J, et al.:questions. Do Cochrane contain useful informationbased abouton health equity? XIevidence, Cochrane Colloare our conclusions the available conquium: Evidence, health care, and culture; 2003. sideration what V, WHO otherM: organisations are 31. Ogilvie D,ofHamilton Egan M,and Petticrew Systematic reviews of health effectsarguments. of social interventions: 1. Finding the evidoing, and logical

3. There is evidence of variation between actual and appropriate care. 4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available.

dence: how far should you go? J Epidemiol Community Health 2005, 59:804-8. For this review we searched PubMed usinginterventions (clinical prac32. Thomson H, Petticrew M, Morrison D: Housing and health – a or systematic review.guidelines) British Medical 2001, tice guidelines public health andJournal (priority 323:187-90. setting or setting priorities) and related articles selected 33. Thomson H, Petticrew M, Douglas M: Health impactfor assessment references [9,10]. We searched the Cochrane Methodolof housing improvements: incorporating research evidence. of Epidemiology and Community Health 2003, ogy Journal Register using priority or priorities. We57:11-6. reviewed the 34. Cooper H, Hedges L, editors: The handbook of research syntheth International Conference on Priorities website of the 5 sis. New York: Russell Sage Foundation; 1994. 35. ShadishCare W: Meta-analysis and the exploration of causal in Health [11] and references that we had in ourmedifiles ating processes: A primer of examples, methods, and issues. [12-15]. Psychological Methods 1996, 1:47-65. 36. Davey Smith G, Egger M, Phillips A: Beyond the grand mean? British Medical Journal 1997, 315:1610-4. Findings 37. Thompson S: Why and how sources of heterogeneity should What shouldSystematic be usedReviews to establish becriteria investigated. in Health priorities? Care: Meta-analysis in 2001:157-175. The context US Institute of Medicine's (IOM) Committee on 38. Alderson S, Higgins JPT, (eds): Methods forP, Green Setting Priorities forInvestigating GuidelinesheterogeneDevelopity. Cochrane Reviewers' Handbook 4.2.3 [updated Novemmentber in2004]; its study of setting for clinical practice Section 4. In Thepriorities Cochrane Library Issue 2 Chichester, UK: Johnpublished Wiley & Sons, 2005.argued that the priority setguidelines inLtd; 1995 39. Oxman AD, Guyatt GH: A consumers guide to subgroup analyting ses. process beMedicine open and Annalsshould of Internal 1992,defensible 116:78-84. [14] They recommended general criteria: prevalence, 40. Oxman AD,six Guyatt G: When to believe a subgroupburden analysis. of In Users' Guide the Medical Literature. A Manualvariability for Evidence-Based Cliniillness, cost oftomanaging the problem, in praccal Practice Edited by: Guyatt G, Rennie D. Chicago: AMA Press; tice, 2002:553-65. potential of a guideline to improve health outcomes, 41. S, Whitley E, Peters T, to Mulheran P, Egger Davey Smith G: and Brookes potential of a guideline reduce costs.M,Because data analyses in randomised controlled trials:they quantifyusedSubgroup to make these judgements is often lacking, suging the risks of false positives and false negatives. Health Techgested explicit opportunities important unmeasured nology Assessment 2001, 5:for [http://www.hta.nhsweb.nhs.uk/ htapubs.htm]. factors to be considered. They further suggested separate 42. Dans A, McAlister F, Dans L, Richardson WS, Straus S, Guyatt G: "tracks" for considering issues forpatients. specificInpopulations, Applying the results to individual Users' Guides to conditions or Literature concerns. They theD. rationales for the Medical Edited by: argued Guyatt G,that Rennie Chicago: AMA Press; 2002:369-84. designating special tracks should be defined clearly; i.e. 43. Macintyre S, Chalmers I, Horton R, Smith R: Using evidence to why inform they warrant special suggested health policy: caseconsideration. study. BMJ 2001, They 322:222-5. 44. Oxman AD, of Lavis J, Fretheim A: The use of research evidence that updating guidelines should also be considered asina WHO recommendations. . separate "track", taking account of issues such as the need 45. Panisset U: A Review of WHO recommendations published in for corrections and the availability of new evidence. 2005. . 46. Kelly M, Swann C, Killoran A, Naidoo B, Barnett-Paige E, Morgan A: Methodological Problems in Constructing the Evidence Base Oortwijn identified 25 criteria used to prioritise in Public Health. Health Development Agency Public health Health Evidenceassessments Steering Group technology andMethodology categorised Reference these intoGroup. four Starter Paper. 9 burden August 2002. . broad categories: of disease, potential effects,

potential costs, and uncertainty regarding application of the technology [15]. In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]:

disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation.

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page 4 6 of 7 6 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation Holger J Schünemann*1, Atle Fretheim2 and Andrew D Oxman2 Address: 1INFORMA/CLARITY Research Group, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 2Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway Email: Holger J Schünemann* - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@kunnskapssenteret.no; Andrew D Oxman - oxman@online.no * Corresponding author

Published: 08 December 2006 Health Research Policy and Systems 2006, 4:25

Received: 07 April 2006 Accepted: 08 December 2006

doi:10.1186/1478-4505-4-25

This article is available from: http://www.health-policy-systems.com/content/4/1/25 © 2006 Schünemann et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Page 1 of 10 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:25

http://www.health-policy-systems.com/content/4/1/25 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines • The International preparation ofNetwork systematic [8]) reviews for existing and evidence profiles should be centrally, in and appropri3. There is evidence of coordinated variation between actual systematic reviews collaboration and relevant with organizations methodological that research produce systematic ate care.reviews. Centrally developed evidence that address these profiles questions. should be The adaptable answerstotospecific the questions local circumstances. are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. • Consideration should be given to models that involve central coordination with work being sideration of what WHO and other organisations are undertaken by centres located throughout the world. doing, and logical arguments. 5. There is an adequate amount of existing evidence available. • While needs, availability of resources, costs, the presence of modifying factors and values need to bewe assessed locally, support for undertaking these assessments may be neededwill to make guidelines For this review searched PubMed using (clinical prac6. The recommendations be acceptable to the potential tice guidelinesapplicable. or public health guidelines) and (priority users. setting or setting priorities) andprovide related articles for selected • WHO should local support for adapting and implementing recommendations by references [9,10]. We searched the Cochrane Methodol7. Implementation the guidelineand is feasible, developing tools, building capacity, learning from international ofexperience, throughwill not exhausogy Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical international networks that support evidence-informed health policies, such as the EvidenceConference on Priorities website of theinformed 5th International change are not so high that they cannot be overcome. Policy Network (EVIPNet). in Health Care [11] and references that we had in our files How should recommendations be adapted? While burden of disease is commonly used as a criterion [12-15]. for priority setting, itrecommendations. should be noted that the use of sum• WHO should provide detailed guidance for adaptation of international mary burden of disease measures, such as disability Findings • Local adaptation processes should be systematic and transparent, they should involve What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing stakeholders, and they should report the key factors that influence decisions, including those The US Institute of Medicine's (IOM) Committee on on disease rather than resource use and interventions, flagged in international guidelines, and the reasons for any modifications that are made. Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of such guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). ting process should be open and defensible [14] They recquality of patient care be andused public health Background What processes should to agree onthroughout priorities? the ommended six general criteria: prevalence, burden of The World Organization (WHO), like many world addition WHO, an increasing of Batista[5]. andInHodge in to a review conducted 10number years ago illness, costHealth of managing the problem, variability in other pracorganisations the world, has recognised the need organizations develop guidelines. Some of thesesetting organizafound only three articles pertinent to priority for tice, potential around of a guideline to improve health outcomes, to use more rigorous processes to ensure thatBecause health data care tions target international thissuggested article wethe define clinical practice guidelinesusers. [10].For They foland potential of a guideline to reduce costs. recommendations informed by the best they available the termframework "international" as "concerning lowing for priority setting: or belonging to at used to make these are judgements is often lacking, sugresearch evidence. This is the for thirteenth of aunmeasured series of 16 least two or all nations". Among organizations that gested explicit opportunities important reviews that have been prepared as background for advice develop guidelines, broadest mandate and 1. Consult with endWHO users has andthe other stakeholders before factors to be considered. They further suggested separate from the WHO Advisory Committee on Health Research spectrum of international consumers and stakeholders, selecting topics. "tracks" for considering issues for specific populations, to WHO onorhow to achieve given 192 countries are members of WHO and the scope conditions concerns. Theythis. argued that the rationales for of WHO's responsibilities. 2. Consider feasibility during the consultation. designating special tracks should be defined clearly; i.e. Adaptation involves modification according to different why they warrant special consideration. They suggested circumstances environmental [1]. In as thea Developing internationally challenges 3. Documentguidelines the process used to selectposes guideline topics.to that updating oforguidelines shouldconditions also be considered context of guidelines, it relies on judgments of whether a ensure and monitor that WHO's guidelines are locally separate "track", taking account of issues such as the need guideline is applicable (i.e. relevant to a local setting in a applicable or adaptable across different settings. The IOM suggested the following procedures [14]: The for corrections and the availability of new evidence. specific setting) or transferable from one setting to needs for adapting guidelines identified in the US survey another. survey of managed plans in the US health found referred are evenprocedures greater for for international • the usetoofabove Delphi-like obtaining guideexpert OortwijnAidentified 25 criteriacare used to prioritise that they relied on national other published lines. Organisations such as through WHO that develop internajudgments or topic rankings correspondence technology assessments and and categorised these intoguidefour lines ascategories: referencesburden for their guidelines. However, tional guidelines need to consider variations in the broad of own disease, potential effects, most of the surveyed plans did not adopt published contexts the guidelines will be applied, • the useinofwhich questions that are specific, explicit including and conpotential costs, and uncertainty regarding application of guidelines "as is" and adapted them for a variety of readifferences in needs, valuesfor and the availability of sistent with standard methods questionnaire constructhe technology [15]. sons [2]. The main reasons were lack of local clinical resources. In this paper we addressed the following question input, inappropriate consideration to tions: In a more recent selective review of forresources, the Newfailure Zealand apply to a specific population, too extensive recommen• experimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as dations, a high of complexity in guidelines users, •atShould WHO develop international recommendations? group judgments indicating that level a topic is suitable for guideline for developand failure to include the most recent information in ment [16]: guidelines. •They What should be done centrally and to locally? also suggested there is a need define more narrowly and precisely topics for guideline development. 1. The topic is clinically important affecting large numbers of Article II of substantial the World morbidity Health Organization (WHO) Con•They How should recommendations adapted? argued that this would result be in more efficient organpeople with or mortality (the burden of stitution defines "setting, validating, monitoring and purization of panels and their work, resolution of some illness). suing the proper implementation of norms and Questions related to guideline implementation and and disapparent controversies, more responsive guidelines, standards" as core functions of the WHO [3]. Accordingly, semination are specifically addressed in another paper in easier implementation. 2. The topic is complex enough to initiate debate about the recWHO issues guidelines (for example [4]) that are being this series [6]. ommendations. used in many countries with the aim of improving the

Page Page24ofof10 7 (page number not for citation purposes)

the use of 4:25 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/25

What WHO is doing now

For example, the Global Strategy on Diet, Physical Activity and Health states that "the purpose of the Regional Consultations with Member States is for countries in each region to provide information on the extent of the problem associated with diet, physical activity and chronic disease, and appropriate prevention strategies for their particular countries. The consultation will focus on the discussion of national, regional and international interventions that will be effective within individual countries and that will take account of national, social, cultural and economic realities. Regional differences, common concerns, or international consensus, will be noted and serve as the basis of the development of the Global Strategy. This consultation process will build on past and current activities and programmes on the issue carried out by WHO Regional Offices and by Member States [10]."

The Guidelines for WHO Guidelines (GWG) note that "Governments have as their main responsibility the health of the population, rather than the disease of the individual, and must consider other factors in addition to the traditional concern for maximizing the benefit to individual patients. WHO needs to assess the implications for population health of any recommendation as well. This requires explicit recognition that resources to provide health interventions (including diagnostic procedures, pharmaceuticals, surgical interventions and psychosocial techniques) are limited. This involves considering the cost-effectiveness of alternative interventions, the opportunity costs of investing in one intervention versus another, the affordability of the interventions, and the feasibility of applying a set of recommendations in different settings [7]."

WHO also provides funding and support for specific guideline adaptation efforts. Specific WHO guideline adaptation projects exist in the area of HIV that have been supported by international workshops organized by regional offices with country involvement [11,12]. Another example is a WHO-sponsored conference by the International Council of Ophthalmology on local adaptation of clinical practice guidelines in China [13]. However, while WHO is developing international guidelines through a variety of efforts, few WHO groups are using systematic and transparent processes that facilitate judgements regarding their applicability and transferability or provide guidance about how to adapt the guidelines [14].

The GWG also recognizes that "WHO takes a global perspective in addressing the needs of (192) member states. Differences in outcome will not only be due to transferring results from a research to a field setting, but also from the different cultural, economic, socio-demographic contexts present in the member-states," and that for "WHO guidelines, the traditional approach of reviewing and reporting evidence on efficacy and safety is certainly crucial but not sufficient. It can be regarded as the first step, but it is also necessary to examine the implications of applying each possible set of recommendations on a population basis. The initial body of evidence to be considered in WHO guidelines will be identical to that of traditional guidelines, but WHO guidelines will need to go further, to take the second step of spelling out the implications of adopting recommendations on costs and on population health. If done adequately, this will allow decision makers in different settings to take the third step of "localizing" the guidelines to their settings, and deciding where the trade-off between additional benefit and additional costs should be set. It will also be useful in determining what is acceptable for the end-users." However, one of the limitations of the GWG is that they have not been operationalized or implemented consistently. Only few WHO guideline processes have followed the GWG [8].

What other organisations are doing?

We are not aware of published surveys that address what other organisations do to ensure appropriate adaptation of guidelines. The items that are part of the AGREE instrument [15] include the following three items most relevant for the assessment of guideline applicability: â&#x20AC;˘ The potential organisational barriers in applying the recommendations should be discussed. â&#x20AC;˘ The potential cost implications of applying the recommendations should be considered. â&#x20AC;˘ The guideline should present key review criteria for monitoring and audit purposes.

Indeed, the need for international development with local adaptation is expressed in WHO statements such as "The strategy recommends a prevention-oriented approach that emphasizes the need for countries to develop coherent, multi-sectoral national strategies with a long-term, sustainable perspective, to make the healthy choices the preferred alternatives at both the individual and community level. We welcome the commitment shown by Member States to the strategy and will be working closely with them to help them implement its recommendations [9]."

The Conference on Guideline Standardization (COGS) checklist for reporting clinical practice guidelines suggests that guidelines should: "Describe the intended users of the guideline (e.g., provider types, patients) and the settings in which the guideline is intended to be used" [16]. The checklist does not include any specific recommendations related to supporting judgements about the applica-

Page 3 of 10 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:25

http://www.health-policy-systems.com/content/4/1/25 http://www.health-policy-systems.com/content/4/1/14

and theorGuidelines bility transferability International of guidelines, Network [8]) or their for existing local adaptation.reviews and relevant methodological research systematic that address these questions. The answers to the questions There are ouris,conclusions however, a based growing on interest the available in considerations evidence, conof how to adapt sideration of guidelines what WHO [17-22]. and other This interest organisations is drivenare by several and doing, factors, logical including arguments. a desire to reduce unnecessary duplication of efforts across organisations, limited resources For this review for many we searched organisations, PubMed particularly using (clinical in lowpracand middle-income tice guidelines or countries public health (LMIC), guidelines) and concerns andabout (priority the sustainability setting or setting of programs priorities)that and are related wellarticles resourced. for selected references [9,10]. We searched the Cochrane MethodolIn this background section or wepriorities. provide selected examples ogy Register using priority We reviewed the International Conference on Priorities of organizations have specifically provided informawebsite of the 5ththat in Health Careto[11] references we thatposed we had in our files tion relevant the and key questions although the [12-15]. WHO is unique in that its mandate includes the more complex task of providing international guidance.

3. forThere otherisreasons evidenceisofperceived variationas between being actual weakerand than appropria simate ple care. evaluation of the methodology would suggest. In other areas, the appropriate action may be inclusion in the 4. guideline There are ofno a commentary existing valid guidelines on the main available economic to use.issues that should be considered in relation to the subject of the 5. guideline There is (for an adequate exampleamount [25]). Another of existing option evidence is the available. provision of basic information that will allow guideline users to 6. work Theout recommendations the resource implications will be acceptable for their to the ownpotential service users. (for example [26]). New Zealand Guideline Group is feasible, will not exhaus7. Implementation of the guideline The process by the New tively use therecommended communities' resources, andZealand barriers Guideline to clinical Group includes following (figure 1) when adaptchange are not sothe high that they steps cannot be overcome. ing overseas evidence-based guidelines [27]: While burden of disease is commonly used as a criterion • Appraise the guidelines (using AGREE for for priority setting, it should be noted thatinstrument) the use of sumqualityburden and process mary of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing • Analyse content forresource scope and applicability on diseasethe rather than use and interventions, because of the assumptions about values inherent in such � Same health professional groups? measures, and settings, because of the technical limitations of such measures (see for example references [17] and [18]). � Same patients, consumers? What processes should be used to agree on priorities? Batista and Hodge in a review conducted 10 years ago � Same interventions? found only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the fol� Same outcomes? lowing framework for priority setting: • Look at the gaps in the issues covered in the overseas 1. Consult with end users and other stakeholders before guidelines selecting topics. � Any clinical questions not covered? 2. Consider feasibility during the consultation. � Look at the sources of evidence 3. Document the process used to select guideline topics. � Is the search strategy available? The IOM suggested the following procedures [14]: � Are there any evidence tables? • the use of Delphi-like procedures for obtaining expert judgments or topic statements rankings through correspondence ref• Are the evidence and recommendations erenced? • the use of questions that are specific, explicit and consistent withthe standard construc• Re-run searchmethods strategy for to questionnaire include the questions tion selected

Findings

What criteria should be used to establish priorities? SIGN The Scottish US Institute Intercollegiate of Medicine's Guideline (IOM)Network Committee (SIGN) on asks guideline Methods for Setting panels Priorities to consider for issues Guidelines of applicability Developwhen in ment guideline its studygroups of setting summarize priorities their forview clinical of the practice total body of evidence guidelines published [23].inThe 1995 guidelines argued that are graded the priority to differsetentiate ting process between should those bebased open on andstrong defensible evidence [14]and They those recbased on weak ommended six evidence. general criteria: This judgement prevalence, is made burden on the of basis ofcost illness, a transparent of managing assessment the problem, of the variability design andin quality pracof each tice, potential study of but a guideline also a judgement to improve onhealth the consistency, outcomes, clinical and potential relevance of aand guideline external to validity reduce costs. of theBecause whole body data of evidence. used to makeThe these aimjudgements is to produce is aoften recommendation lacking, they sugthat is evidence-based, gested explicit opportunities but whichfor is important relevant tounmeasured the way in which health factors to be considered. care is delivered Theyinfurther Scotland suggested and is therefore separate implementable. "tracks" for considering The following issues for specific populations, points are included inorthe conditions concerns. described Theyconsidered argued that judgments the rationales SIGN for panels are asked designating special to make: tracks should be defined clearly; i.e. why they warrant special consideration. They suggested • Generalisability that updating of guidelines of study findings should also be considered as a separate "track", taking account of issues such as the need • Directness for corrections of application and the availability to the target of new population evidence.for the guideline. Oortwijn identified 25 criteria used to prioritise health • Clinical impact technology assessments (i.e. theand extent categorised of the impact theseon into thefour target patient broad categories: population, burden andof thedisease, resources potential needed to effects, treat them.) costs, and uncertainty regarding application of potential the technology [15]. • Implementability (i.e. how practical it would be for the NHS In a more in Scotland recenttoselective implement review the recommendation.) for the New Zealand Guidelines Group, the following criteria were identified as SIGN guideline indicating that adevelopment topic is suitable groups forare guideline provided developwith a form in ment [16]: which to record the main points from their considered judgement [24]. Once they have considered these issues, 1. The topic the group is clinically is asked important to summarise affecting itslarge view numbers of the eviof dence and people with assign substantial a level morbidity of evidence or mortality to it, before (the burden going on of to derive a graded recommendation. During this process illness). SIGN guideline developers are also able to downgrade a recommendation 2. The topic is complex if they enough think tothe initiate evidence debateisabout not generalthe recisable, not directly applicable to the target population, or ommendations.

• experimentation with formal arrive � To include literature at more least one yearprocedures prior to thetodate of at group judgments publishing They also suggested therewould is a need to define more � Check if any large study radically change the narrecrowly and precisely topics for guideline development. ommendations They argued that this would result in more efficient organization of panels planning and their work, resolution of some • Implementation apparent controversies, more responsive guidelines, and easier implementation. � Redesign the implementation plan to meet local circumstances

Page Page44ofof10 7 (page number not for citation purposes)

the use of 4:25 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/25

Figure 1 guidelines using the AGREE instrument (from [23]) Adapting Adapting guidelines using the AGREE instrument (from [23]).

Guide to Community Preventive Services The Guide to Community Preventive Services states the following in a discussion of its methods [28]: "The Guide should not be viewed as the sole source for informed public health decision making because local contextual information is also important. Many issues not addressed in the Guide will affect which interventions are implemented (e.g., resource availability, social justice, community participation, cultural appropriateness, local burden of diseases and risk factors, and political considerations). However, the Guide provides systematically collected and detailed information on several issues of importance to public health practitioners and decision makers; information which is difficult or inefficient to develop locally. Guide reviews and recommendations will be most useful in conjunction with a participatory community planning process that clarifies needs and goals and that considers the Guide's evidence reviews and recommendations in conjunction with additional applicable community specific information."

tional organizations to identify processes that these organizations use to adapt guidelines locally beyond what was known for existing organizations as described in the background section. We also searched PubMed using "guideline" AND "adaptation OR applicability OR template OR transferability" (MESH headings/keywords) for studies and systematic reviews comparing different strategies to increase adaptation, acceptance and transferability (we identified 637 citations of which 203 citations were identified as systematic reviews using the clinical queries filter for systematic reviews). We reviewed the titles of all citations and retrieved abstracts and full text articles if the citations appeared relevant to the topic. We checked the reference lists of articles relevant to the questions and used snowballing as a technique to obtain additional information. We also searched the Cochrane Library and Google for articles and methods related to guideline adaptation ("guideline adaptation"). In addition, we searched databases maintained by the Agency for Healthcare Research and Quality (AHRQ, [30]) and the Guidelines International Network (GIN, [31]). The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

Methods

The methods used to prepare this review are described in the introduction to this series [29]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). For this review we analyzed existing guidelines for guidelines of national or interna-

Findings

We did not identify systematic reviews addressing the key questions. We found individual studies and projects pub-

Page 5 of 10 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:25

http://www.health-policy-systems.com/content/4/1/25 http://www.health-policy-systems.com/content/4/1/14

and theinGuidelines lished the peer reviewed International literature Network and [8]) on the for Internet existing that we willreviews systematic use to illustrate and relevant the responses methodological to the key research questions. that address these questions. The answers to the questions are our conclusions based on the available evidence, conShould WHO international recommendations? sideration of develop what WHO and other organisations are Threatsand of new emerging diseases (e.g. severe acute respidoing, logical arguments. ratory syndrome (SARS) and avian influenza A (H5N1) infection) as well pandemics of chronic such as For this review weassearched PubMed usingdiseases (clinical pracobesity and heart diseasehealth have guidelines) prompted international tice guidelines or public and (priority action and are clear examples the existence of internasetting or setting priorities) andofrelated articles for selected tional health problems and the need for international recreferences [9,10]. We searched the Cochrane Methodolommendations. theorinternational of ogy Register usingGiven priority priorities. Weorientation reviewed the Conference on organizaPriorities WHO and of large international website of the the advantages 5th International in Health [11] and we had inreducing our files tions (e.g.Care accessing andreferences pooling that of resources, [12-15]. unnecessary duplication, and involving international scientists), there is an important role for international recommendations [32]. International recommendations Findings What criteria should be usedvariation to establish priorities? may be most helpful when in settings and local The US Institute of important. Medicine'sTherefore, (IOM) Committee on circumstances is less consideration Methods for Settingbaseline Priorities Guidelines Developof need (prevalence, riskfor or health status), setting ment in its studyofofresources) setting priorities for clinical practice (e.g. availability and modifying factors (facguidelines published in 1995 argued that the priorityinto settors that modify translation of recommendation ting process be open and defensible They recpractice suchshould as microbiological resistance [14] patterns) can ommended six general criteria: the prevalence, burden of be key components that influence strength of a recomillness, cost and of managing problem, in pracmendation should bethe specified in variability recommendations tice, potential to improve health outcomes, formulated byof thea guideline WHO [33]. and potential of a guideline to reduce costs. Because data used to make these judgements is often lacking, they sugAn illustration of the need for adaptation is provided by gested explicit opportunities important unmeasureda Rhinehart and colleagues whofor attempted to implement factors to be infection considered. They program further suggested nosocomial control based onseparate the US "tracks" forDisease considering issues forguidelines specific populations, Centers for Control (CDC) in an urban conditions concerns. They[34]. argued that theofrationales for Indonesianor public hospital Adoption unmodified designating special should be definedfactors clearly;such i.e. CDC guidelines wastracks impeded by modifying why they warrant special consideration. They as conditions of the physical plant, absence of suggested an infecthat of guidelines should be considered as a tion updating control infrastructure, limitedalso sterilization capabiliseparate taking account oflaboratory issues suchsupport, as the need ties, lack "track", of clinical microbiologic and for the use availability new evidence. the corrections expense ofand single medicalofdevices. After on-site evaluations, CDC guidelines were extensively modified so Oortwijn identified 25 criteria used conditions to prioritiseand health that they were appropriate for local cultechnology assessments and categorised these into four ture [34]. After implementation, many physical changes broad categories: burdenand of handling disease, potential effects, had been accomplished, of reusable and potential and uncertainty of disposablecosts, medical devises had regarding improvedapplication considerably the technology [15].of clinical practice policies was incomalthough adoption plete. In a more recent selective review for the New Zealand Guidelines Group, the following identified as Global recommendations shouldcriteria applywere to most settings indicating thatadaptation a topic is to suitable for guideline developyet allow for local circumstances. The facment [16]: tors that influence recommendations should be laid out explicitly. If differences in context are likely to lead to dif1. The topic is clinically important affectingthese large should numbers be of ferent recommendations or decisions, people substantial morbidity or mortality (thebeburden of flaggedwith [33,35]. Contextual issues that should considillness). ered include modifying factors, need, values and resources. Table 1 provides a checklist of factors that influ2. Thethe topic is complex enough to initiate debate about the recence applicability or transferability of guidelines. It ommendations. can be used during the guideline development process to

3. There is evidence of variation actual andsystematiapproprihelp ensure that these factorsbetween are considered ate care. cally and transparently, and to clearly label factors that are important to consider in specific settings where the guide4. There no existing guidelines available to use. line will are be applied or valid adapted. What should done centrally locally? 5. There is an be adequate amount ofand existing evidence available. The research evidence on what reflects the best distribution of recommendations responsibilities during development inter6. The will bethe acceptable to the of potential national guidelines is sparse. Global evidence (i.e. the best users. evidence from around the world) is the best starting point forImplementation judgements about andislikely modifying 7. of theeffects guideline feasible, will not factors. exhausSynthesizing and making resources, available and this barriers evidence tively use the communities' to should clinical be coordinated the work can be change are not socentrally, high that although they cannot beactual overcome. done anywhere. For example, the Agency for Health Care and Quality (AHRQ) funds Evidence-based CenWhile burden of disease is commonly used Practice as a criterion trespriority throughout theitUS and Canada with methodologfor setting, should be noted thatthe the use of sumical competency undertake systematic reviews, the UK mary burden oftodisease measures, such as disability National life Institute for Health andcriticised Clinicalfor Excellence adjusted years (DALYs) has been focusing (NICE) funds National responsible on disease rather thanCollaborating resource use Centres and interventions, for guideline (in areas such as acute because of thedevelopment assumptions about values inherent in care, such cancer andand chronic conditions), and thelimitations Cochrane Collabmeasures, because of the technical of such oration has Collaborative Review spread measures (see50 for example references [17] Groups and [18]). around the world that are responsible for preparing and What processes should be used(in to particular agree on priorities? updating systematic reviews areas such as Batista and Hodge in a diseases, review conducted 10addiction). years ago breast cancer, infectious and tobacco found articles pertinentor to prepare priority systematic setting for WHO only couldthree adapt, commission clinical [10]. They suggested the folreviews practice that are guidelines required for guideline development in lowing framework priority setting: collaboration with for organizations such as these that conduct systematic reviews and follow suggestions to make 1. Consult with end usersfor and other stakeholders before these reviews more useful policymakers [36]. Lavis and selecting colleaguestopics. suggest that donors and international agencies can encourage more informed public policymaking by 2. Considernational feasibility during the consultation. supporting and regional efforts to undertake systematic reviews and assess their local applicability, and by 3. Document the process used to select guideline topics. supporting regional or worldwide efforts to coordinate review and assessment processes [37]. The IOM suggested the following procedures [14]: Similarly, adaptable evidence profiles [33] based on new â&#x20AC;˘ use ofhigh Delphi-like for obtaining expert orthe existing quality procedures systematic reviews that include judgments topic rankings through correspondence informationoron critical outcomes should also be prepared or coordinated centrally. Applying the criteria listed in â&#x20AC;˘ the 1use of flagging questions that are specific, explicit and contable and important factors that influence the sistent with standard methods for questionnaire construcapplicability of guidelines in evidence profiles could faciltion itate local adaptation and help groups to replace the flagged elements with locally appropriate information. â&#x20AC;˘ experimentation with more formal procedures arrive Because resources for guidelines development areto limited, at group judgments particularly in LMIC, support for local adaptation of guidelines should also be supported centrally, by WHO They also suggested there isoffices. a needGiven to define more narheadquarters or by regional that WHO also rowly and precisely for guideline has limited capacity, topics consideration shoulddevelopment. be given to They that this would result in more efficient organdoingargued this collaboratively with other organizations and to ization of panels ande.g. their work, the resolution of some developing capacity; through development of apparent controversies, moreasresponsive guidelines, frameworks and tools, such those being developedand by easier implementation. the International Clinical Epidemiology Network (INCLEN) Knowledge Plus Program [20], GIN [31] and

Page Page64ofof10 7 (page number not for citation purposes)

the use of 4:25 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/25

Table 1: Checklist for identifying guidelines requiring adaptation

Factors influencing the applicability or transferability of guidelines across different settings

Response (positive answers increase the likelihood that recommendations should be flagged as requiring adaptation)

1. Is there important variation in need (prevalence, baseline risk or health status) that might lead to different decisions?

� Yes � Unclear � No � Yes � Unclear � No � Yes � Unclear � No � Yes � Unclear � No � Yes � Unclear � No

2. Is there important variation in the availability of resources that might lead to different decisions? 3. Is there important variation in costs (e.g. of drugs or human resources) that might lead to different decisions? 4. Is there important variation in the presence of factors that could modify the expected effects (e.g. resistance patterns of microbiological pathogens), which might lead to different decisions? 5. Is there important variation in the relative values of the main benefits and downsides that might lead to different decisions?

others; through training; through networks such as the Evidence-Informed Policy Network (EVIPNet) [38], and by learning from the experience of organizations around the world that are engaged in supporting evidenceinformed health policies in specific settings [39].

Decisions during local adaptation processes should be transparent and follow procedures that are similar to those used in developing the guidelines, including reporting the key factors that influence any modifications. Two fairly similar approaches have recently appeared that produced frameworks for identifying candidate guidelines for local adaptation. The Practice Guideline Evaluation and Adaptation Cycle (PGEAC) is a 10 step approach (figure 2) [17,41,42]. Graham and colleagues describe three alternatives in the PGEAC approach: (a) adopt one guideline with all its recommendations; (b) adopt one guideline, endorsing some of its recommendations but not endorsing recommendations that lack strong evidence or cannot be implemented or adapted locally; or (c) take the best recommendations from each of the guidelines and adapt them for inclusion of the new guideline [17]. If recommendations are modified the rationale for changes should be explicitly stated in the resulting local guideline document.

Apparently successful examples of collaboration between central and country level groups exist. Wabitsch and colleagues described that during the adaptation of the WHO global HIV/AIDS guidelines [11] standard techniques involving consensus building were successfully employed to adapt these guidelines to local settings (Malawi and Barbados). The results showed that the process preserved the structure but involved significant modification to the processes of clinical care. Given the factors that influence the formulation of recommendations, the modification of individual recommendations confirms that an adaptation process was required. How should recommendations be adapted? In addition to supporting appropriate adaptation of its own guidelines, WHO should consider adapting guidelines developed by other organizations, given the potential value of WHO endorsement and savings, if high quality guidelines already exist. Detailed guidance on appropriate methods for adapting guidelines would help WHO guideline groups to adapt existing guidelines, when this is appropriate. Adaptation of recommendations is required because several judgments influence recommendations. Therefore, recommendations dealing with identical questions may differ between developers despite reliance on the same evidence. Implementation, which follows the process of adaptation, is topic of another paper in this series and tools for the evaluation of implementation of guidelines have been developed [6,40].

The other approach has been developed by the international working group ADAPTE [18,19,22] and partly overlaps with the PGEAC approach. Recently the groups developing both adaptation approaches merged with the purpose of developing a generic manual on guideline adaptation. The manual will undergo pilot testing. The group is calling itself the ADAPTE group http:// www.adapt.org. Whatever adaptation process is chosen the process should be made explicit, undergo review by peers, and involve consumers, policymakers and other stakeholders who may provide input about policy guidance. Balance sheets or evidence profiles are designed to assist decision-makers regarding outcomes in their practice set-

Page 7 of 10 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:25

http://www.health-policy-systems.com/content/4/1/25 http://www.health-policy-systems.com/content/4/1/14

3. There is evidence of variation between actual and appropriate care.

6. The recommendations will be acceptable to the potential users.

4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available.

Findings

What criteria should be used to establish priorities? The US Institute of Medicine's (IOM) Committee on Methods for Setting Priorities for Guidelines Development in its study of setting priorities for clinical practice guidelines published in 1995 argued that the priority setting process should be open and defensible [14] They recWhat processes should be used to agree on priorities? ommended six general criteria: prevalence, burden of Batista and Hodge in a review conducted 10 years ago illness, cost of managing the problem, variability in pracfound only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, clinical practice guidelines [10]. They suggested the foland potential of a guideline to reduce costs. Because data lowing framework for priority setting: used to make these judgements is often lacking, they suggested explicit opportunities for important unmeasured 1. Consult with end users and other stakeholders before factors to be considered. They further suggested separate selecting topics. "tracks" for considering issues for specific populations, conditions or concerns. They argued that the rationales for 2. Consider feasibility during the consultation. designating special tracks should be defined clearly; i.e. The Practice Figure 2 warrant Guideline Evaluation and Adaptation (PGEAC) (from [17, 41]) why they special consideration. They Cycle suggested The Practice Evaluation andalso Adaptation Cycle (PGEAC) [17, 41]). 3.(from Document the process used to select guideline topics. that updatingGuideline of guidelines should be considered as a separate "track", taking account of issues such as the need The IOM suggested the following procedures [14]: for corrections and the availability of new evidence. ting [43,44]. For guideline adaptation they should include systematically considers needs for local adaptation when data for the specific setting to which are adapted (for developing that it has capacity â&#x20AC;˘ the use ofguidelines; Delphi-likeand procedures forsufficient obtaining expert Oortwijn identified 25 criteria usedthey to prioritise health all the considerations in table 1). During adaptation to support or both developing and supporting the adaptajudgments topic rankings through correspondence technology assessments and categorised these into confour sumer involvement (i.e. to integrate their values and pertion of high priority guidelines. To do this as effectively broad categories: burden of disease, potential effects, spectives) costs, and involvement of other stakeholders should and as possible should explicit collaborate â&#x20AC;˘ theefficiently use of questions thatWHO are specific, and with conpotential and uncertainty regarding application of be emphasized [35,45]. other in developing guidelines and, sistentorganizations with standard both methods for questionnaire constructhe technology [15]. importantly, in developing capacity in LMIC and supporttion ing appropriate adaptation in countries that lack In a more recent selective review for the New Zealand Discussion Given WHO's mandate; limited resources that are availaresources. We provide other recommendations about how â&#x20AC;˘ experimentation with more formal procedures to arrive Guidelines Group, the following criteria were identified as ble to develop quality guidelines that are informed WHO canjudgments improve the implementation of organizational at group indicating that high a topic is suitable for guideline developby the[16]: best available evidence, particularly in LMIC; and changes to guideline development in other articles in this ment the potential to reduce unnecessary duplication, WHO series [6,46-50]. They also suggested there is a need to define more narshould continue to develop rowly and precisely topics for guideline development. 1. The topic is clinically importantinternational affecting large guidelines. numbers of Further workthat this would result in more efficient organHowever, guidelines will or often require They argued people withthese substantial morbidity mortality (theadaptation burden of WHO has adapting guidelines but and tailoring to local contexts and WHO should, so far as ization of ample panels experience and their in work, resolution of some illness). this effortcontroversies, should be more coordinated andguidelines, disseminated possible, provide support to help ensure that internaapparent responsive and amongimplementation. WHO guideline groups. Through coordinating tional guidelines are enough adapted to local cireasier 2. The topic is complex to appropriately initiate debate about the recguideline development within WHO and collaborating cumstances. To do this WHO must ensure that it ommendations.

Page Page84ofof10 7 (page number not for citation purposes)

the use of 4:25 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/25

with other organizations, WHO could capitalise on this experience, improve the quality of its guidelines, and help to ensure that its guidelines are appropriately adapted and result in appropriate actions and health improvements. Similar suggestions have been made previously by investigators involved with WHO guideline projects. For example, "Countries should discuss and find ways of collaboration and formation of linkages and support with National HIV/AIDS program in order to enhance the implementation of IMCI algorithm which includes HIV/ AIDS" [11].

3. 4.

6. 7.

A systematic review of studies evaluating methods for adaptating guidelines is unlikely to retrieve high quality evidence given the paucity of research in this area. Further development and evaluation of frameworks and tools to support the appropriate adaptation of guidelines is needed. Given the limited capacity for this in many countries, comparisons of simpler processes that require fewer resources should be compared with more rigorous processes to determine the most efficient methods for ensuring that guidelines support well-informed decisions and actions appropriate for the specific contexts in which they are taken.

9. 10.

11.

12.

Competing interests

13.

14. 15.

16.

17.

Authors' contributions

HJS prepared the first draft of this review. AF and ADO contributed to drafting and revising it. All authors read and approved the final manuscript.

18. 19.

Acknowledgements We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper and the others in this series. The authors thank Mr. John Fox for administrative assistance and help with organizing the literature review.

20. 21.

22.

References 1. 2.

www.wordreference.com: Word reference dictionary. [http:// www.wordreference.com/definition]. United States General Accounting Office. Report to the Chairman, Subcommittee on Health, Committee on Ways and Means, House of Representatives. Practice Guidelines.

23. 24.

Managed Care Plans Customize Guidelines to Meet Local Interests. 1996, GAO/HEHS-96-95:. World Health Organization. Constitution of the World Health Organization. Off Rec Wld Hlth Org, 2, 100 1948. World Health Organization. Prevention of Recurrent Heart Attacks and Strokes in Low and Middle Income Populations: Evidence-based Recommendations for Policy Makers and Health Professionals. Geneva , World Health Organization; 2003. Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care 2003, 15(1):31-45. Fretheim A Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 15. Disseminating and implementing guidelines. Health Res Policy Syst 2006. Global Programme on Evidence for Health Policy. Guidelines for WHO Guidelines. EIP/GPE/EQC/2003.1. Geneva , World Health Organization; 2003. Schünemann HJ, Hill S, Kakad M, Bellamy R, Uyeki T, Hayden F, Yazdanpanah Y, Beigel J, Chotpitayasunondh T, Vist G, et al.: WHO Rapid Advice Guidelines for the pharmacological management of sporadic human infection with avian influenza A (H5N1) virus. The Lancet Infectious Disease 2007, 7:21-31. World Health Organization (WHO). Global Strategy on Diet, Physical Activity and Health. In Bulletin of the WHO WHA57/A57_R17 Geneva , World Health Organization; 2002. World Health Organization. World Health Organization Process for a Global Strategy on Diet, Physical Activity and Health. WHO/NMH/EXR.02.2 Rev.1. Geneva , World Health Organization; 2003. Wabitsch R, Margolis CZ, Malkin JE, Abu-Saad K, Waksman J: Evaluating the process of tailoring the global HIV/AIDS practice guidelines for use in developing countries. Int J Qual Health Care 1998, 10(2):147-154. Report on the Workshop on Adaptation of IMCI Guidelines to Include HIV AIDS. Harare 18 to 23 June 2001 2001 [http:// www.who.int/child-adolescent-health/New_Publications/HIV/ report_HIV_Harare.htm]. ICO International Clinical Guidelines: Report on COS Preferred Practice Pattern Guidelines Development Workshop Beijing, China, June 24-26, 2005 [http://www.icoph.org/guide/ chinarp.html] Personal communication with WHO Evidence in Policy - no date. . AGREE, Collaboration: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12(1):18-23. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003, 139(6):493-498. Graham ID, Harrison MB, Brouwers M, Davies BL, Dunn S: Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by health care organizations. J Obstet Gynecol Neonatal Nurs 2002, 31(5):599-611. Fervers B, Graham ID: Guideline adaptation - the way forward . In Guidelines International Network Conference 2005 Lyon; 2005. Burgers JS, Cook R: Adaptation of Guidelines . Guidelines International Network: 2005 2005 [http://www.g-i-n.net/download/files/ Rob_Cook___Adaptation_of_guidelines.pdf]. Lyon INCLEN Knowledge ‘Plus’ Program (K+). The INCLEN Trust. [http://www.inclentrust.org] Adapting existing HTAs from one country to other. European Network for Health Technology Assessment - EUnetHTA [http://www.sst.dk/upload/planlaegning_og_behandling/ cemtv/eunethta/wp5description.pdf] Fervers B Burgers JS, Haugh MC, Latreille J, Mlika-Cabanne N, Paquet L, Coulombe M, Poirier M, Burnand B: Adaptation of clinical guidelines: literature review and proposition for a framework and procedure. Int J Qual Health Care 2006, 13:167-176. Yuen KY: Clinical features and rapid viral diagnosis of human disease associated with avian influenza A H5N1 virus. Lancet 1998, 351:467-471. Scottish Intercollegiate Guideline Network. SIGN 50: A guideline developers' handbook. Section 5. :(accessed on

Page 9 of 10 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:25

http://www.health-policy-systems.com/content/4/1/25 http://www.health-policy-systems.com/content/4/1/14

17, 2006) [http://www.sign.ac.uk/guidelines/fulltext/50/ and March the Guidelines International Network [8]) for existing section5.html]. systematic reviews and relevant methodological research 25. Scottish Intercollegiate Guideline Network. Cardiac Rehathat bilitation. address these questions. answers the questions In SIGN Publication The Volume No. 57. to Edinburgh , Scottish Intercollegiate Guideline Network; 2002. are our conclusions based on the available evidence, con26. Scottish Intercollegiate Guideline Network. In Management of sideration ofVolume whatSIGN WHO andNo.other organisations are osteoporosis Publication 71. Edinburgh , Scottish Intercollegiate Guidelinearguments. Network.; 2002. doing, and logical 27.

46. Oxman AD, Fretheim A, Schünemann AD: Improving Use of 3. There is evidence of variation between actual andthe appropriResearch Evidence in Guideline Development: 7. Deciding ate care. what evidence to include. Health Res Policy Syst 2006, 4:19. 47. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of Research Evidence Guideline Development: 4. There are no existinginvalid guidelines available14. to Reporting use. guidelines. Health Res Policy Syst 2006, 4:26. 48. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of Research Guideline Development: Evalua5. There is an Evidence adequate in amount of existing evidence16. available. tion. Health Res Policy Syst 2006, 4:28. 49. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of 6. The recommendations will be acceptable to the potential Research Evidence in Guideline Development: 1. Review of users.guidelines for guidelines. Health Res Policy Syst 2006, 4:13. 50. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 9. Grading 7. Implementation of the guideline is feasible, will notSyst exhausevidence and recommendations. Health Res Policy 2006, tively4:21. use the communities' resources, and barriers to clinical

New Zealand Guidelines Group. Notes on the Adaptation / Synthesis of Guidelines (21-Jan-04) [http://www.nzgg.org.nz/]

For wePappaioanou searchedM,PubMed using (clinical prac28. this Briss review PA, Zaza S, Fielding J, Wright-De Aguero L, Truman BI, Hopkins Mullenhealth PD, Thompson RS, Woolf Carandetice guidelines orDP, public guidelines) andSH,(priority Kulis VG, Anderson L, Hinman AR, McQueen DV, Teutsch SM, Harris setting or setting priorities) and related for selected JR: Developing an evidence-based Guidearticles to Community Prereferences We searched Cochrane Methodolventive [9,10]. Services--methods. Thethe Task Force on Community Services. Am J Prev Med 2000, 18(1 ogy Preventive Register using priority or priorities. WeSuppl):35-43. reviewed the 29. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of th International Conference on Priorities website of the 5 Research Evidence in Guideline Development: Introduction. Health Res Policy Systand 2006,references 4:12. in Health Care [11] that we had in our files 30. Agency for Healthcare Research and Quality [http:// [12-15]. www.guidelines.gov]

change are not so high that they cannot be overcome. While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]).

31. Guidelines International Network [http://www.g-i-n.net] 32. Gaebel W, Weinmann S, Sartorius N, Rutz W, McIntyre JS: SchizoFindings phrenia practice guidelines: international survey and comWhat criteriaBrshould be2005, used187:248-255. to establish priorities? parison. J Psychiatry 33. AD, Fretheim A, Schünemann(IOM) HJ: Improving the Useon of The Oxman US Institute of Medicine's Committee Research Guidelinefor Development: Synthesis Methods for Evidence Setting in Priorities Guidelines8.Developand presentation of evidence. Health Res Policy Syst 2006, 4:20. mentRhinehart in its study of setting priorities for clinical practice 34. E, Goldmann DA, O'Rourke EJ: Adaptation of the Centers for published Disease Control guidelines of guidelines in 1995 argued for thatthe theprevention priority setnosocomial infection in a pediatric intensive care unit in ting Jakarta, processIndonesia. should beAm open [14] They recJ Medand 1991,defensible 91(3B):213S-220S. ommended sixHJ,general criteria: prevalence, 35. Schünemann Fretheim A, Oxman AD: Improvingburden the Use of of Research in Guideline Development: 10. Integratillness, cost ofEvidence managing the problem, variability in pracing values and consumer involvement. Health Res Policy Syst tice, 2006, potential 4:22. of a guideline to improve health outcomes, 36. RL, Morris PS, McDonald Bailie RS: Making systematic and Gruen potential of a guideline toEL, reduce costs. Because data more useful for policy-makers. Bull World Health Organ usedreviews to make these judgements is often lacking, they sug2005, 83(6):480. gested opportunities forE:important unmeasured 37. Lavisexplicit JN, Posada FB, Haines A, Osei Use of research to inform public Lancet 2004, 364(9445):1615-1621. factors to policymaking. be considered. They further suggested separate 38. Hamid M Bustamante-Manaog T, Dung T, Akkhavong K, Fu H, Ma Y: "tracks" for considering issues for specific EVIPNet: translating the spirit of Mexico.populations, Lancet 2006, conditions or concerns. They argued that the rationales for 366::1758-1760. 39. Moynihan special R Oxman tracks AD, Lavis JN, Paulsen E: Evidence-Informed designating should be defined clearly; i.e. Health Policy: Using Research to Make Health Systems whyHealthier. they warrant special consideration. They suggested A review of organizations that support the use of in developing guidelines, technology that research updatingevidence of guidelines should also be considered as a assessments, and health policy, for the WHO Advisory Comseparate "track", taking account of issues such as the need mittee on Health Research. Oslo , Norwegian Knowledge Cenfor corrections andServices; the availability of new evidence. tre for the Health 2006. 40. Shiffman RN Dixon J, Brandt C, Essaihi A, Hsiao A, Michel G, O'Connell R: The GuideLine Implementability Appraisal (GLIA): Oortwijn identified criteria toused to obstacles prioritisetohealth development of an 25 instrument identify guideline implementation. Inform Decis Mak 2005 , 5:23. technology assessmentsBMC andMed categorised these into four 41. Graham ID, Harrison MB, Lorimer K, Piercianowski T, Friedberg E, broad categories: burden of disease, potential effects, Buchanan M, Harris C: Adapting national and international leg potential costs, and uncertainty application of ulcer practice guidelines for localregarding use: the Ontario Leg Ulcer Community [15]. Care Protocol. Adv Skin Wound Care 2005, the technology 18(6):307-318. 42. Graham ID, Harrison MB, Brouwers M: Evaluating and Adapting In aPractice more recent selective review New Zealand Guidelines for Local Use:for A the Conceptual Framework. In In Clinical in criteria Practice Edited Pickering as S, Guidelines Group, theGovernance following were by: identified Thompson J. London , Harcourt Publishers; 2003:213-229. indicating a topic is suitable for 43. Braddickthat M, Stuart M, Hrachovec J: The useguideline of balancedevelopsheets in clinical guidelines. J Am Board Fam Pract 1999, mentdeveloping [16]: 12(1):48-54. 44. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt 1. The is clinically affecting largeR,numbers of GH,topic Harbour RT, Haugh important MC, Henry D, Hill S, Jaeschke Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell people with substantial morbidity or mortality (the burden D, of Oxman AD, Phillips B, Schunemann HJ, Edejer TT, Varonen H, Vist illness). GE, Williams JW Jr., Zaza S: Grading quality of evidence and strength of recommendations. Bmj 2004, 328(7454):1490. 45. Fretheim Schünemann HJ, Oxman AD: Improving thethe Use of 2. The topic isA,complex enough to initiate debate about recResearch Evidence in Guideline Development: 3. Group ommendations. composition. Health Res Policy Syst 2006, 4:15.

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation. BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

Page Page 104ofof10 7 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 14. Reporting guidelines Andrew D Oxman*1, Holger J Schünemann2 and Atle Fretheim3 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway, 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy and 3Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway Email: Andrew D Oxman* - oxman@online.no; Holger J Schünemann - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no * Corresponding author

Published: 08 December 2006 Health Research Policy and Systems 2006, 4:26

Received: 07 April 2006 Accepted: 08 December 2006

doi:10.1186/1478-4505-4-26

This article is available from: http://www.health-policy-systems.com/content/4/1/26 © 2006 Oxman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Page 1 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:26

http://www.health-policy-systems.com/content/4/1/26 http://www.health-policy-systems.com/content/4/1/14

and the Guidelines International Network [8]) for existing Background The World reviews Health Organization like many other systematic and relevant (WHO), methodological research organisations around the world, has recognised the need that address these questions. The answers to the questions to use rigorousbased processes toavailable ensure that health concare are ourmore conclusions on the evidence, recommendations informed by the best available sideration of whatare WHO and other organisations are researchand evidence. is the 14th of a series of 16 reviews doing, logical This arguments. that have been prepared as background for advice from the this WHO Advisory Committee on using Health(clinical Research to For review we searched PubMed pracWHO on how to this. guidelines) and (priority tice guidelines or achieve public health setting or setting priorities) and related articles for selected Guidelines[9,10]. are formal thatMethodolshould be references We advisory searched statements the Cochrane robust enough to meet the unique circumstances and conogy Register using priority or priorities. We reviewed the on are Priorities straints of situation Conference to which they being website of the specific 5th International in Health Care and references that we had in ourhave files applied [1]. The[11] basic nature and intent of guidelines [12-15]. been variously labelled as guidance, guides, guiding principles, recommendations, policies, protocols, best practice, algorithms, consensus statements, expert committee Findings What criteria shouldintegrated be used to care establish priorities? recommendations, pathways, manuals, The US Institute of Medicine's Committee on tool kits, handbooks, model lists,(IOM) technical updates and Methods Setting Priorities forare Guidelines Developprinciples for [1,2]. Whatever they called, rigorously ment in itsguidelines, study of setting prioritiescomplicated for clinical research practice developed can translate guidelines published in 1995 argued that theThey priority findings into actionable recommendations. aresetan ting processstep should be open from and defensible [14] They and recimportant in moving research to action ommended criteria: prevalence, burden of ensuring thatsix thegeneral best available research evidence informs illness, cost ofactions managing problem, in pracdecisions and [3]. the However, for variability users of guidelines tice, of a guideline health outcomes, to bepotential able to apply criteria to improve assess whether guidelines and of a guideline to reduce costs. Because data havepotential been rigorously developed and are likely to be valid used to make these judgements is often lacking, sugand applicable [3-6], the information needed they to make gested explicit opportunities for important unmeasured these judgements must be reported [7]. Unfortunately, factors to be considered. They further separate critical information is often absent fromsuggested published guide"tracks" for considering issues for specific populations, lines [5,7,8]. conditions or concerns. They argued that the rationales for designating tracksthe should be defined clearly; i.e. In this paperspecial we address following questions: why they warrant special consideration. They suggested that updating of guidelines also be considered as a • What standard types of should recommendations or reports separate "track", taking account of issues such as the need should WHO use? for corrections and the availability of new evidence. • How should recommendations be formulated and Oortwijn reported? identified 25 criteria used to prioritise health technology assessments and categorised these into four broad categories: burden disease, systematic potential reviews effects, We address questions aboutofreporting potential costs, and uncertainty regarding [9] and dissemination [10] in other papers application in this series.of the technology [15].

3. Thereother is evidence of variation between actual and appropriWhat organisations are doing In acare. recent survey of organisations that produce clinical ate practice guidelines, all 31 organisations that responded (response 86%), and of 57 (81%) 4. There arerate no existing valid 46 guidelines availableoftounits use. that support the use of research evidence by governments in developing health policy, reported producing fullavailable. versions 5. There is an adequate amount of existing evidence of guidelines with references and notes [12]. Several organisations use different for different types of 6. The recommendations willformats be acceptable to the potential recommendations, and a majority produce different verusers. sions of guidelines, such as executive summaries, summariesImplementation of take-home of messages, separate versions 7. the guideline is feasible, willfor notdifferent exhaustarget use users, tools for application (e.g., algorithms or tively the and communities' resources, and barriers to clinical flow charts). guideline standard forchange are notMany so high that theyproducers cannot be have overcome. mats that they use and some organisations, such as the U.S. National Clearing House, have While burdenGuidelines of disease is commonly used as developed a criterion standard reporting guidelines by for priorityformats setting,for it should be noted that theproduced use of sumother organisations [13]. measures, such as disability mary burden of disease adjusted life years (DALYs) has been criticised for focusing The disease UK National for Health Excelon rather Center than resource use and and Clinical interventions, lence (NICE) that recommendations should be because of thestates assumptions about values inherent in clear such and concise, should contain sufficient information measures, andbut because of the technical limitations of such that they (see can for be understood without[17] reference to other measures example references and [18]). supporting material [14]. This is particularly important What should be are used to agree on priorities?from whereprocesses recommendations published in isolation Batista and Hodge in in a review 10 years ago the background details the full conducted guideline. Any terminolfound only three articles pertinent to priority ogy included in the recommendations thereforesetting needsfor to clinical practice [10]. They suggested the folbe clearly definedguidelines and unambiguous. lowing framework for priority setting:

Methods

TheConsult methods used tousers prepare review are described in 1. with end andthis other stakeholders before the introduction selecting topics. to this series [15]. Briefly, the key questions addressed in this paper were vetted amongst the authors andfeasibility the ACHR Subcommittee on the Use of 2. Consider during the consultation. Research Evidence (SURE). We did not conduct a full systematic review. We searched PubMed three databases 3. Document the process used to selectand guideline topics. of methodological studies (the Cochrane Methodology Register the US Guideline Clearinghouse The IOM[16], suggested theNational following procedures [14]: [17], and the Guidelines International Network [18]) for existing reviews and relevant methodological • the usesystematic of Delphi-like procedures for obtaining expert research that these questions. The answers to the judgments or address topic rankings through correspondence questions are our conclusions based on the available evidence, consideration what andexplicit other organisa• the use of questionsofthat areWHO specific, and contions are doing, and logical arguments. sistent with standard methods for questionnaire construction For this review we searched PubMed using [the MeSH terms 'Documentation/standards' 'PracticetoGuide• experimentation with more formaland procedures arrive lines/standards'] and related articles; the Cochrane Methat group judgments odology Register using [the key words 'CMR: Review methodology – applicability andnarthe They also suggested there is&a recommendations' need to define more text words or topics reporting)] and ['Levels of evidence rowly and (format precisely for guideline development. and strength of recommendations']; the National They argued that this would result in more efficient Guideorganlines Clearinghouse annotated bibliography the ization of panels and their work, resolutionusing of some terms format, reporting and structure; and guidelines, checked theand refapparent controversies, more responsive erenceimplementation. lists of retrieved articles. easier

What WHO is doing now

Although Guidelines WHO recomIn a morethe recent selectivefor review forGuidelines the New Zealand mends "thatGroup, a uniform, readily-recognizable Guidelines the following criteria were printing identifiedforas mat be developed for WHO guidelines," there are, as yet, indicating that a topic is suitable for guideline developno standard ment [16]: formats for WHO policies, recommendations or guidelines. A survey of WHO guidelines published in 2005 WHOimportant publishesaffecting a large large number of rec1. Thefound topic isthat clinically numbers of ommendations of many different in many different people with substantial morbidity or types, mortality (the burden of formats [2], and a review of WHO documents did not find illness). any standards for reporting WHO recommendations [11]. 2. The topic is complex enough to initiate debate about the recommendations.

Page 4 2 of 7 6 (page number not for citation purposes)

the use of 4:26 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/26

Findings

number of instruments for evaluating clinical practice guidelines that can also be used as checklists for reporting [1,4-6,33]. The content used in the National Guideline Clearing house also represents a standard for reporting imposed on organisations that want their guidelines included in that database [13]. It includes 52 items under the following headings: scope, methodology â&#x20AC;&#x201C; including rating scheme and cost analysis, recommendations, evidence supporting the recommendations, benefits/harms of implementing the recommendations, contraindications, qualifying statements, implementation of the guideline, Institute of Medicine (IOM) national healthcare quality report categories, identifying information and availability, and disclaimer; in addition to indexing attributes.

What standard types of recommendations or reports should WHO use? Given the wide variety of different types of recommendations that are made by WHO, there is likely to be a need for several standard types of recommendations. Systematically developed clinical practice guidelines can take 18 months or more and as many as 15 meetings [19]. Systematically developed public health guidelines also require substantial resources and time [20]. Given the time and resources required to produce guidelines, many organisations, particularly HTA organisations, have developed rapid assessment processes [21-25]. There is variation in the scope, methods, time to complete assessments, and the formats used to report rapid assessments. Another type of recommendation or guideline that is receiving increasing attention, also because of the resources and time required to develop guidelines systematically, are guidelines developed by other organisations that have been adapted or endorsed [26-28]. Another approach being taken in several countries is to create databases or clearinghouses of clinical practice guidelines with the aim of facilitating their evaluation and adaptation for local use by health care organizations [29].

Similarly, some journals have standard formats for reporting clinical practice guidelines, including structured abstracts with the following headings [34]: Objective a succinct statement of the objective of the guideline, including the targeted health problem, the targeted patients and providers, and the main reason for developing recommendations concerning this problem for this population.

Systematically developed clinical recommendations, public health recommendations, and health systems recommendations all require similar processes to ensure their quality. Decision makers also require similar types of information to be able to critically appraise whether guidelines have been rigorously developed and are likely to be valid [3,4,7,20,30,31].

Options principal practice options that were considered in formulating the guideline. Outcomes significant health and economic outcomes identified as potential consequences of the practice options.

How should recommendations be formulated and reported? The Conference on Guideline Standardization (COGS) used a two-stage modified Delphi process to develop standards for reporting clinical practice guidelines [7]. Representatives of 22 organisations active in guideline development reviewed the proposed items and commented favourably. The items were consolidated into 18 topics (Table 1) to create the COGS checklist, which provides a framework to support comprehensive documentation of guidelines. While it is possible that some guideline developers may not include content for every item, it is suggested that they should address explicitly whether the guideline development team considered that item.

Evidence methods used to gather, select, and synthesize evidence, and the date of the most recent evidence obtained. Values persons and methods used to assign values (relative importance) to potential outcomes of alternative practice options. Benefits, harms, and costs the type and magnitude of the main benefits, harms, and costs that are expected to result from guideline implementation.

While many organisations have their own standard formats for reporting guidelines, this is the only consensus standard for reporting guidelines across organisations. We have, however, summarised the key items included in guidelines for guidelines, which provides the basis for a similar, but more comprehensive checklist for conducting or reporting guidelines [32]. In addition, there are a

Recommendations a brief and specific list of key recommendations.

Page 3 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:26

http://www.health-policy-systems.com/content/4/1/26 http://www.health-policy-systems.com/content/4/1/14

Table 1: The COGS checklist for reporting clinical guidelines3. (from Shiffman et al.of [7])* and the Guidelines International Network [8])practice for existing There is evidence variation between actual and approprisystematic reviews and relevant methodological research ate care. Topic Description that address these questions. The answers to the questions are our conclusions available evidence, con-that includes 4. There no existing available use. 1. Overview material based on theProvide a structured abstract the are guideline's releasevalid date, guidelines status (original, revised,toupdated), sideration of what WHO and and other are print organisations and electronic sources. 2. Focusand logical arguments. Describe the primary disease/condition and intervention/service/technology the guideline addresses. doing, 5. There is an adequate amount ofthat existing evidence available.

Indicate any alternative preventive, diagnostic or therapeutic interventions that were considered during development.

For this review we searched PubMed using (clinical prac6. The recommendations will be acceptable to the potential 3. Goal Describe the goal that following the guideline is expected to achieve, including the rationale for tice guidelines or public health development guidelines)ofand (priority users. a guideline on this topic. setting or setting priorities) and related articles for selected 4. Users/setting Describe the intended users of the guideline (e.g., provider types, patients) and the settings in which the guideline is intended to be used. references [9,10]. We searched the Cochrane Methodol7. Implementation of the guideline is feasible, will not exhaus5. Target population Describe the population for guideline recommendations and list any exclusion criteria. ogy Register using priority or priorities. Wepatient reviewed the eligible tively use the communities' resources, and barriers to clinical 6. Developer the organization(s) responsible for guideline development and the names/credentials/potential Conference on Priorities website of the 5th International Identify change are not so high that they cannot be overcome. conflicts of interest of individuals involved in the guideline's development. in7. Health Care [11] and references that the wefunding had insource/sponsor our files Funding sources/sponsor Identify and describe its role in developing and/or reporting the guideline. [12-15]. Disclose potential conflict of interest.While burden of disease is commonly used as a criterion 8. Evidence collection Describe the methods used to searchfor thepriority scientificsetting, literature,itincluding of that dates the and use databases shouldthe berange noted of sumsearched, and criteria applied to filtermary the retrieved evidence. burden of disease measures, such as disability Findings 9. Recommendation grading criteria Describe the criteria used to rate the quality of evidence that supports the recommendations and the What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing system for describing the strength of the recommendations. Recommendation strength communicates the The US Institute of Medicine'simportance (IOM) of Committee on disease resource and interventions, adherence to on a recommendation andrather is based than on both the qualityuse of the evidence and the magnitude of anticipated benefits or because harms. Methods for Setting Priorities for Guidelines Developof the assumptions about values inherent in such 10. Method synthesizing evidence Describe evidence was used to measures, create recommendations, e.g., tables, meta-analysis, ment in itsforstudy of setting priorities forhow clinical practice and because ofevidence the technical limitationsdecision of such analysis. guidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). 11. Prerelease review Describe how the guideline developer reviewed and/or tested the guidelines prior to release. ting process should be open and State defensible [14] They rec12. Update plan whether or not there is a plan to update the guideline and, if applicable, an expiration date for this What processes should be used to agree on priorities? ommended six general criteria:version prevalence, burden of of the guideline. Batista and Hodge inofathe review conducted 10subject yearstoago illness, cost of managing the problem, variability inand prac13. Definitions Define unfamiliar terms those critical to correct application guideline that might be misinterpretation. found only three articles pertinent to priority setting for tice, potential of a guideline to improve health outcomes, 14. Recommendations and rationaleto reduce State the recommended and the specific circumstances under which perform it. the Justifyfolclinical practice guidelines [10]. They tosuggested and potential of a guideline costs. Becauseaction data precisely each recommendation by describing the linkage between the recommendation and its supporting evidence. lowing framework for priority used to make these judgements is oftenthe lacking, they sug- and the Indicate quality of evidence recommendation strength, based onsetting: the criteria described in 9. gested explicit opportunities for important unmeasured 15. Potential benefits and harms Describe anticipated benefits and potential risks associated with implementation of guideline recommendations. 1. Consult with end users and other stakeholders before factors to be considered. They further suggested separate 16. Patientfor preferences Describe the role of patient preferences when a topics. recommendation involves a substantial element of personal selecting "tracks" considering issues for specific populations, choice or values. conditions or concerns. They argued that the rationales for 17. Algorithm Provide (when appropriate) a graphical description of the stages and decisions in clinical care described by 2. Consider feasibility during the consultation. designating special tracks should defined clearly; i.e. thebe guideline. why they warrant special consideration. They suggested 18. Implementation considerations Describe anticipated barriers to application of the recommendations. Provide reference to any auxiliary documents providers oraspatients that are intendedthe to facilitate Suggest review topics. criteria 3. Document processimplementation. used to select guideline that updating of guidelines should also befor considered a for measuring changes in care when the guideline is implemented. separate "track", taking account of issues such as the need The IOM suggested the following procedures [14]: for corrections and the availability of new evidence. *COGS = Conference on Guideline Standardization.

â&#x20AC;˘ the use of Delphi-like procedures for obtaining expert Shekelle and colleagues evaluated the effect of different judgments or topic rankings through correspondence levels of specificity of recommendations on clinicians test ordering clinical vignettes â&#x20AC;˘ the use behaviour of questionsusing that are specific, explicit[36]. and They confound that standard cliniciansmethods receiving non-specificconstrucrecomsistent with forthe questionnaire mendations ordered fewer indicated tests for appropriate tion clinical vignettes than did physicians receiving specific recommendations. Furthermore, compared to physicians â&#x20AC;˘ experimentation with more formal procedures to arrive receiving non-specific guidelines, physicians receiving at group judgments specific guidelines ordered significantly more appropriate tests vignettes andtosignificantly They for alsocorresponding suggested there is a need define morefewer nartests inappropriate vignettes. The authors concluded rowlyfor and precisely topics for guideline development. that clarity applicability of a guideline They the argued thatand this clinical would result in more efficient organmight thatresolution contributeoftosome the ization be of important panels andattributes their work, effects of controversies, practice guidelines. did notguidelines, find any other apparent more We responsive and comparisons of different ways of formulating recommeneasier implementation. dations, and it is likely that the way in which recommen-

Oortwijn identified 25 criteria used to prioritise health Validation technology assessments and categorised these into four the results of any external comparison witheffects, guidebroad categories: burdenreview, of disease, potential lines developed by uncertainty other groups, or clinical testing of potential costs, and regarding application guideline use. [15]. the technology Sponsors In a more recent selective review for the New Zealand key persons or groups that criteria developed, funded, or Guidelines Group, the following were identified as endorsed the guideline. indicating that a topic is suitable for guideline development [16]: While many organisations have standards for how recommendations formulated, we are not aware any con1. The topic isare clinically important affecting largeofnumbers of sensuswith standards for how recommendations people substantial morbidity or mortality (theshould burden be of formulated. Most guidelines development groups now illness). grade the quality of evidence and the strength of recommendations, a variety of to different systems are 2. The topic is but complex enough initiate grading debate about the recused [12,35]. ommendations.

Page 4 of 7 6 (page number not for citation purposes)

the use of 4:26 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/26

dations are formulated may need to be adapted to the specific characteristics of a guideline. However, there is a consensus that recommended actions should be stated precisely [7].

ity, and a core function of WHO is to provide its member states, particularly LMIC, with technical advice that is informed by the best available research evidence. Through collaborating with other organisations and establishing standards for reporting, and possibly endorsing guidelines developed by other organisations, WHO may be able to expand its potential for supporting access to guidelines that are appropriate for LMIC or can easily be adapted to conditions in LMIC.

Both checklists for evaluating guidelines and for reporting guidelines include items that may be dependent on the specific setting in which a guideline is developed or used. There are a number of international organisations, including WHO, that develop guidelines that are intended to be used in a variety of settings around the world. We did not find any published papers that addressed methods for taking into account setting specific factors in international guidelines, although several groups are working on methods for adapting guidelines developed in one setting for use in another [27,28,37-39].

Given WHO's role as the world's leading public health agency and its mandate to provide evidence-informed guidance, it is of major importance for WHO to address and develop methods for formulating and reporting recommendations that are developed internationally, but need to be adapted and implemented in specific settings.

Discussion

Further work

While the content of WHO guidelines, recommendations and policies will vary, depending on the topic, it would be desirable to have standard formats across different topics to facilitate recognition and use by decision makers and ensure that all the information needed to judge the quality of a guideline, determine its applicability and, if needed, adapt it is reported. There is likely a need for at least three standard formats: full systematically developed guidelines that are sponsored by WHO, rapid assessments, and guidelines that are endorsed by WHO. Standards such as those advocated by COGS should form the basis for developing a uniform format for full guidelines developed by WHO. Although the COGS standards were developed for clinical practice guidelines, the same considerations are relevant to public health and health systems recommendations. Further consideration is needed regarding the inclusion of additional items that need to be considered in WHO guidelines, such as applicability to different settings, equity, and scaling up. In particular, work is needed to develop a template for decision-making frameworks when different conditions are likely to lead to different decisions in different settings [40,41]. In addition, different versions of guidelines should be developed for different target audiences, including a structured executive summary and key messages [31,34].

As part of the methodological work that is needed to addresses the challenges of developing international guidelines, specific attention should be given to the development of standard templates that provide decision makers with the relevant global evidence that is needed to inform a decision and offers practical methods for incorporating the context specific evidence and judgements that are needed [40,41]. In addition to ensuring that standard formats are used for WHO guidelines to ensure complete reporting, attention should be paid to ensuring that the format that is used, and derivative versions, are understandable and useful to the intended target audiences.

Competing interests

A different format for rapid assessments would help to distinguish these from full guidelines and could be designed to reduce the work and time necessary to complete a report. Rapid assessments should, nonetheless, include the same information as full guidelines, indicating explicitly what the group preparing the guideline did not do, as well as the methods that were used.

Authors' contributions

ADO prepared the first draft of this review. HJS and AF contributed to drafting and revising it.

WHO has limited resources and capacity for developing guidelines. At the same time, low and middle-income (LMIC) countries also have limited resources and capac-

100

Page 5 of 6 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:26

and the Guidelines International Network [8]) for existing References 1. Global Programme on Evidence for methodological Health Policy: Guidelines for systematic reviews and relevant research Guidelines. Geneva: World Health Organisation, 2003 (EIP/ that WHO address these questions. The answers to the questions GPE/EQC/2003.1). are conclusions based on the available evidence, con2. our Panisett U: A review of WHO recommendations published in 2005. . of what WHO and other organisations are sideration 3. Guyatt G, Yayward R, Richardson WS, Green L, Wilson M, Sinclair J, doing, and logical arguments. et al.: Moving from evidence to action. In Users' Guides to the Medical Literature Edited by: Guyatt G, Rennie D. Chicago: AMA Press;

2002:175-99. For this review we searched PubMed using (clinical prac4. Institute of Medicine: Guidelines for Clinical Practice: From tice guidelines ortopublic guidelines) andAcademy (priority Development Use. health Washington, DC: National Pr; 1992. setting or setting priorities) and related articles for selected 5. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines folreferences We The searched the Cochrane lowing [9,10]. guidelines? methodological qualityMethodolof clinical ogy practice Registerguidelines using priority priorities. reviewed the in the or peer reviewed We medical literature. JAMA of 1999, website the281:1900-5. 5th International Conference on Priorities 6. The AGREE Collaboration: Development and validation of an in Health Care [11] and references we had inthe our files international appraisal instrumentthat for assessing quality of clinical practice guidelines: the AGREE project. Qual Saf [12-15].

Health Care 2003, 12:18-23. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, DeshFindings pande AM: Standardized Reporting of Clinical Practice Guidelines: A Proposal Conference on Guideline What criteria should befrom used the to establish priorities? Standardization. Ann Intern Med 2003, 139:493-8. The Grilli US R, Institute Medicine's Committee on 8. Magrini N,ofPenna A, Mura G,(IOM) Liberati A: Practice guidelines developed by specialty societies: the need forDevelopa critical Methods for Setting Priorities for Guidelines Lancetof 2000, 355:103-6. mentappraisal. in its study setting priorities for clinical practice 9. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of guidelines published 1995 argued that the priority setResearch Evidence in in Guideline Development: 8. Synthesis presentation of evidence. Res Policy[14] Syst 2006. ting and process should be open andHealth defensible They rec10. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of ommended general criteria: Development: prevalence, 15. burden of Research six Evidence in Guideline Dissemiillness, cost of managing the problem, variability in pracnating and implementing guidelines. Health Res Policy Syst 2006. 11. AD, Fretheim A, Schünemann HJ: A review WHO doctice, Oxman potential of a guideline to improve healthofoutcomes, uments that guide the development, dissemination and and implementation potential of a guideline to reduce costs. Because data of recommendations by WHO. . 12. Oxmanjudgements AD, Lavis JN, is Paulsen Evidence-Informed usedMoynihan to makeR,these oftenE: lacking, they sugHealth Policy: Using Research to Make Health Systems gested explicit opportunities for important unmeasured Healthier. A review of organizations that support the use of factors to be considered. further suggestedtechnology separate research evidence in They developing guidelines, assessments, and health issues policy, for WHO Advisory Com"tracks" for considering forthe specific populations, mittee on Health Research. Oslo: Norwegian Knowledge Centre conditions or concerns. They argued that the rationales for for the Health Services; 2006. designating tracks should be defined clearly; i.e. 13. Guidelinespecial Summary Sheet. National Guideline Clearing[http://www.guideline.gov/about/CompleteSummaryDe why house they warrant special consideration. They suggested scrip.aspx] that updating of guidelines should be considered as a 14. What are the key components of aalso template for recommendations could be the question National Instutites for Health separate "track", taking accounthere? of issues such as the need and Clinical Excellence Handbook [http://www.nice.org.uk/pdf/ for corrections and the availability of new evidence. GDM_Allchapters_0305.pdf]. 15. Oxman AD, Schünemann HJ, Fretheim A: Improving the use of research evidence 25 in guideline development: introduction. Oortwijn identified criteria used to prioritise health Health Res Policy Syst 2006. technology assessments and categorised these into four 16. Cochrane Methodology Register [http://www.cochrane.org/ access_data/cmr/accessDB_cmr.asp] broad categories: burden of disease, potential effects, 17. National Guideline Clearinghouse [http://www.guidelines.gov] potential costs, and uncertainty regarding application of 18. Guidelines International Network (GIN) [http://www.g-ithe technology [15]. n.net] 19. Raine R, Sanderson C, Black N: Developing clinical practice guidelines: a challenge to current methods. BMJ 2005, In a331:631-3. more recent selective review for the New Zealand Guidelines the following were identified as 20. Briss PA,Group, Zaza S, Pappaioanou M, et criteria al.: Developing an evidencebased guide preventive Am indicating that to a community topic is suitable for services-methods. guideline developJ Prev Med 2000, 18(1S):35-43. mentStevens [16]:A, Colin-Jones D, Gabbay J: 'Quick and clean': authorita21. tive health technology assessment for local health care contracting. Health Trends 1995, 27:37-42. 1. The topic is clinically important affecting large numbers of 22. Hailey D, COrabian P, Harstall C, Schneider W: The use and people with of substantial morbidity or mortality (the Int burden of impact rapid health technology assessments. J Technol Assess Health Care 2000, 16:651-6. illness). 23. Perleth M, Luhmann D, Givis B, Droste S: Rapid Assessments – quick evaluation of medical technology. Gesundheitswesen 2001, 63(Suppl 2. The topic is1):S79-84. complex enough to initiate debate about the rec24. Canadian Coordinating Office for Health Technoloty Assessment: ommendations. Rapid response services in health technology assessment: 7.

http://www.health-policy-systems.com/content/4/1/26 http://www.health-policy-systems.com/content/4/1/14

information from and additional infor3. There is evidence ofINAHTA variationmembers between actual and approprimation. Ottawa: CCOHTA; 2004. ate care. 25. Milne R, Clegg A, Stevens A: HTA responses and the classic HTA

report. J Public Health Med 2003, 25:102-6. 26. Schünemann Fretheim A, Oxman AD: Adaptation, 4. There are no H, existing valid guidelines available to applicabiluse. ity, transferability. . 27. Graham ID, Harrison MB, Brouwers M, Davies BL, Dunn S: Facilitating the useadequate of evidence in practice: evaluating andavailable. adapting 5. There is an amount of existing evidence clinical practice guidelines for local use by health care organizations. J Obstet Gynecol Neonatal Nurs 2002, 31:599-611. 6. The recommendations be K, acceptable to the potential 28. Graham ID, Harrison MB, will Lorimer Piercianowski T, Friedberg E, users.Buchanan M, Harris C: Adapting national and international leg ulcer practice guidelines for local use: the Ontario Leg Ulcer Community Care Protocol. Adv Skin Wound Care 2005, 7. Implementation of the guideline is feasible, will not exhaus18:307-18. 29. Resources. National Clearinghouse tivelyGuideline use the communities' resources,Guideline and barriers to clinical [http://www.guideline.gov/resources/guideline_resources.aspx] change are not so high that they cannot be overcome. 30. Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EAF, Millson ME, et al.: An approach to the development of practice guidelines for community health interventions. Can While burden of disease is commonly used as a criterion J Public Health 1994, 85(Suppl 1):S8-13. for it should be noted that the use ofK,sum31. priority Lavis JN, setting, Davies HTO, Oxman AD, Denis JL, Golden-Biddle Ferlie reviews that inform manmaryE: Towards burden systematic of disease measures, suchhealthcare as disability agement and policymaking. Journal of Health Services Research and adjusted life years (DALYs) has been criticised for focusing Policy 2005, 10(Suppl 1):35-48. 32. disease Schünemann HJ, Fretheim A, Oxman use AD: Improving the Use of on rather than resource and interventions, Research Evidence in Guideline 1. Guidelines because of the assumptions aboutDevelopment: values inherent in such for guidelines. Health Res Policy Syst 2006. measures, because of the PC, technical limitations 33. Grahamand ID, Calder LA, Hebert Carter AO, Tetroe JM:of A such comparison of clinical practice guideline appraisal measures (see for example references [17] andinstruments. [18]). Int J Technol Assess Health Care 2000, 16:1024-38. 34. Hayward RS, Wilson MC, Tunis SR, Bass EB, Rubin HR, Haynes RB: What processes shouldabstracts be used of to articles agree on priorities? More informative describing clinical practice Intern Med 1993, 118:731-7. Batista and guidelines. Hodge in Ann a review conducted 10 years ago 35. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of found only three articles pertinentDevelopment: to priority setting for Research Evidence in Guideline 9. Grading evidence and recommendations. Health suggested Res Policy Systthe 2006.folclinical practice guidelines [10]. They 36. Shekelle PG, Kravitz Beart J, setting: Marger M, Wang M, Lee M: Are lowing framework forRL, priority nonspecific practice guidelines potentially harmful? A randomized comparison of the effect of nonspecific versus specific guidelines on physician decision Health before Serv Res 1. Consult with end users and other making. stakeholders 2000, 34:1429-48. selecting topics. 37. INCLEN Knowledge 'Plus' Program (K+). The INCLEN Trust [http://www.inclentrust.org/index.php?option=con tent&task=view&id=182&Itemid=223#KPP] 2. Consider feasibility during the consultation. 38. Adapting existing HTAs from one country to other. European Network for Health Technology Assessment – EUnetHTA [http://www.sst.dk/upload/planlaegning_og_behandling/ 3. Document the process used to select guideline topics. cemtv/eunethta/wp5description.pdf] 39. Fervers B, Graham I: Guideline Guideline adaptation: the way The IOM suggested the following procedures [14]: forward? The Guidelines International Network [http://www.g-i-n.net/ download/files/Plenary_2__Fervers_Graham.pdf]. 40. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of • theResearch use of Delphi-like procedures for obtaining expert Evidence in Guideline Development: 7. Deciding what evidence to rankings include. Health Res Policy Syst 2006. judgments or topic through correspondence 41. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 13. Adapta• thetion, useapplicability of questions are specific, explicit and and that transferability. Health Res Policy Syst con2006.

sistent with standard methods for questionnaire construction Publish with Bio Med Central and every scientist can read your work free of charge • experimentation with more formal procedures to arrive "BioMed Central will be the most significant development for at group judgments disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation. BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

101

Page 4 6 of 7 6 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 15. Disseminating and implementing guidelines Atle Fretheim*1, Holger J Schünemann2 and Andrew D Oxman1 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway and 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy Email: Atle Fretheim* - atle.fretheim@nokc.no; Holger J Schünemann - hjs@buffalo.edu; Andrew D Oxman - oxman@online.no * Corresponding author

Published: 08 December 2006 Health Research Policy and Systems 2006, 4:27

doi:10.1186/1478-4505-4-27

Received: 07 April 2006 Accepted: 08 December 2006

This article is available from: http://www.health-policy-systems.com/content/4/1/27 © 2006 Fretheim et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

102

Page 1 of 4 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:27

What criteria should be used to establish priorities? The USshould Institute of do Medicine's (IOM)and Committee on • What WHO to disseminate facilitate the Methods Setting Priorities for Guidelines Developuptake of for recommendations? ment in its study of setting priorities for clinical practice guidelines published in 1995 that the by priority set• What should be done at argued headquarters, regional ting process should be open and defensible [14] They recoffices and in countries? ommended six general criteria: prevalence, burden of illness, costrelated of managing the problem, variabilityofinguidepracQuestions to adaptation and evaluation tice, of a guideline to improve health linespotential are addressed in other papers in this seriesoutcomes, [1,2]. and potential of a guideline to reduce costs. Because data used to is make thesedoing judgements What WHO now? is often lacking, they sugThere isexplicit no general WHO strategy for guideline implegested opportunities for important unmeasured mentation. methods They that are usedsuggested vary fromseparate departfactors to beThe considered. further ment to for department, and mayfor vary from populations, case to case "tracks" considering issues specific within departments. Field strategies conditions or concerns. Theytesting arguedand that rollout the rationales for that are usedspecial to promote uptake recommendations designating tracksthe should beofdefined clearly; i.e. are often informed by the findings of implementation why theynot warrant special consideration. They suggested research, and itofisguidelines generally difficult to estimate the impact that updating should also be considered as a of the implementation are such used,as since evalseparate "track", taking strategies account ofthat issues the need uations are rarely if they are doneevidence. at all. for corrections andrigorous, the availability of new

There are,identified however, 25 examples of rigorous evaluations, Oortwijn criteria used to prioritise health such as implementation of the Integrated Management technology assessments and categorised these into four for Childhood Illnesses (IMCI) guideline, which has been broad categories: burden of disease, potential effects, evaluated costs, in a randomised controlled trial application [3]. WHO has potential and uncertainty regarding of alsotechnology in some cases the [15].reviewed the relevant evidence-base, for example for strategies to improve the use of drugs in developing countries [4]. review for the New Zealand In a more recent selective Guidelines Group, the following criteria were identified as Although that recommendations inevitably needdevelopto be indicating a topic is suitable for guideline adapted and implemented at country level, WHO headment [16]: quarters and regional offices can support these activities [2].The topic is clinically important affecting large numbers of 1. people with substantial morbidity or mortality (the burden of illness). What are other organisations doing? In an international survey of organisations that develop guidelines health technology assessments, almost 2. The topic or is complex enough to initiate debate about the half recof the 95 respondents reported using provider-mediated ommendations.

http://www.health-policy-systems.com/content/4/1/27 http://www.health-policy-systems.com/content/4/1/14

interventions as part their strategy implementing of 3. There is evidence of of variation betweenfor actual and appropriguidelines ate care. [5]. Examples of this were conducting audits and hosting work-shops for practitioners. However, many respondents did not provide specific examples. 4. There are no existing valid guidelines available to use. In There a smaller survey of prominent 5. is aninternational adequate amount of existing evidence guideline available. developers, nearly all of the 18 organisations used educational and conferences as part of imple6. The materials recommendations will be acceptable to their the potential mentation strategies [6]. Other common approaches were users. audit and feed-back, use of local opinion leaders, and organizational interventions (e.g. financialwill incentives or 7. Implementation of the guideline is feasible, not exhausdisincentives). tively use the communities' resources, and barriers to clinical change are not so high that they cannot be overcome.

Methods

The methods to prepare this review areas described in While burdenused of disease is commonly used a criterion the priority introduction this series [7]. that The the keyuse questions for setting,to it should be noted of sumaddressed in this were vetted amongst authors mary burden of paper disease measures, such asthedisability and the ACHR Subcommittee thecriticised Use of Research Eviadjusted life years (DALYs) hason been for focusing dence (SURE). Practice and on disease ratherThe thanCochrane resource Effective use and interventions, Organisation Care (EPOC) group undertakes because of theof assumptions about values inherent systemin such atic reviewsand of because interventions improvelimitations health careofdelivmeasures, of theto technical such ery and (see health systems. EPOC measures for care example references [17]has andundertaken [18]). periodic overviews of systematic reviews to assess and What processes should beavailable used to agree on priorities? summarise the evidence from existing Cochrane Batista Hodge reviews in a review conducted 10 years ago and nonand Cochrane [8-10]. The research findings found only three to priority setting reported here arearticles drawn pertinent from these overviews and for an clinical practice guidelines that [10].is They suggested the folupdate of those overviews underway. In addition, lowing framework forand priority we searched PubMed threesetting: databases of methodological literature (within the databases of The Cochrane 1. Consult users Guideline and other Clearinghouse stakeholders before Library, thewith US end National [11] selecting topics. and the Guidelines International Network [12]) for existing systematic reviews and relevant methodological 2. Consider during the consultation. research thatfeasibility address these questions. The search-term we used was "guidelines and implementation and systematic 3. Document the process used to select guideline topics. review". The suggested thesystematic following procedures [14]: The We IOM did not conduct reviews ourselves. answers to the questions are our conclusions based on the • the use evidence, of Delphi-like procedures for obtaining available consideration of what WHO andexpert other judgments or topic rankings througharguments. correspondence organisations are doing and logical • the use of questions that are specific, explicit and conFindings What should WHO domethods to disseminate and facilitate the sistent with standard for questionnaire construcuptake of recommendations? tion Most research on implementation and dissemination strategies for guidelines have focused on clinical to practice • experimentation with more formal procedures arrive guidelines, with change in clinical practice being the priat group judgments mary outcome of interest. An overview of systematic reviews interventions aimed at tochanging provider They alsoofsuggested there is a need define more narbehaviour that:topics "In general, passive approaches are rowly andfound precisely for guideline development. generally ineffective unlikely tomore resultefficient in behaviour They argued that this and would result in organchange. Most other and interventions are effective under ization of panels their work, resolution of some circumstances; none are more effective under allguidelines, circumstances. apparent controversies, responsive and Promising approaches include educational outreach (for easier implementation. prescribing) and reminders" [9]. A more recent compre-

103

Page 4 2 of 7 4 (page number not for citation purposes)

the use of 4:27 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/27

hensive review of evaluations of the effects of strategies for guideline implementation found that "The majority of interventions observed modest to moderate improvements in care", but there was "considerable variation in the observed effects both within and across interventions"[13].

mendations by providing tools such as NorthStar, support and coordination of efforts.

Further work

Rigorous evaluations of the effectiveness of strategies for implementing and disseminating recommendations are needed. Given that the use and impact of WHO recommendations is likely to be limited without an active implementation strategy, it is of paramount interest to the organisation to invest in generating the knowledge needed for successful implementation.

Few evaluations of interventions to change professional practice have been conducted in low-income countries [14]. Guidance from WHO is often directed towards policymakers. There is limited research to inform the choice of strategies to improve the uptake of WHO recommendations by policymakers. The findings of systematic reviews of studies of decision-making by health care managers and policymakers have found that factors such as interactions between researchers and health care policy-makers and timing/timeliness appear to increase the prospects for research use among policymakers [15,16].

Competing interests

What should be done at headquarters, by regional offices and in countries? We did not identify any research findings that could inform the answer to this question.

Authors' contributions

Discussion

AF prepared the first draft of this review. HJS and ADO contributed to drafting and revising it. All authors read and approved the final manuscript.

Passive dissemination of guidelines alone is not likely to adequately ensure appropriate uptake of recommendations in most circumstances. However, the conclusion in an extensive review of guidelines implementation strategies was: "There is an imperfect evidence base to support decisions about which guideline dissemination and implementation strategies are likely to be efficient under different circumstances" [13]. Thus, WHO needs to carefully consider the likely benefits and costs of alternative implementation strategies in relationship to specific contexts, and to evaluate the impact of selected strategies.

References 1.

There are tools available that are designed to assist in the design and evaluation of implementation strategies, such as NorthStar, developed by the EC-funded Research-based continuing education and quality improvement (ReBEQI) project [17]. NorthStar provides a range of information, checklists, examples and tools based on current research on how to best design and evaluate implementation strategies.

2. 3.

4. 5.

Health authorities at national or sub-national levels are better able than WHO to tailor implementation strategies to their specific circumstances. However, they frequently lack capacity and resources to do this. WHO headquarters and regional offices can play an important role in supporting member states in their efforts to implement recom-

6. 7.

104

Oxman AD, Sch端nemann H, Fretheim A: Improving the use of research evidence in guideline development: 16. Evaluation. Health Res Policy Syst 2006, 4:28. Sch端nemann H, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 13. Adaptation, applicability and transferability. Health Res Policy Syst 2006, 4:25. El Arifeen S, Blum LS, Hoque DM, Chowdhury EK, Khan R, Black RE, Victora CG, Bryce J: Integrated Management of Childhood Illness (IMCI) in Bangladesh: early findings from a cluster-randomised study. Lancet 2004, 364:1595-1602. Laing R, Hogerzeil H, Ross-Degnan D: Ten recommendations to improve use of medicines in developing countries. Health Policy Plan 2001, 16:13-20. Moynihan R, Oxman AD, Lavis J, Paulsen E: A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research. Oslo, Norwegian Knowledge Centre for the Health Services; 2006. Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J: Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care 2003, 15:31-45. Oxman AD, Fretheim A, Sch端nemann S: Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst 2006, 4:12.

Page 3 of 4 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:27

8. LA, Grilli R, Grimshaw JM, Harvey E, Oxman[8]) AD, for Thomson MA: and Bero the Guidelines International Network existing Closing the gap between research and practice: an overview systematic reviews and relevant methodological research of systematic reviews of interventions to promote the implethat mentation address these questions. The answers to the questions of research findings. The Cochrane Effective Practice and Organization Care Review Group. BMJ con1998, are our conclusions based onofthe available evidence, 317:465-468. sideration and otherG,organisations are 9. GrimshawofJM,what ShirranWHO L, Thomas R, Mowatt Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA: Changing provider behavior: doing, and logical arguments.

an overview of systematic reviews of interventions. Med Care 2001, 39:II2-45. For this review we searched PubMed usingHealth (clinical 10. Getting evidence into practice. Effective Care prac1999, 5:1-16. tice guidelines or public health guidelines) and (priority 11. National Guideline Clearinghouse Agency for Healthcare Research setting setting and related articles for selected and or Quality 2006priorities) [http://www.guidelines.gov]. references [9,10]. We searched the(GIN) Cochrane Methodol12. Guidelines International Network Guidelines International [http://www.g-i-n.net]. ogy Network Register2006 using priority or priorities. We reviewed the 13. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale Conference onM,Priorities website of the 5th International L, Whitty P, Eccles MP, Matowe L, Shirran L, Wensing Dijkstra R, Donaldson C: [11] Effectiveness and efficiency of guideline dissemin Health Care and references that we had in our files ination and implementation strategies. Health Technol Assess [12-15]. 2004, 8:iii-72. 14. Haines A, Kuruvilla S, Borchert M: Bridging the implementation gap between knowledge and action for health. Bull World Findings Health Organ 2004, 82:724-731 [http://www.who.int/bulletin/vol What criteria should be used to establish priorities? umes/82/10/724.pdf]. 15. S, Vist G, Trommald M, Oxman A: Health policy-makers' The Innvaer US Institute of Medicine's (IOM) Committee on perceptions of theirPriorities use of evidence: a systematicDevelopreview. J Methods for Setting for Guidelines Health Serv Res Policy 2002, 7:239-244. mentLavis in its studyH,ofOxman setting for clinicalK,practice 16. J, Davies A, priorities Denis JL, Golden-Biddle Ferlie E: Towards systematicinreviews that inform care manguidelines published 1995 argued that health the priority setand policy-making. J Health Serv Res Policy 2005, 10 ting agement process should be open and defensible [14] They recSuppl 1:35-48. ommended six general criteria: prevalence, burden of 17. Research-based continuing education and quality improvement www org 2006the [http://www.rebeqi.org]. illness, cost ofrebeqi managing problem, variability in prac-

tice, potential of a guideline to improve health outcomes, and potential of a guideline to reduce costs. Because data used to make these judgements is often lacking, they suggested explicit opportunities for important unmeasured factors to be considered. They further suggested separate "tracks" for considering issues for specific populations, conditions or concerns. They argued that the rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration. They suggested that updating of guidelines should also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence. Oortwijn identified 25 criteria used to prioritise health technology assessments and categorised these into four broad categories: burden of disease, potential effects, potential costs, and uncertainty regarding application of the technology [15]. In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]: 1. The topic is clinically important affecting large numbers of people with substantial morbidity or mortality (the burden of illness). 2. The topic is complex enough to initiate debate about the recommendations.

http://www.health-policy-systems.com/content/4/1/27 http://www.health-policy-systems.com/content/4/1/14

3. There is evidence of variation between actual and appropriate care. 4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available. 6. The recommendations will be acceptable to the potential users. 7. Implementation of the guideline is feasible, will not exhaustively use the communities' resources, and barriers to clinical change are not so high that they cannot be overcome. While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]). What processes should be used to agree on priorities? Batista and Hodge in a review conducted 10 years ago found only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the following framework for priority setting:

They also suggested there is a need to define more narpapers will be: rowly Your andresearch precisely topics for guideline development. available free of charge to the entire biomedical community They argued that this would result in more efficient organupon acceptance ization peer of reviewed panels and andpublished their immediately work, resolution of some cited in PubMed and archived on PubMed Central apparent controversies, more responsive guidelines, and yours — you keep the copyright easier implementation. BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

105

Page 4 of 7 4 (page number not for citation purposes)

Health Research Policy and Systems Improving the use of research evidence in guideline development

BioMed Central

Open Access

Review

Improving the use of research evidence in guideline development: 16. Evaluation Andrew D Oxman*1, Holger J Schünemann2 and Atle Fretheim1 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway and 2INFORMA, S.C. Epidemiologia, Istitituto Regina Elena, Via Elio Chianesi 53, 00144 Rome, Italy Email: Andrew D Oxman* - oxman@online.no; Holger J Schünemann - hjs@buffalo.edu; Atle Fretheim - atle.fretheim@nokc.no * Corresponding author

Published: 08 December 2006 Health Research Policy and Systems 2006, 4:28

doi:10.1186/1478-4505-4-28

Received: 07 April 2006 Accepted: 08 December 2006

This article is available from: http://www.health-policy-systems.com/content/4/1/28 © 2006 Oxman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

106

Page 1 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:28

http://www.health-policy-systems.com/content/4/1/28 http://www.health-policy-systems.com/content/4/1/14

and• the Periodic Guidelines reviewInternational of guidelines by Network experts[8]) not for involved existing in developing alsobetween be considered. 3. Therethe is guidelines evidence ofshould variation actual and approprisystematic reviews and relevant methodological research ate care. • Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with that address these questions. The answers to the questions members serving fixed periods with a rotating membership. are our conclusions based on the available evidence, con4. There are no existing valid guidelines available to use. How should the WHO impactand of guidelines or recommendations be evaluated? sideration of what other organisations are doing, and headquarters logical arguments. 5. There is an amount of for existing available. • WHO and regional offices should support member states andadequate those responsible policyevidence decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, Forpractical this review we searched PubMedofusing (clinical prac6. The recommendations will be acceptable to the potential support and coordination efforts. tice guidelines or public health guidelines) and (priority users. • Before-after evaluationsand should be used cautiously and when there are important uncertainties regarding the effects of setting or setting priorities) related articles for selected a policy [9,10]. or its implementation, randomised evaluations should7. beImplementation used when possible. references We searched the Cochrane Methodolof the guideline is feasible, will not exhausogyWhat Register using priority or priorities. We reviewed the tively use the communities' resources, and barriers to clinical responsibility should WHO take for ensuring that important uncertainties are addressed by future th International Conference on Priorities website of the 5 change are not so high that they cannot be overcome. research when the evidence needed to inform recommendations is lacking? in Health Care [11] and references that we had in our files • Guideline panels should routinely identify important uncertainties and research priorities. This source of potential [12-15]. While burden of disease is commonly used as a criterion priorities for research should be used systematically to inform priority-setting processes for global research. for priority setting, it should be noted that the use of summary burden of disease measures, such as disability Findings What criteria should be used to establish priorities? adjusted life years (DALYs) has been criticised for focusing The US Institute of Medicine's (IOM) Committee on on disease rather than resource use and interventions, Background What WHO is doing now Methods for Setting Priorities for Guidelines Developbecause of the assumptions about values inherent in such The World Health Organization (WHO), like many other The Guidelines for WHO Guidelines suggest that ment in its study of setting priorities for clinical practice measures, and because of the technical limitations of draft such organisations around the world, has recognised the need guidelines should be subjected to a self-test by the techniguidelines published in 1995 argued that the priority setmeasures (see for example references [17] and [18]). to more should rigorous ensure that health cal development and the steering/liaison groups using a tinguse process beprocesses open andtodefensible [14] They care recrecommendations are informed by the bestburden available checklist (Table should 1) [3]. be used to agree on priorities? What processes ommended six general criteria: prevalence, of researchcost evidence. This is the the problem, last of a series of 16 in reviews Batista and Hodge in a review conducted 10 years ago illness, of managing variability practhat potential have beenofprepared as to background for advice from However, the checklist is not being used and most guidefound only three articles pertinent to priority setting for tice, a guideline improve health outcomes, the WHO Advisory Committee on Health Researchdata to lines appear to beguidelines deficient [4]. AnThey unpublished, house clinical practice [10]. suggestedinthe foland potential of a guideline to reduce costs. Because WHO howthese to achieve this. is often lacking, they sugreview of WHO guidelines the AGREE appraisal lowing framework for priorityusing setting: used toon make judgements instrument [5] found that the vast majority of guidelines gested explicit opportunities for important unmeasured Providing technical advice to its member states is a core did not meet most of the AGREE criteria [6]. Although 1. Consult with end users and other stakeholders before factors to be considered. They further suggested separate function of the World Health Organization (WHO). draft guidelines are often sent for peer review, and the selecting topics. "tracks" for considering issues for specific populations, Ensuring the of the advice that is given is an inherGuidelines for WHO Guidelines suggests external review, conditions orquality concerns. They argued that the rationales for ent responsibility of WHO. In this paper we address the this is not always done. WHO's Regulations for Expert 2. Consider feasibility during the consultation. designating special tracks should be defined clearly; i.e. following questions related to evaluation of guidelines Committees, in fact, require that "The expert committee why they warrant special consideration. They suggested and their implementation: shall draw up and approve its report before the closure of its 3. Document the process used to select guideline topics. that updating of guidelines should also be considered as a meeting." [7] separate "track", taking account of issues such as the need •forHow should and the quality of guidelines orevidence. recommendaThe IOM suggested the following procedures [14]: corrections the availability of new tions be appraised? We are not aware of any assessments of the extent to which WHO guidelines, recommendations or policies are • the use of Delphi-like procedures for obtaining expert Oortwijn identified 25 criteria used to prioritise health • When should guidelines or recommendations be kept up to date or any policies for withdrawing ones that judgments or topic rankings through correspondence technology assessments and categorised these into four updated? are out of date. The Guidelines for WHO Guidelines offers broad categories: burden of disease, potential effects, the encouragement for undertaking rigorous • thefollowing use of questions that are specific, explicit and conpotential costs, and uncertainty regarding application of • How should the impact of guidelines or recommendastudieswith to provide missing evidence, but we are not aware sistent standard methods for questionnaire constructhe technology [15]. tions be evaluated? of any other policies linking important uncertainties in tion guidelines to WHO's priorities for research: "Sometimes it In a more recent selective review for the New Zealand •Guidelines What responsibility should WHO take for ensuring that will be necessary to issue where no rigorous • experimentation with guidelines more formal procedures tostudies arrive Group, the following criteria were identified as important uncertainties are addressed by future research exist, based on the best available evidence. But after issuance of at group judgments indicating that a topic is suitable for guideline developwhen the evidence needed to inform recommendations is such guidelines, the opportunity could be taken to undertake ment [16]: lacking? rigorous studies to provide missing evidence to evaluate the They also suggested there is a need to and define more nareffectiveness or impact of the guidelines in the actual settings rowly and precisely topics for guideline development. 1. The topic is clinically important affecting large numbers of Related questions updating where they arethat intended to be result used. This would allow them to They argued this would in more efficient organpeople with substantialregarding morbidity or mortality of (thesystematic burden of reviews and implementation are addressed in other artibe revised or updated if needed." ization of panels and their work, resolution of some illness). cles in this series [1,2]. apparent controversies, more responsive guidelines, and What other organisations are doing easier implementation. 2. The topic is complex enough to initiate debate about the recIn a survey of 152 organizations that produce guidelines, ommendations. technology assessments, or support the use of research in

107

Page 4 2 of 7 (page number not for citation purposes)

the use of 4:28 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/28

Table 1: Checklist for WHO Treatment Guidelines (from Guidelines for WHO Guidelines [3]).

Yes

Questions

Reference Points

Origin 1 Are the Cluster and Department issuing the guidelines clearly identified? Objective, target audience 2 Does the guideline list its objectives, including the patient categories and situation(s) for which the guidelines are intended? 3 Does the guideline describe the professional groups to which it is addressed? Guideline Development Group 4 Does the Guideline Development Group include all relevant professional groups, public health experts and end users, including individuals from geographic areas where the guidelines will be applied? 5 Does the Group include methodological experts in fields such as search methodology, critical appraisal and cost-effectiveness analysis? Conflict of interest 6 Are all funding sources named, and is there no conflict of interest? 7 Have all members of the Guideline Development Group and external reviewers declared their interests, and have these interests been recorded in the guideline document? 8 Does the document describe the method used to minimize any undue influence on the Guideline Development Group and the external reviewers? Evidence 9 Was there a systematic comprehensive search for evidence, and has the search strategy been recorded in the guideline? 10 Has the strength and quality of the evidence on effectiveness been graded? 11a 11b 12 13

What percent of recommendations are evidence-based?* Are the recommendations which are not evidence-based explicitly labeled as "expert opinion" based? Is there explicit consideration of other issues, such as safety and potential misuse in a variety of settings? Is there explicit consideration of issues of cost effectiveness?

14 Is the strength of the recommendation linked to the evidence? 15 Do the recommendations take into account potential resource constraints? Review 16 Were the comments by the external peer review adequately addressed? 17 Did all members of the Guideline Development Group approve the final document? 18 Did all members of the Steering Group approve the final document? 19 Is there a plan for reviewing new evidence and updating the guideline? Presentation, clarity 20 Are the recommendations clearly formulated? 21 Does the guideline identify and advise on ineffective practices? Implementation plan 22 Is there a plan for dissemination and local adaptation of the guideline? 23 Are funds available for dissemination and local adaptation for the guideline? 24 Are there suggested criteria for monitoring the use in intended settings?

Introduction Introduction Introduction List of members of the guideline development group List of members of the guideline development group List of funding sources Annex on documentation of process Annex on documentation of process Annex on documentation of process Annex on documentation of process; evidence table Summary of recommendations Summary of recommendations Annex on documentation of process; evidence table Annex on documentation of process; evidence table Summary of Recommendations Implementation issues Annex on documentation of process Annex on documentation of process Annex on documentation of process Introduction Summary of Recommendations Summary of Recommendations Companion document Companion document Implementation Issues

This Checklist is intended for the following purposes: (1) As a guide for developing or updating WHO treatment Guidelines. (2) As a check-list for Executive and Regional Directors when giving final approval for publication. To qualify for publication and inclusion in the WHO database of treatment guidelines, a tick mark signifying YES must be placed beside all the 24 criteria, except 11a. *These are recommendations based on information other than expert opinion.

developing health policy a large majority reported using both an internal review process (80%) and external review by experts (82%) [8]. Only 44% reported external review by target users (58% of guideline producers), 43% reported comparing their products with products or input from other groups, and 31% reported using pilot testing.

Fifty-two percent of the units that produced guidelines reporting updating them regularly and 45% reported updating irregularly. Thirty-five percent reported collecting data about uptake systematically, and 32% reported systematically evaluating the usefulness or impact of their guidelines in some other way.

108

Page 3 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:28

andanother In the Guidelines survey of International 18 prominent Network guidelines [8]) fordevelopexisting ment programs, systematic reviews more and than relevant halfmethodological reported monitoring research or evaluating that addressthe these effects questions. of at The least answers some toguidelines. the questions All reported are our conclusions using some based typeonofthequality available system evidence, for good conguideline development. sideration of what WHO Eleven and used other both organisations external and are internaland doing, review, logicalsixarguments. used external review only, and one internal review only. In addition seven compared their guidelines For this review withwe guidelines searched from PubMed other using groups (clinical and three pracusedguidelines tice pilot testing. or All public reported health updating guidelines) theirand guidelines (priority at least occasionally, setting or setting priorities) althoughand only related half had articles formal for selected update procedures.[9,10]. We searched the Cochrane Methodolreferences ogy Register using priority or priorities. We reviewed the website of the 5th International Conference on Priorities Methods The methods prepare this review in in Health Careused [11]to and references that weare haddescribed in our files the introduction to this series [9]. Briefly, the key ques[12-15]. tions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Findings What criteria should be usedWe to did establish priorities? Research Evidence (SURE). not conduct a full sysThe US review. Institute Medicine's (IOM) on tematic Weof searched PubMed and Committee three databases Methods for Settingstudies Priorities Guidelines Developof methodological (the for Cochrane Methodology ment in its of settingGuideline priorities Clearinghouse, for clinical practice Register, thestudy US National and guidelines published in 1995Network argued that the priority setthe Guidelines International for existing systemting be openmethodological and defensible [14] They that recatic process reviews should and relevant research ommended general The criteria: prevalence, burden are of address thesesix questions. answers to the questions illness, cost of managing pracour conclusions based onthe theproblem, availablevariability evidence, in considtice, potential ofWHO a guideline to improve health outcomes, eration of what and other organisations are doing, and logical potential of a guideline to reduce costs. Because data arguments. used to make these judgements is often lacking, they suggested for that important For thisexplicit reviewopportunities we used articles we hadunmeasured previously factors to be considered. Theyoffurther separate identified, including a review clinicalsuggested practice guideline "tracks" considering issues for specific populations, appraisalfor instruments [10], to locate related articles in conditions or searched concerns.the They argued that the rationales for PubMed; we National Guidelines Clearingdesignating special tracks should be defined clearly; i.e. house annotated bibliography using the category guidewhy they warrant consideration. suggested line evaluation withspecial the terms appraisal or They impact, and for that updating of guidelines should be considered as a all categories using updating; and also we checked the referseparate takingarticles. accountWe of issues such as the ence lists"track", of retrieved also searched forneed and for corrections the availability of new evidence. scanned reviewsand of methods for setting research priorities that were linked to guidelines development programs by Oortwijn 25 reviews criteria of used to prioritise health searching identified PubMed for research priorities, for technology andresearch categorised these and into pracfour articles that assessments addressed both priorities broad categories: of disease, effects, tice guidelines, andburden by searching the Webpotential using Google for potential and uncertainty regarding of sites that costs, addressed methods for setting application priorities for the technology [15].research priorities. research and global In a more recent selective review for the New Zealand Findings How shouldGroup, the quality of guidelines or recommendations Guidelines the following criteria were identified as be appraised? indicating that a topic is suitable for guideline developGraham and colleagues in a systematic review of instrument [16]: ments for assessing the quality of clinical practice guidelines instruments published up to 1999 [10]. All 1. Thefound topic 13 is clinically important affecting large numbers of instruments were developed after 1992 and(the contained 8 people with substantial morbidity or mortality burden of to 142 questions or statements. Only the Cluzeau instruillness). ment, which formed the basis of the AGREE instrument [11,12]' included at enough least one item for each of the the rec10 2. The topic is complex to initiate debate about attributes that the authors identified across instruments. ommendations.

http://www.health-policy-systems.com/content/4/1/28 http://www.health-policy-systems.com/content/4/1/14

This instrument that of between Shaneyfelt and colleagues 3. There is evidenceand of variation actual and appropri[13]. were the only instruments that had been validated. ate care. They concluded that there was insufficient evidence to support exclusive of guidelines any one instrument, 4. There the are no existinguse valid available to although use. the AGREE instrument has received the most evaluation. Vlayen colleagues updatedofthe review by Graham and 5. Thereand is an adequate amount existing evidence available. colleagues up to 2003 [14]. They found 24 different appraisal tools. The 24 will instruments included total of 6. The recommendations be acceptable to thea potential 469 questions that they also grouped into 10 dimensions: users. validity, reliability/reproducibility, clinical applicability, clinical flexibility, multidisciplinary process, clarity, 7. Implementation of the guideline is feasible, will not exhausscheduled dissemination, and tively use thereview, communities' resources, implementation, and barriers to clinical evaluation. They found that addressed change are not so high thatthree they instruments cannot be overcome. all 10 dimensions and three additional instruments based on theburden Cluzeau one of which, AGREE While of instrument, disease is commonly used asthe a criterion instrument, was theitonly onebe to noted have been validated. They for priority setting, should that the use of sumfound burden that the of AGREE instrument wassuch a validated, easymary disease measures, as disability to-use, and instrument, which was internaadjusted life transparent years (DALYs) has been criticised for focusing tionally developed and widely noted two on disease rather than resourceaccepted, use andbut interventions, limitations that they considered although it because of the assumptions about important: values inherent in such can be usedand to compare clinical practicelimitations guidelines,of it does measures, because of the technical such not set a threshold to classify them as[17] goodand or bad, measures (see for example references [18]).and it does not assess the quality of the evidence supporting the What processes should be used to agree on priorities? recommendations. Batista and Hodge in a review conducted 10 years ago found only three articleswas pertinent to priority for The AGREE instrument developed throughsetting a process clinical practice guidelines They suggested the and folof item generation, selection[10]. and scaling, field-testing lowing framework priority setting: refinement [5]. Thefor final version of the instrument contained 23 items grouped into six domains: scope and pur1. Consult with end users and other stakeholders before pose, stakeholder involvement, rigour of development, selecting topics. clarity and presentation, applicability, and editorial independence. 2. Consider feasibility during the consultation. When should guidelines or recommendations be updated? Shekelle and the colleagues, a review of 17topics. guide3. Document process based used toon select guideline lines published by AHRQ, estimated that no more than 90%IOM weresuggested still valid the afterfollowing 3.6 yearsprocedures and that about The [14]:half the guidelines were outdated in 5.8 years. They recommend that should be reassessedforevery three expert years, â&#x20AC;˘ theguidelines use of Delphi-like procedures obtaining based on the 95% confidence for their estijudgments or lower topic rankings throughinterval correspondence mate of when one of ten guidelines would no longer be up-to-date. suggest ways explicit of expeditiously â&#x20AC;˘ the use of They questions thatseveral are specific, and conassessing thestandard need formethods updatingfor guidelines including consistent with questionnaire construcducting limited searches by groups that are familiar with tion the topic, such as Cochrane review groups, focusing searches on research thatmore the guidelines panel considered â&#x20AC;˘ experimentation with formal procedures to arrive would a pivotal role in requiring revision of the at groupplay judgments guideline, periodic review of the guidelines by experts not involved developing theis guidelines, and considering They alsoin suggested there a need to define more narguidelines as for an ongoing rather rowly and development precisely topics guidelineprocess, development. than argued a discreet with members of guideline panels They thatevent, this would result in more efficient organserving of fixed periods with work, a rotating membership. ization panels and their resolution of some apparent controversies, more responsive guidelines, and Gartlehner and colleagues compared the approach sugeasier implementation. gest by Shekelle and colleagues of a limited search using

109

Page 4 of 7 (page number not for citation purposes)

the use of 4:28 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/28

review articles, commentaries and editorials, to a conventional process using typical systematic review methods in terms of comprehensiveness and effort [15]. They applied both approaches independently to assess the need to update six topics from the 1996 Guide to Clinical Preventive Services from the US Preventive Services Task Force [16]. They found that although the limited search approach identified fewer eligible studies than the traditional approach, none of the studies missed was rated as important by task force members acting as liaisons to the project with respect to whether the topic required an update. On average, this approach produced substantially fewer citations to review than the traditional approach. The effort involved and potential time saving depended largely on the scope of the topic. They found that involving experts in assessing how current the guidelines were was not helpful, in contrast to Shekelle and colleagues.

designs, including interrupted time series analyses, controlled before-after studies and uncontrolled before-after studies [20]. The advantage of using randomised designs for impact assessments is that they give greater confidence that the measured impact of a program is attributable to whatever implementation strategy was used and not to some other factor [21-23]. It is generally not possible to predict differences in the size, or even the direction, of estimates of treatment effects for the same intervention when it is generated in randomized and non-randomized studies [22]. There have been similar findings for impact evaluations of development programs [21] and implementation strategies. For example, a systematic review of continuous quality improvement found improvements in 41 of 43 single site before-after studies and most of 13 multi site before-after studies, but no improvements in three randomised trials [24].

Johnston and colleagues found that an updating strategy for cancer practice guidelines identified 80 pieces of new evidence over a one-year period relating to 17 of 20 guidelines [17]. They found on average four pieces of new evidence per guideline, but there was considerable variation across the guidelines. Of the 80 pieces, 19 contributed to modifications of clinical recommendations in six practice guidelines, whereas the remaining evidence supported the original recommendations. Their updating process yielded important findings, but was resource intensive. They found that it would be possible to reduce the scope of the sources searched routinely to MEDLINE, the Cochrane Library and meeting proceedings.

A wide variety of techniques to gather data have been used singly or in combinations, including questionnaires, interviews, observation, audit and using routinely collected data. Self-report may not be consistent with more objective measures of practice. Collecting reliable data in low and middle-income countries (LMIC) can be a major challenge, where available records and routinely collected data may be lacking. We did not find any systematic reviews of strategies for collecting data for impact evaluations in LMIC. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? Priority-setting exercises for global health research have used various methods and processes [25]. We have not found examples of priority setting programs based on important uncertainties identified in guidelines. A number of exercises have, however, used systematic reviews to inform priority-setting processes [26,27]. A comparison of four sources of potential priorities for the NHS Health Technology Assessment Programme found that a widespread consultation of healthcare commissioners, providers and consumers was the largest source of suggestions, but the success rate of this source, in terms of being commissioned, was low. Research recommendations from systematic reviews provided the second largest source of priorities and the best success rate of all sources.

The findings of these three studies of the need to update guidelines is consistent with findings from studies of the need to update systematic reviews, which generally support the conclusion that in situations where time or resources are limited, thorough quality assessments should likely take precedence over extensive literature searches [1]. How should the impact of guidelines or recommendations be evaluated? Strategies ranging from passive dissemination to intensive, complex interventions have been used to implement guidelines and a range of study designs has been used to evaluate the impact of these strategies using a range of outcome measures [18,19]. Passive strategies have often not been effective, however there is limited evidence to support decisions about which guideline dissemination and implementation strategies are likely to be efficient under different circumstances [2,18].

Discussion

There are at least 24 different instruments available for assessing the quality of clinical practice guidelines. We did not find similar tools developed for assessing the quality of public health or health policy recommendations, although the domains that are addressed by clinical prac-

Study designs that can be used to evaluate the impact of guidelines include randomised designs, particularly cluster randomised trials, a range of observational study

110

Page 5 of 7 (page number not for citation purposes)

Health Research Policy and Systems 2006, 4:14 4:28

and the tice guidelines Guidelines appraisal International instruments Network are[8]) applicable for existing to public health systematic reviews and health and relevant policy recommendations. methodological research that address these questions. The answers to the questions Up our are to now, conclusions self-assessment based on of the guidelines available using evidence, the Guideconlines for WHO sideration of what Guidelines WHO checklist and other has organisations not been successare ful. Moreover, doing, and logical most arguments. guidelines programmes rely on external review, as well as internal review. WHO should put this For into review place processes we searched to ensure PubMed that using both(clinical internalpracand external tice guidelines revieworof public guidelines health guidelines) is undertaken and routinely (priority using appropriate setting or setting priorities) criteria. and related articles for selected references [9,10]. We searched the Cochrane MethodolProcesses should be put into placeWetoreviewed ensure that ogy Register using also priority or priorities. the Conference on Priorities guidelines routinely to determine if they website of are the monitored 5th International in Health Care and references that we this had in are in need of [11] updating. To ensure that is our donefiles as [12-15]. expeditiously as possible, people who are familiar with the topic, such as Cochrane review groups, should conduct limited searches routinely. Guideline panels should Findings What criteria should used require to establish priorities? identify research thatbe would revision of the guideThe Medicine's Committee on line US andInstitute searchesofshould focus(IOM) particularly on this Methods Setting Priorities for Guidelines Developresearch. for Periodic review of guidelines by experts not ment in its study of setting priorities should for clinical involved in developing the guidelines also practice be conguidelines published in 1995 argued that the priority setsidered, and consideration should be given to establishing ting process should beare open and defensible [14] They recguideline panels that ongoing with members serving ommended general criteria: prevalence, burden of fixed periodssix with a rotating membership. illness, cost of managing the problem, variability in practice, potential of a guideline health outcomes, Recommendations may needtotoimprove be adapted to specific setand a guideline to reduce costs. settings, Because data tings,potential can onlyofbe implemented in specific and used make these judgements is often sugtheir to impact can only be assessed in lacking, specific they settings. gested explicit opportunities for offices, important unmeasured WHO headquarters and regional however, should factors be considered. They further suggested separate supporttomember states and those responsible for deciding "tracks" for considering issues for specific populations, and implementing policies to evaluate the impact of their conditions or concerns. Theyregarding argued that the rationales for policies by providing advice impact assessment, designating special tracks should be clearly; i.e. practical support and coordination of defined efforts. Before-after why they warrant suggested evaluations shouldspecial be usedconsideration. cautiously, if atThey all, and when that of guidelines shouldregarding also be considered as a thereupdating are important uncertainties the effects of separate taking account of issues such evaluations as the need policy or"track", its implementation, randomised for corrections thepossible. availability of new evidence. should be usedand when Oortwijn 25 criteria used to prioritise health Guideline identified panels should routinely identify important technology categorised these into four uncertaintiesassessments and researchand priorities. This source of potenbroad categories: burdenshould of disease, effects, tial priorities for research be usedpotential systematically to potential costs, and uncertainty inform priority-setting processes regarding for global application research. of the technology [15].

Further work

Work is needed to selective ensure that the AGREE or a In a more recent review for theinstrument New Zealand similar instrument is following suitable for assessing the broad Guidelines Group, the criteria were identified as range of guidelines, recommendations and policies that indicating that a topic is suitable for guideline developWHO[16]: produces. In particular, its suitability for assessing ment public health and health policy recommendations should be The assessed. items should also benumbers added to 1. topic isAdditional clinically important affecting large of addresswith concerns about equity,orwhich are (the not burden currently people substantial morbidity mortality of addressed in the AGREE instrument, and considerations illness). that are specific to guidelines that are developed internationally rather than inenough a specific country or setting. Work 2. The topic is complex to initiate debate about the recis also needed on developing practical methods to collect ommendations.

http://www.health-policy-systems.com/content/4/1/28 http://www.health-policy-systems.com/content/4/1/14

reliable that can be usedbetween in impact evaluations in 3. There data is evidence of variation actual and appropriLMIC. ate care. 4. There are nointerests existing valid guidelines available to use. Competing ADO and AF work for the Norwegian Knowledge Centre forthe Services,amount an agency funded by theavailable. Norwe5. ThereHealth is an adequate of existing evidence gian government that produces systematic reviews and health assessments. three authors con6. The technology recommendations will be All acceptable to the are potential tributors to the Cochrane Collaboration. ADO and HJS users. are members of the GRADE Working Group. HJS is documents editor andofchair of the documents development 7. Implementation the guideline is feasible, will not exhausand implementation committee thebarriers American Thotively use the communities' resources,forand to clinical racic Society senior of the be American College of change are notand so high thateditor they cannot overcome. Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines. While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability Authors' contributions ADO prepared the(DALYs) first draft thiscriticised review. HJS and AF adjusted life years hasof been for focusing contributed to drafting revising it. and interventions, on disease rather thanand resource use because of the assumptions about values inherent in such measures, and because of the technical limitations of such Acknowledgements We gratefully(see acknowledge WHO and the Norwegian measures for example references [17] Knowledge and [18]).Centre forthe Health Services for financial support, the other authors of these reviews processes for their contributions, othertomembers of the ACHR SURE, What should bethe used agree on priorities? and all of those who have provided us with feedback on earlier versions of Batista and Hodge in a review conducted 10 years ago this paper and the others in this series.

found only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the folReferences lowing framework for priority setting: 1. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of

Research Evidence in Guideline Development: 8. Synthesis

and presentation evidence. Policy Syst 2006.before 1. Consult with endofusers and Health otherRes stakeholders 2. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of selecting topics. Research Evidence in Guideline Development: 15. Dissemi-

nating and implementing guidelines. Health Res Policy Syst 2006. Global Programme on Evidence for Health Policy: Guidelines for 2. Consider feasibilityGeneva: during the Health consultation. WHO Guidelines. World Organisation, 2003 (EIP/ GPE/EQC/2003.1). 4. Panisett U: Athe review of WHO published 3. Document process usedrecommendations to select guideline topics.in 2005. . 5. AGREE Collaboration: Development and validation of an interappraisalthe instrument assessing the quality of The national IOM suggested followingforprocedures [14]: clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12:18-23. • theOxman use of procedures forresearch obtaining expert 6. AD,Delphi-like Lavis J, Fretheim A: The use of evidence in WHO recommendations. judgments or topic rankings. through correspondence 7. Regulations for Expert Advisory Panels and Committees: In World Health Organisation. Basic Documents, 2003; 101-9. Text the Thirty-fifth World Healthexplicit Assembly (resolu• theadopted use of by questions that are specific, and contionwith WHA35.10), inmethods replacement of the regulations adopted sistent standard for questionnaire construcby the Fourth World Health Assembly, 1951. Amendments tion were adopted at the Forty-fifth, Forty-ninth, Fifty-third and Fifty-fifth World Health Assemblies (decision WHA45(10), resolution WHA49.29, resolution WHA53.8 and resolution • experimentation with more WHA55.24, respectively. . formal procedures to arrive at group judgments 8. Moynihan R, Oxman AD, Lavis JN, Paulsen E: Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier. In A review of organizations that support the use of research Theyevidence also suggested there is technology a need assessments, to define and more in developing guidelines, healthnarpolicy, and for the precisely WHO Advisory Committee Health Research Oslo: Norwerowly topics foronguideline development. Knowledge Centre for the Health Services; 2006. Theygian argued that this would result in more efficient organ9. Oxman AD, Fretheim A, Schünemann HJ: Improving the use of ization of panels and their work, resolution of some research evidence in guideline development: introduction. Health Res Policy Syst 2006.more responsive guidelines, and apparent controversies, 10. Graham ID, Calder LA, Hebert PC, Carter AO, Tetroe JM: A comeasier implementation. parison of clinical practice guideline appraisal instruments. Int J Technol Assess Health Care 2000, 16:1024-38. 3.

111

Page 4 6 of 7 (page number not for citation purposes)

the use of 4:28 research evidence in guideline development Health Research PolicyImproving and Systems 2006, http://www.health-policy-systems.com/content/4/1/28

11.

12. 13.

14.

15.

16. 17.

18. 19. 20. 21.

22. 23.

24. 25.

26.

27.

The AGREE Collaboration: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003, 12:18-23. AGREE Instrument: Appraisal of Guidelines Research and Evaluation (AGREE) Collaboration. [http://www.agreecollabora tion.org/instrument/]. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer reviewed medical literature. JAMA 1999, 281:1900-5. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D: A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005, 17:235-42. Gartlehner G, West SL, Lohr KN, Kahwati L, Johnson JG, Harris RP, et al.: Assessing the need to update prevention guidelines: a comparison of two methods. Int J Qual Health Care 2004, 16:399-406. US Preventive Services Task Force: Guide to Clinical Preventive Services. 2nd edition. Alexandria, VA: International Medical Publishing; 1996. Johnston ME, Brouwers MC, Browman GP: Keeping cancer guidelines current: results of a comprehensive prospective literature monitoring strategy for twenty clinical practice guidelines. Int J Technol Assess Health Care 2003, 19:644-55. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, et al.: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004, 8:. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, et al.: Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001, 39:II2-45. Eccles M, Grimshaw J, Campbell M, Ramsay C: Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 2003, 12:47-52. Savedoff W, Levine R, Birdsall N: hen We Will Ever Learn? Closing the Evaluation Gap. Consultation Draft. Center for Global Development [http://www.cgdev.org/section/initiatives/_active/eval gap/]. (Accessed 16 March 2006). Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include. Health Res Policy Syst 2006. Kleijnen J, Gøtzsche P, Kunz RA, Oxman AD, Chalmers I: So what's so special about randomisation. In Non-Random Reflections on Health Care Research: On the 25th Anniversary Of Archie Cochrane's Effectiveness and Efficiency Edited by: Chalmers I, Maynard A. London: BMJ Publishers; 1997:93-106. Shortell SM, Bennett CL, Byck GR: Assessing the impact of continuous quality improvement on clinical practice: what it will take to accelerate progress. Milbank Q 1998, 76:593-624. Global Forum for Health Research: Priority setting in health research. The 10/90 Report on Health Research 2003–2004 [http:// www.globalforumhealth.org/filesupld/1090_report_03_04/ 109004_chap_4.pdf]. (Accessed 16 March 2006). The James Lind Alliance: Confronting important uncertainties about the effects of treatments. Health and Social Campaigners News 2004:3-10 [http://www.jameslindlibrary.org/pdf/ effects_of_treatments.pdf]. (Accessed 16 March 2006). Chase D, Milne R, Stein K, Stevens A: What are the relative merits of the sources used to identify potential research priorities for the NHS HTA programme? Int J Technol Assess Health Care 2000, 16:743-50.

BioMedcentral central

Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp

112

Page 7 of 7 (page number not for citation purposes)