[Title will be auto-generated] by Duncan Dibble

Introduction to Comparative Effectiveness Research (CER)

Editor: Ruth Whittington MSc(Hons), NZSRN CEO Rx Values Group Ltd Formerly Rx Communications Ltd and Greenflint Ltd, Rx Values Group Ltd comprises Rx Medical Communications Ltd, Rx Market Access Ltd, and Rx Evidence Ltd. Rx works with the leaders in pharmaceuticals, biotechnology, healthcare provision and academia to enhance the value of healthcare through evidence, communications and training. This book has been reprinted by Genentech with permission from Rx Values Group Ltd.

Introduction to Comparative Effectiveness Research

© 2011 Rx Values Group Ltd, Flintshire, UK. Adapted from ISBN - 978-0-9545494-7-3 All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system, without permission in writing from Rx Values Group Ltd. Although great care has been taken in compiling and checking the information in this book to ensure it is accurate, Rx Values Group Ltd shall not be held responsible for the continued currency of the information or for any errors, omissions or inaccuracies in this publication. The opinions expressed in this publication are not necessarily those of Rx Values Group Ltd.

Introduction to Comparative Effectiveness Research

Foreword Comparative effectiveness research (CER) is a term that has come to the forefront in recent times; particularly as regulatory authorities deliberate how much more effective and/or safe new therapies may be compared to existing alternatives, and payers struggle to manage tight budgets and evermore expensive healthcare demands. The ability to accurately assess the features, risks, benefits, and costs of one treatment or healthcare program versus another is an important undertaking; especially where there are many evaluation methods and arguments used, multiple therapeutic options available, and a potentially large budget impact of treatment choices. Evidence-based medicine (EBM) involves the use of the best available clinical evidence, including information provided by CER, to select the most appropriate treatment for individual patients. The development of treatment guidelines and algorithms all depends on the evidence; however, assessing, interpreting and, above all, applying that evidence is a complex task. This booklet gives an overview of some of those methods of assessment and evidence collection. This booklet is intended for a broad audience, including those who do not have a background in clinical pharmacoepidemiology or health economics research, as well as for those with more advanced experience. It aims to assist them to understand how and when those methods are applied; their uses, limitations, and applications to healthcare provision. This booklet provides a rapid guide to the most important concepts and terminology, as well as providing a guide to evaluating the evidence provided by these research methods. We trust you find it both informative and useful. Genentech CER Task Force

Introduction to Comparative Effectiveness Research

Contents 1.

Comparative effectiveness

1.1

Introduction

1.2

What is comparative effectiveness?

1.3

Stakeholder perspectives for using CER

1.4

Why do we need to evaluate comparative effectiveness?

1.5

How do we decide when to perform a comparative effectiveness study?

1.6

Development of CER in the USA

1.7

Comparative effectiveness outside the USA

Study methodology

2.1

What is the basis of CER?

2.2

Study designs

2.2.1

Case report

2.2.2

Case series

2.2.3

Cross-sectional studies

2.2.4

Case-control studies

2.2.5

Cohort studies

2.2.6

Randomized controlled trials

Application of study methods in comparative effectiveness

3.1

Systematic reviews of existing research

3.2

Meta-analyses

3.3

Randomized controlled trials

3.4

Indirect evidence and mixed treatment comparison

3.5

Pharmacoeconomic studies

3.6

Additional data sources

3.6.1

Administrative claims database analyses

3.6.2

Observational studies

3.6.3

Registries

3.6.4

Other data sources

Introduction to Comparative Effectiveness Research

3.7

Summary table of comparative effectiveness study types

3.8

Possible pitfalls in study methods

3.8.1

Biases

3.8.2

Confounding by the indication

3.9

Innovative methods to support CER

3.10

Other issues

3.11

What do we measure in comparative effectiveness studies?

3.11.1

Outcomes used in comparative effectiveness studies

3.11.2

Statistical measures used in comparative effectiveness studies

3.11.3

Statistical and clinical significance

3.12

How are comparative effectiveness results used?

Evidence-based medicine

4.1

What is evidence-based medicine?

4.2

Collecting the evidence

4.3

Making the evidence available for consideration by decision-makers

4.4

Using the evidence

4.4.1

How do healthcare practitioners make sense of all the evidence?

4.4.2

How do payers make sense of all the evidence?

4.4.3

How do policy-makers and government use the evidence?

How to evaluate a comparative effectiveness study or article

5. 5.1

Data collection

5.2

Data analysis

5.3

Interpretation

Glossary of terms and acronyms

References

Introduction to Comparative Effectiveness Research

Comparative Eff ectiveness Effectiveness

1. 1.

Comparative Comparativeeffectiveness effectiveness

1.1 1.1 Introduction Introduction Rising healthcare provide a challenge challenge to to federal federal Rising healthcare costs provide and to private private payers, payers, andstate state governments governments as as well as to with withthe thecontinuing continuingdevelopment developmentand andwidening wideninguse of more expensive medical technologies. Hard use of more expensive medical technologies.evidence Hard isevidence often unavailable about which treatments work best is often unavailable about which treatments for which whether the added the benefits of work bestpatients, for whichand patients, and whether added more but expensive, benefieffective, ts of more effoften ective,more but often more treatments expensive, warrant theirwarrant use in routine The primary treatments their usepractice. in routine practice. purpose of comparative effectiveness research (CER) The primary purpose of comparative effectiveness isresearch to provide to make informed healthcareis toevidence provide evidence to make informed related decisions. It is a complex field that healthcare-related decisions. It is a complexincludes field consideration of the risks, benefits, andbenefi costs ts of and that includes consideration of the risks, one treatment compared with onewith or more costs of one treatment compared one other or more treatments for the same indication, and the research other treatments for the same indication, and the isresearch usually isconducted in the setting ‘real-world’ usually conducted in theofsetting of ‘realhealthcare interactions. The data from CER world’ healthcare interactions. Theobtained data obtained from are vital in theeffpractice of evidence-based comparative ectiveness research are vitalmedicine, in the which integrates individual clinical expertise with the practice of evidence-based medicine, which integrates best available clinical and scientific evidence. individual clinical expertise with the best available clinical and scientific evidence. 1.2 What is comparative effectiveness? The (IOM) established definition 1.2 Institute Whatof is Medicine’s comparative effectiveness? of CER is commonly referenced by stakeholders, and Within the healthcare sector, comparative effectiveness has beenisadopted by evaluation the Genentech TaskofForce: analysis a rigorous of theCER impact “CER is the generation and synthesis of evidence different treatment options for a given medical that compares benefits and harms alternative condition for a the particular population ofof patients, in methods to prevent, diagnose, treat and monitor a order to make informed healthcare-related decisions. clinical condition, or to improve the delivery of care. This comparison of treatments is not limited to The purpose of is to assist consumers,approaches clinicians, medications (forCER example, two therapeutic purchasers, and policy makers to make informed to the treatment of emphysema are surgical reduction decisions that will improve healthcare at both the of lung volume and standard medical therapy), and the 1 individual and population levels.” comparison may potentially consider the relative costs.

©©Rx RxValues Values Group Group Ltd. Ltd. All All rights rights reserved reserved

Comparative effectiveness is a set of analytic tools that allows for the comparison of one treatment – drug, device or procedure – with another treatment on the basis of risks, benefits, and relative cost 1

Treatment X

Treatment Y

Risks Benefits Costs

Introduction to Comparative Effectiveness and Evidence-Based Medicine 1 Introduction to Comparative Effectiveness Research 1

Within the healthcare sector, comparative effectiveness analysis is a rigorous evaluation of the impact of different treatment options for a given medical condition for a particular population of patients, in order to make informed healthcarerelated decisions. This comparison of treatments is not limited to medications (for example, two therapeutic approaches to the treatment of emphysema are surgical reduction of lung volume and standard medical therapy), and the comparison may be extended to consider relative costs. A number of definitions for CER have been proposed, all of which share the following attributes: • One treatment is compared with one or more other treatments • Comparison of treatments is not limited to medications • Both risks and benefits are included in the assessment. 1.3 Stakeholder perspectives for using CER The following table summarizes the various uses of CER based on differing stakeholder needs: • Improving patient access to the best therapeutic options • Facilitating evidence-based treatment decisions • Using rigorous science to produce valid evidence to fill gaps • Evaluating all interventions across the healthcare continuum • Being balanced in economic considerations and clinical evidence • Being conducted through an open and transparent process • Involving multiple stakeholders and instituting a comprehensive process, especially when conducted by a federally-established CER body. The different stakeholders have different requirements from CER (Table 1).

Introduction to Comparative Effectiveness Research

Comparative Effectiveness

Table 1. Stakeholders require different things from CER Stakeholder

Uses of CER

Patients and consumers

—— Have more informed conversations with their healthcare providers —— Improve their health outcomes

Physicians and healthcare providers

—— Improve their decision-making capacity

Public and private payers

—— Inform purchasing decisions

Life sciences industry

—— Support value proposition for products

Comparative effectiveness can contribute information about drug safety and effectiveness that is not available from pre-marketing studies

—— Help educate provider networks

—— Contribute to evidence base Employers

—— Manage cost growth —— Increase productivity and improve employee health outcomes

Regulators

—— Improve and protect public health

Introduction to Comparative Effectiveness Research 3

Efficacy: “is the extent to which medical interventions achieve health improvements under ideal circumstances” i.e. does it work? 2

1.4 Why do we need to evaluate comparative effectiveness? In order to make optimal clinical decisions about the use of a drug, a prescriber needs to know whether, and to what extent, the drug is actually able to produce the intended beneficial effect in a real-world setting, as well as in a randomized controlled trial (RCT) (Figure 1). The pre-marketing randomized controlled studies of efficacy typically investigate whether a drug has the ability to bring about a single intended effect in a small, well-defined, homogeneous study population. Figure 1. Comparative efficacy versus comparative effectiveness

Efficacy

Comparative efficacy

Effectiveness

Comparative effectiveness

is the extent to which an intervention does more good than harm ... compared with one or more alternative interventions

Effectiveness: “is the extent to which medical interventions achieve health improvements in real practice setting” i.e. does it work in the real-world? 2

... compared with one or more alternative interventions

... under ideal circumstances

... when provided under usual circumstances of healthcare practice

Randomized controlled trial

Real-world setting

Comparative effectiveness can contribute information about drug safety and effectiveness that is not available from pre-marketing studies. Patients taking part in a Phase III RCT may not necessarily reflect the broader population that will be exposed to the drug in routine medical practice. For example, pre-marketing trials might exclude elderly patients or women of child-bearing potential in the study exclusion criteria. Patients are usually excluded if they have serious concomitant illness, and generally

Introduction to Comparative Effectiveness Research

Comparative Effectiveness

are allowed to take only certain, specified concomitant medications outlined in the study protocol. In addition, patients taking part in trials tend to have unrealistically high levels of compliance with medication. Therefore, it is not known whether the drug will have the same effect when used in the much wider context of general clinical practice. Furthermore, efficacy is frequently tested against a placebo as opposed to alternative drugs or therapies available for the same indication. Clinical trials of efficacy may be sufficient to allow the regulatory authorities to approve the drug for use, but important clinical questions remain to be answered by CER:

The relative scarcity of rigorous data concerning comparative effectiveness means that decisions regarding which treatments to use often depend on anecdotal evidence, evidence from RCTs, conjecture, and/ or the experience and judgment of individual physicians

• Does the drug intervention achieve the same beneficial effect when used in routine clinical practice? • Does the drug intervention have other beneficial effects, including long-term effects for the same indication? • Can the drug intervention achieve these effects better than alternative therapies available for the same indication? • What effect do the many different situations seen in medical practice have on the effect of the drug intervention? For example: -- variations in drug regimen – dosage, duration, compliance, and switching -- characteristics of the disease – severity, subcategories, and natural history -- characteristics of the patient – demographics, diet, genetics, other concomitant illnesses, and medications • How does the drug’s safety profile vary in the realworld setting versus RCT settings?

Considerable gaps might exist in the information about drug effectiveness/ safety at the time a drug is approved for marketing

Introduction to Comparative Effectiveness Research 5

There are instances where studies of comparative effectiveness have disproved widely held assumptions about the merits of different treatments. The examples below illustrate how comparative effectiveness studies can help to identify a particular group of patients that may or may not benefit from a particular drug. They also highlight the potential risk for sponsor companies that carry out comparative, post-marketing studies. Examples of findings of comparative effectiveness studies A study into comparative effectiveness of management strategies in gastroesophageal reflux disease found there was no difference between omeprazole, lansoprazole, pantoprazole, and rabeprazole for relief of symptoms at 8 weeks3. Diuretics were found to be more effective in patients aged 55 years or older than commonly used, more expensive, newer drugs such as angiotensinconverting enzyme inhibitors and calcium-channel blockers in preventing heart attacks4. A trial of two statin drugs was sponsored by the maker of one of the drugs and the data showed that the competitor product was more effective at lowering cholesterol and reducing mortality risk5. Patients with stable coronary artery disease treated by angioplasty with metal stent plus a drug regimen had better blood flow and fewer symptoms of heart problems initially, but the differences between the groups declined over time and there was no difference in 5-year survival6 1.5 How do we decide when to perform a comparative effectiveness study? The decision whether to conduct a comparative effectiveness study is based on an evaluation of the costs and risks, weighed against the benefits of conducting the study. In terms of the costs, the majority of these will relate to conducting the study itself, such as the financial costs, and the time and effort involved. The risks include the potential to detect an adverse outcome that is falsely attributed to the drug or the risk of providing false reassurances about the safety of the drug. Both of these risks can be minimized by appropriate and rigorous study conduct, including design, analysis, and data interpretation. Appropriately conducting CER will likely generate evidence to demonstrate that one drug or intervention is more effective or safer than another. 6

Introduction to Comparative Effectiveness Research

Comparative Effectiveness

1.6 Development of CER in the USA In 1978, the National Center for Health Care Technology was established in the USA. Its mandate was to conduct and promote research on healthcare technology. Before its closure in 1981 it had undertaken major evaluations of coronary artery bypass graft surgery, and made about 75 recommendations to the Medicare program about coverage. In the same period, the Office of Technology Assessment was established as an advisory agency to Congress on healthcare issues. It studied a variety of healthcare topics, including the costs and benefits of screening tests for several diseases, before its demise in 1995.

AHRQ Mission: To improve the quality, safety, efficiency, and effectiveness of health care for all Americans 7

In 1999, the Agency for Healthcare Research and Quality (AHRQ) was established and has become the most prominent agency supporting various types of research on the comparative effectiveness of medical treatments. AHRQ is the health services research arm of the US Department of Health and Human Services (DHHS), complementing the biomedical research mission of its sister agency, the National Institutes of Health (NIH). It provides a major source of funding and technical assistance for health services research and research training at leading US universities and other institutions. AHRQ works with the public and private sectors to build the knowledge base for what works (and does not work) in health and healthcare, and to translate this knowledge into everyday practice and policy-making. One initiative has been the creation of treatment guidelines that summarize the available medical evidence of appropriate treatments for various conditions. The National Guidelines Clearinghouse8 has been set up by the AHRQ to provide a means for obtaining objective and detailed information about clinical practice guidelines. AHRQ has also endorsed about a dozen evidence-based practice © Rx Values Group Ltd. All rights reserved

Introduction to Comparative Effectiveness Research 7

Comparative Effectiveness

as r

le n he dical tive er

n this n, ble

mics

NICE Mission: To analyse the clinical and costeffectiveness of new and existing medicines, procedures and other technologies and to provide guidance on appropriate treatments for specific diseases

centers (EPCs) around the country. Most recently, following the Medicare Modernization Act (2003), AHRQ has conducted and supported research on outcomes, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and healthcare services in 10 priority areas through its Effective Health Care Program. The Drug Effectiveness Review Project (DERP) is a collaboration of organizations that are joined together to obtain the best available evidence on effectiveness and safety comparisons between drugs in the same class and to apply the information to public policy and decision-making in local settings. The DERP is a series of comprehensive, updated, and unbiased systematic reviews conducted by EPCs with oversight and coordination from the Oregon EPC. Participants in DERP use the project reports to inform local decision-makers. Other agencies involved in research related to comparative effectiveness include the Department of Veterans Affairs, the NIH, and the Centers for Medicare & Medicaid Services (CMS). Medical insurers, and the pharmaceutical and medical device industries are also involved in undertaking CER. In the USA, the American Recovery and Reinvestment Act (ARRA) – signed into law on February 17, 2009 – mandated $1.1B federal funds for CER9. In March 2010, the Patient Protection and Affordable Care Act (PPACA) was passed and included the establishment of a new non-profit patient-centered outcomes research institute (PCORI)10.

l, al

8 andIntroduction to Comparative Research Effectiveness Evidence-Based Medicine Effectiveness 9

Comparative Effectiveness

1.7 Comparative effectiveness outside the USA Perhaps the best-known national agency involved in clinical effectiveness research is the National Institute for Health and Clinical Excellence (NICE) in the UK, which was established in 1999 as part of the UK National Health Service (NHS). Its missions are to analyze both the clinical and cost-effectiveness of new and existing medicines, procedures and other technologies, and to provide guidance on appropriate treatments for specific diseases. If NICE approves a drug, device, or procedure, then it must be funded by the NHS. Data on clinical effectiveness are obtained from systematic reviews of existing research and these findings are combined with models of cost-effectiveness. Other countries, such as Australia, Canada, France, and Germany have similar structures in place although funding arrangements and organization vary. For example, in 2004 an independent institute for quality and efficiency in healthcare, IQWiG, was set up in Germany. Since 2008 IQWiG has been tasked with evaluating the cost-benefit of pharmaceuticals in Germany. In contrast, in France le Haut Autorité de Santé (HAS) hosts the Commission d’Evaluation des Médicaments. The body assesses the Medical Benefit (SMR) and the Improvement of Medical Benefit (ASMR), the second of which is the comparative effectiveness assessment of drugs. Increasingly, other countries are setting up similar national agencies.

NICE Mission: To analyze the clinical and cost-effectiveness of new and existing medicines, procedures and other technologies and to provide guidance on appropriate treatments for specific diseases

Asia and Latin America are not quite as advanced in this area as are most of the European countries. In Japan, for example, local health economics data are available but limited, and clinical practice guidelines are also in place, but neither of these are required by the government11. In Latin America local health economics data are also limited; however, health economics data are being used increasingly in Argentina, Brazil, and Mexico by decision-makers, and © Rx Values Group Ltd. All rights reserved

Introduction to Comparative Effectiveness Research 9

national clinical guidelines are enforced for some priority medical conditions in Argentina, Brazil, Chile, Columbia, Mexico, and Venezuela12. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) website provides a very useful resource for those wishing to understand more about individual country healthcare systems and reimbursement processes at http://www.ispor.org/HTARoadMaps/. Figure 2. Some of the European national agencies evaluating comparative effectiveness data and/or health technology assessments (HTAs) Scotland: SMC (Scottish Medicines Consortium)

Denmark: DACEHTA (Danish Centre for Evaluation and Health Technology Assessment)

Sweden: TLV (Dental and Pharmaceutical Benefits Board)

France: HAS (National Authority for Health)

Germany: IQWiG (Institute for Quality and Efficiency in Health Care)

England: NICE (National Institute for Health and Clinical Excellence)

Ireland: HIQA (Health Information and Quality Authority)

Wales : AWMSG (All Wales Medical Strategy Group)

Introduction to Comparative Effectiveness Research

Study Study Methodology Methodology

2. 2.

Studymethodology methodology Study

2.1 What What is is the 2.1 the basis basisof ofcomparative CER? eff ectiveness research? Before we look at the study methods available for CER, Before at the study methods available forof we needwe tolook understand the basic study methods comparative effectiveness research, we need to epidemiology. understand the basic study methods of epidemiology. The basis of epidemiology lies in determining whether The basis of epidemiology lies in determining whether a relationship (association) exists between a factor a relationship (association) exists between a factor (exposure) (for example, an environmental exposure (exposure) (for example, an environmental exposure or a clinical intervention such as drug therapy) or or a clinical intervention such as drug therapy) or a characteristic (for example, an increased serum a characteristic (for example, an increased serum cholesterol level), and the development of the disease cholesterol level), and the development of the disease in question or change in clinical condition (outcome) in question or change in clinical condition (outcome) in a population. The results obtained for the sample in a sample group of subjects. The results obtained are used to draw a conclusion about a population in for the sample are used to draw a conclusion about general, using appropriate clinical, epidemiological, a population in general, using appropriate clinical, and statistical methods. If an association is found we epidemiological, and statistical methods. If an then have to determine what type of association it is. association is found we then have to determine what Three types of association type of association it is. between the factor and the outcome may be observed in an epidemiological study Three types of association between the factor and the (Figure 3): outcome may be observed in an epidemiological study •(Figure The association is spurious (artifactual) because 3): the study has been influenced by errors, which may • be Therandom association is spurious (artefactual) because the or systematic error (also known as bias) study has been influenced by errors, which may be (Figure 3[A]) random or systematic (also known as bias) • The association seems real, but is only observed (Figure 3[A]) because of the presence of another variable • (a The associationvariable) seems real, is only observed confounding thatbut is independently becausetoofboth the the presence of another (a related exposure and thevariable outcome. confounding variable) that is independently related The association between the exposure and the to both the exposureconfounded and the outcome. outcome is therefore (FigureThe 3[B]) association between the exposure and the outcome • The observed association is real and is a direct is therefore confounded (Figure 3[B]) causal association between the intervention and • outcome The observed association (Figure 3[C]). is real and is a direct causal association between the intervention and outcome (Figure 3[C]).

The basis of The basis of epidemiology lies in epidemiology lies in determining whether determining whether a a relationship (association)(association) exists relationship between a factor exists between a factor or or characteristic characteristic (exposure) (exposure) and the and the development development or change or change in clinical in clinical condition (outcome). condition (outcome).

Introduction to Comparative Effectiveness and Evidence-Based Medicine 11 Introduction to Comparative Effectiveness Research 11

Figure 3. Types of associations (B) Confounded

Factor

Random or systematic errors

Confounding variable

Outcome

Observed association

(A) Artifactual

Outcome

2.2 Study designs Below is a description and summary of multiple study designs (Table 2 – page 19). A number of the designs can be employed in conducting CER. 2.2.1 Case report A case series is observational and can be conducted prospectively or retrospectively13. A case report is a simple description of the events observed in a single patient. For example, a case report may be published about a young woman who was taking oral contraceptives and suffered a pulmonary embolism. Rarely can a case report make a statement about whether the drug caused the occurrence of the observed event. However, case studies are an inexpensive and easy method of generating hypotheses that can be tested by more rigorous study designs. 2.2.2 Case series A case series is observational and can be conducted prospectively or retrospectively13. A case series is a collection of patients who have been exposed to the same drug and whose clinical outcomes are evaluated and described. Alternatively, a case series may be a collection of patients with a single outcome in whom previous exposure is evaluated and described. This type of study has been used in post-marketing surveillance to quantify the incidence of an adverse reaction or to provide evidence that a particular event of concern does not occur

Introduction to Comparative Effectiveness Research

Study Methodology

in a population larger than that studied in the premarket testing. Case series do not have a control group so cannot be used for hypothesis testing, and they are not particularly useful in determining the causal relationship between a drug and an adverse event. 2.2.3 Cross-sectional studies Cross-sectional studies are a snapshot of the population at a certain point in time; both the exposure and the disease outcome are measured simultaneously (Figure 4). The question we are trying to answer is ‘What is the prevalence of the disease and how does it relate to the presence or absence of a particular variable (or exposure)?’. Prevalence describes the proportion of people who have a disease or condition at one point in time (i.e. the number of people who have the disease or condition divided by the number of people at risk)13. This type of study identifies prevalent cases of the disease, that is, we know they existed at the time of the study but we do not know their disease duration. Furthermore, we cannot determine the temporal relationship between the exposure and the disease.

Prevalence: the proportion of individuals in a population who have the disease at a specific time point. Incidence: the number of new cases of the disease in a population in a given time period.

Introduction to Comparative Effectiveness Research 13

Figure 4. Schematic representation of a cross-sectional study Exposed Not exposed

Population

Simultaneously measure at one point in time

Determinary associations Have disease Do not have disease

2.2.4 Case-control studies Case-control studies are observational, retrospective, non-experimental studies that identify patients with a particular disease of interest (known as the cases) and patients without the disease (known as the controls), and then retrospectively determine previous exposure (Figure 5). These studies benefit from having a control group, so they can be used for hypothesis testing. The question we want to answer is: ‘What are the odds that a case was exposed?’ and, to this end, we measure the odds ratio (OR) (Section 3.11.2). Case-control studies are useful in that they can rapidly generate hypotheses for potential disease risk factors (especially for rare outcomes) using information from relatively few subjects, but they also have major limitations including potential sampling bias and bias caused by retrospective measurement of the predictor variables13.

Introduction to Comparative Effectiveness Research

Study Methodology

Figure 5. Schematic representation of a case-control study Exposed

Disease present (case)

Not exposed Population Exposed

Disease present (control)

Not exposed Retrospective 2.2.5 Cohort studies Cohort studies are observational studies that identify a population free of the disease or condition of interest, following them over time. A cohort study can be prospective or retrospective. In a prospective cohort study, patients in the population under study with and without exposure are identified and prospective data are collected on development of the disease (Figure 6). In a retrospective cohort study, the patient cohort and study measures were collected in the past, while the outcomes are measured in the present (Figure 7). Risk factors are measured before the development of the disease or outcome, to help determine the temporal relationship between the predictors and outcomes (in order to help understand which came first). The question we are trying to answer is: ‘What is the ratio of the risk of disease in exposed subjects to the risk of disease in non-exposed subjects?’. This ratio is called the relative risk (RR) (Section 3.11.2). In order to calculate the RR we need to know the proportion of patients with the disease in the exposed and non-exposed groups within the population.

Introduction to Comparative Effectiveness Research 15

Figure 6. Schematic representation of a prospective cohort study Develop disease Exposed Do not develop disease Population Develop disease Not exposed Do not develop disease Prospective Figure 7. Schematic representation of a retrospective cohort study Develop disease Population

Exposed Do not develop disease

Develop disease Population

Not exposed Do not develop disease Retrospective

Introduction to Comparative Effectiveness Research

Study Methodology

2.2.6 Randomized controlled trials Randomized controlled trials (RCTs) are interventional, experimental studies in which the treatment the patient receives is determined by chance, by a process of randomization. Figure 8 illustrates the design of a simple RCT where a new treatment (treatment A) is being compared with a current treatment (treatment B). Figure 8. Schematic representation of a simple RCT

Population

Randomization

Treatment A

Improved

Disease progression

Treatment B

No disease progression

Not improved

An alternative design is a crossover study, shown in Figure 9, where subjects receive therapy A or therapy B and after being observed for a certain period of time on one therapy they are switched to the other therapy. In effect, each patient acts as his or her own control, thereby avoiding the variation in patient characteristics that may affect comparison of the two treatments. One disadvantage of a crossover study design is the potential for the first treatment effect to carry over into the second. Therefore, an adequate wash-out period (stopping therapy between the two treatment phases) is necessary.

Introduction to Comparative Effectiveness Research 17

Figure 9. Schematic representation of a crossover study

Treatment A

Treatment B

Period of time

Group A

Group B

The use of randomization in an RCT attempts to ensure that the study groups are comparable with respect to any baseline characteristics, and therefore any results that are associated with the difference in treatments between groups, are more likely to be caused by the treatment difference. RCTs can be expensive to perform and may be fraught with logistical problems. Additionally, the results are not representative of ‘real-world’ populations because of the stringent inclusion and exclusion criteria, so RCTs are often referred to as ‘efficacy’ rather than ‘effectiveness’ studies. RCTs are considered efficacy trials, because the studies are conducted in an ideal, controlled setting. However, pragmatic RCTs are considered effectiveness trials, because they include populations more representative of the ‘real-world’ and treat patients within routine clinical practice, for example, Phase IV trials and other post-marketing surveillance.

Introduction to Comparative Effectiveness Research

Study Methodology

Table 2. Summary of study designs Type of study

Description

Case report

• Report of events observed in single patients. • Inexpensive and simple. • Suitable for hypothesis generation but not testing.

Case series • Evaluation of outcome in a series of patients with a common exposure. • Easy quantitation of incidence. • No control group so cannot be used for hypothesis testing. Cross-sectional study

• Takes a snapshot of the population at a single point in time. • Cannot determine temporal associations.

Case-control • Compares cases with disease and controls without study disease, looking retrospectively for different exposures. • Relatively rapid and inexpensive to perform. • Can study multiple outcomes and uncommon diseases. • Possible selection bias and biased exposure data. Cohort study • Identifies subsets in a defined population and follows them over time looking for differences in outcome. • Can study multiple outcomes and uncommon exposures. • Most robust epidemiological design. • Prospective study can be expensive and may require many years of follow-up. RCT • Patients are randomized to receive experimental or control therapy. • Most convincing design in assessing a cause-effect relationship. • Controls for unknown or unmeasurable confounding variables by the process of randomization. • Expensive, could be difficult to perform and may require many years of follow-up. • Not always reflective of real-world clinical practice.

Introduction to Comparative Effectiveness Research 19

fo noitacilppA sdohteM ydutS

3. Application of study methods in ni sdohtem yduts fo noitacilppA comparative effectiveness ssenevitceffe evitarapmoc

Aevnumber itarapmofocapplications ni deyolpmare e eremployed a snoitacilin ppCER. a fo rThese ebmuare nA described below and summarized in Table 3 on page 28. dna woleb debircsed era esehT .hcraeser ssenevitceffe 3.1

.62 egap no 3 elbaT ni desirammus Systematic reviews of existing research

secompilation sylana-ateM 1.3 A systematic review is a methodical of o noitcellocfora saaparticular denfied nresearch eeb sah squestion, isylana-ateM studiesfconducted ehat t ggenerating nitargetni fan o eoverall soprupconclusion eht rof sdobased htem on citythe lana aimed sum of .their seiduresults. ts tnereItffis id important dna tnedntoepassess edni mcarefully orf sgnidnfi the strengths eht fo stand luserweaknesses eht sloop siof sylthe anaavailable -atem a ,eevidence cnesse nI and to.reconcile conflicting findings. This is probably tluser nommoc elgnis a evig ot seiduts laudithe vidni implement, lleasiest arevo nand a edmost ivorpinexpensive nac sisylanaapproach -atem ,etatoirp orppa erehW ot deilppa era sdohtem laand citsitsuch atS .sstudies gnidnfiform eht fthe o yrbuilding ammus ablocks sa tceof ffeevidencefo erusaem tsom sah sisylana-ateMbased .weivemedical r erutarepractice. til evitatiThe lauqComparative citametsys dEffectiveness na hguoroht a fo puorg a gnisirammuReviews s rof loopublished t a sa ,ecnby atpAHRQ ecca dare enibased ag dnaon ,desystematic su neeb netfo reviews of the published literature; for example na hcus ,yllacipyT .noitacidni ralucitrap a rof ypareht ralucitrap a htiw gnithe laed sTCR drugs elpmaxe rof ,tnemtaert fo yreview cacffie of ehsecond t fo erusgeneration aem llarevantidepressant o na tneserp dlu ow siin sythe lana 14 treatment of adult depression . seiduts latnemirepxe-non fo sesylana-atem ,yltnecer eroM .oitar sddo llarevo na rof snosaer fo noitarolpxe eht no erom sucof ot dnet esehT .nekatrednu neeb evah .saib fo stceffe eht gnidulcni The ,seidCochrane uts laudivCollaboration idni fo stluserisean ht nexample eewteb of tnaemeergasid worldwide effort to conduct systematic clinical reviews; it is continuously hcraeser gupdating nitsixe fits o sreports weiveron citan amelectronic etsyS 2.3 database called the Cochrane Library, which is available dna ,tnemelpmi ot hcaorppa evisnepxeni tsom dn15a tseisae eht ylbaborp si sihT ehT .ecitcarp lacidem don esaCDs b-ecand nedon ivethe fo sInternet kcolb gn.idliub eht mrof seiduts hcus

no desab era ,QRHA yb dehsilbup )REC( sweiveR ssenevitceffE evitarapmoC 3.2 Meta-analyses -dnoces fo weiver eht elpmaxe rof ;erutaretil dehsilbup eht fo sweiver citametsys Meta-analysis has been defined as a collection of .21noisserped tluda fo tnemtaert eht ni sgurd tnasserpeditna noitareneg analytic methods for the purpose of integrating the tcudnoc ot troffe edfindings iwdlrowfrom a fo eindependent lpmaxe na si and noitadifferent roballoCstudies enarhcoinCaehT cinortcele na no stroper stsystematic i gnitadpu review. ylsuounIn itnessence, oc si ti ;swaemeta-analysis iver lacinilc citapools metsys tenretnI eht no dna sDC no the elbaresults liava si of hcithe hw individual ,yrarbiL enstudies arhcoC etohtgive dellaacsingle esabatad common result. Where appropriate,.)gmeta-analysis ro.enarhcoc.wcan ww( provide an overall measure of effect as a summary yb rehtona eno tsniaga derapmoc eb nac stnemtaert ,sweiver citametsys nI of the findings. Statistical methods are applied to a ssessa ot tnatropmi si tI .obecalp tsniaga stnemtaert elgnis fo stroper gninibmoc thorough and systematic qualitative literature review. elicnocer ot dna ecnedive elbaliava eht fo sessenkaew dna shtgnerts eht ylluferac Meta-analysis has most often been used, and gained .sgnidnfi gnitciflnoc

20 91

Introduction enicideM destoaBComparative -ecnedivE dnEffectiveness a ssenevitceffEResearch evitarapmoC ot noitcudortnI

Application of Study Methods

acceptance, as a tool for summarizing a group of RCTs dealing with a particular therapy for a particular indication. Typically, such an analysis would present an overall measure of the efficacy of treatment, for example an overall odds ratio or a relative risk. More recently, meta-analyses of non-experimental studies have been undertaken. These tend to focus more on the exploration of reasons for disagreement between the results of individual studies, including the effects of bias. 3.3 Randomized controlled trials In order to detect what may be a relatively small difference in efficacy between the drug under study and an alternative therapy, large RCTs are required. This makes them costly to perform and they can take a long time to complete. RCTs have the advantage of controlling for confounding by the process of randomization and they provide an unbiased estimate of beneficial effects and adverse outcomes. RCTs can be supplemented by computer modelling techniques to simulate the effects of treatments in different populations. RCTs may compare one or more active treatments to placebo or involve the head-tohead comparison of two active treatments. 3.4 Indirect evidence and mixed treatment comparison The ideal method of comparing two treatments is a direct comparison of Treatment A versus Treatment B in a well executed RCT. However, in the absence of a direct comparison, the results of studies comparing each of the treatments against placebo may be pooled, as in a meta-analysis, to produce an indirect comparison of the two treatments. For example, the results of groups of studies comparing Treatment A versus placebo and Treatment B versus placebo can be combined to provide an indirect comparison of Treatment A versus Treatment B. However, in this process, the benefits of randomization within the individual studies may be lost, resulting in an unequal distribution of baseline characteristics in the treatment groups and possible bias in the results. An alternative model is to examine the magnitude of the treatment effects, such as odds ratio (OR) or relative risk (RR), in different studies and to compare the individual treatment effects. This method preserves the randomization of the treatment groups in the constituent studies and may allow an unbiased, indirect estimate of the overall treatment effect16. However, heterogeneity between the studies may impact the pooled magnitude of treatment effects. In a systematic review of the treatment of chronic insomnia commissioned by AHRQ, indirect comparisons were made using this methodology and the results accurately reflected the current evidence17.

Introduction to Comparative Effectiveness Research 21

Application of Study Methods

e rves

na mnia

es.

of dy

C d

Indirect treatment comparisons (ITC) are made between competing interventions that have not undergone a direct head-to-head comparison in an RCT. In its simplest form, as shown in Figure 10, ITC uses the common comparator intervention C in two studies: intervention A versus intervention C and intervention B versus intervention C, to yield an indirect comparison of intervention A versus intervention B. Mixed treatment comparisons (MTC) can be conducted when both indirect and direct sources of evidence are used. They are an extension of the traditional meta-analysis (where all included studies compare the intervention with the same comparator) in that they include multiple different pair-wise comparisons across a range of interventions. For example, in the comparison of intervention A versus intervention B, three types of study may be used. Studies of A versus B provide a direct comparison and studies of A versus C and B versus C provide indirect comparisons (Figure 11). MTC allows the relative efficacy or safety of an intervention compared with alternative interventions in the absence of direct head-to-head comparisons. MTC can also be used in more complex situations, such as network meta-analysis, where three or more treatments are being compared18,19.

ice

s in y

o at ber d to nal

al ce Effectiveness Evidence-Based Medicine Effectiveness 23 22 andIntroduction to Comparative Research

Application of Study Methods

Figure 10. Schematic representation of an indirect treatment comparison (ITC)

Indirect comparison Intervention A

Intervention B

Intervention C

Figure 11. Schematic representation of a multiple treatment comparison (MTC)

Intervention A

Direct and indirect comparisons are used

Intervention B

Intervention C

Introduction to Comparative Effectiveness Research 23

mi uts oba tuo irav itap

3.5

7.3 ahP oc noc fo

Health economics research has two major objectives:

eH aht aeh bus no

Part of the role of a health economist is to identify, measure, value, and compare the costs and consequences of a particular medical intervention. Pharmacoeconomics ‘identifies, measures, and compares the costs (i.e. resources consumed) and consequences (e.g. economic, clinical, and humanistic) of pharmaceutical products and services’20.

eH • •

Pharmacoeconomic studies

Health economics is a branch of economics that is concerned with the efficient allocation of healthcare resources, and pharmacoeconomics is a subdiscipline of health economics that focuses solely on pharmaceuticals. • To improve public health through rational decision-making • To determine the relative values of alternative therapies.

The most important and widely used types of pharmacoeconomics analysis are the CEA and cost-utility analysis (CUA).

raP em noc ehT ahp ana •

ssenevitc24 effE evIntroduction itarapmoC ottono itcudortnI Effectiveness 42 Comparative Research

•

Intermediate endpoints – these are often reported in clinical trials and they are usually important measures of treatment efficacy. However, they tend Application of to focus on clinical measures (e.g. a change in blood Study Methods pressure or lab tests) and do not reflect patientcentered outcomes; and a lack of a common denominator between studies can make comparisons difficult. • CEA is a technique for comparing the relative value twooutcomes or more treatment strategies, new • of Final – these can be usefulwhere if theyarefl ect strategy compared with a patient current such one in issues ofisimportance to the asthe lifecalculation of the incremental cost-effectiveness years gained, symptom-free days, reduced disease ratio. In othermortality, words, thebleeding, difference in costs is progression, myocardial divided by the difference infarction, stroke, etc. in effects between the two treatment strategies under evaluation, and • Patient-reported outcomes (PROs) – these can be a ratio is computed revealing the cost per unit particularly useful as they collect data directly from benefit for using one treatment strategy compared the patient, so they can focus directly on the with the alternative. Healthcare decision makers treatment effects known only to the patient, or the may compare incremental cost-effectiveness ratios (ICERs) between treatments in order to set priorities for funding decisions. In general, Introduction to Comparative © Rx Values Group Ltd. All rightsare reserved if resources in short supply, the lower the Effectiveness and Evidence-Based Medicine 31 costs per unit of effectiveness or benefit, the more the treatment is preferred. Benefits are usually measured in natural units (i.e. cost per unit of effect, e.g. mmHG drop in blood pressure; percentage reduction in low density lipoprotein (LDL); reduction in fracture rates etc.). • CUA is a form of CEA that takes into account the effect of the cost and the benefit of a treatment strategy on both the quantity and quality of life. Utility is usually measured in quality-adjusted life-years (QALYs). The basic idea is that an extra year of poor health does not have the same value to a patient as a year of good health. A QALY is calculated by assigning a value between 0 (poorest health state) to 1 (best health state) for each year the patient lives with an illness. QALYs are known as “preference units” because they take into consideration a patient’s preference. By utilizing a single outcome measure like the QALY, payers can then compare the cost-effectiveness of therapies across various disease areas to aid in budget prioritization.

Introduction to Comparative Effectiveness Research 25

3.6 Additional data sources Clinical and outcomes studies are conducted using various types of data sources. These are described below. 3.6.1 Administrative claims database analyses These studies utilize data originally collected mainly for billing purposes. Typically, these are claims databases maintained by insurers in which patients are categorized according to medical classification systems such as the International Classification of Disease-9th revision-Clinical Modification (ICD-9-CM). Examples of commercial and payer claims databases include LifeLink, Medstat MarketScan, Blue Cross Blue Shield, and Kaiser Permanente; and the government databases include Medicare, Medicaid, and Veteran Administration (VA). Additionally, administrative and claims databases can be linked to electronic medical records to provide more complete patient health records, as is done with some of the above examples. 3.6.2 Observational studies In an observational, non-experimental study, subjects and interventions in naturally occurring situations are observed, with the goal of obtaining unconstrained realworld data. These types of studies may include cross-sectional, case-control, and retrospective and prospective cohort studies. In a prospective observational study, the physician makes the treatment decision then the patient is considered for inclusion into the study, unlike the controlled clinical trial where the patient is generally included in the trial and then is randomized to treatment. Naturalistic/ observational studies may be subject to certain limitations because the population being studied may not be representative of the overall patient population, and selection of patients may be biased. In addition, analysis may be difficult because the patients may switch treatments within the study period. Methods of restricting study populations to increase homogeneity can be applied to limit the effect of confounding factors and yield results closer to those of RCTs21. 3.6.3 Registries A patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure22. A registry may be used to describe the natural history of disease, including disease manifestations, treatment patterns, and effectiveness22. It may also be used to monitor and measure safety and risk, describe and measure quality of care, and/or determine clinical effectiveness or cost effectiveness of healthcare products and services. Examples of registries are the National Registry for Myocardial Infarction (NRMI), the Epidemiologic Study of Cystic 26

Introduction to Comparative Effectiveness Research

Application of Application Methods Study Methods

Fibrosis (ESCF), andmay the be National Cooperative Growth addition, analysis difficult because the patients Study (NCGS) of optimal Nutropin AQ and Nutropin may switch treatments within the study period. dosing inStatistically pubertal growth hormone-deficient patients. powering the study to achieve meaningful Disease registries document natural history of disease results may therefore be problematic. and burden of illness, and can be used to compare Methods of restricting study populations to increase effectiveness and safety of different therapies or devices. homogeneity can be applied to limit the effect of They can be used for planning for secondary prevention confounding factors and yield results closer to those of (detection/screening) and tertiary prevention (reducing RCTs14. the negative impact of disease through rehabilitation and reduction of related complications) and may also be a3.4.3 source ofRegistries subjects for clinical research. Registries are prospective, observational cohort Prior to or early in product development, it is important important studies that provide specialised and timely to understand the natural history of a disease which information such as natural history of disease.for They amay therapy to document also is bebeing used developed; to monitor that and is, measure safety and elements such as incidence, prevalence, patient risk risk, describe and measure quality of care, and/or factors, disease progression, clinical management, determine clinical effectiveness or cost effectiveness of and case fatality. Often, this type of information comes healthcare products and services. The most common only from case series or small cohort studies in single types of registries are disease (an example is the populations. A disease registry is an effective way to prostate cancer registry CaPSURE15) and product collect such data, and can serve as a foundation for registries, registry for infliximab in subjects both clinical(e.g. andTREAT marketing activities. These data can 16 with moderately to severely active Crohn’s disease inform market projections and can provide insight into).

The role of a health economist is to identify, measure, value, and compare the costs and consequences of a particular medical intervention

protocol and thenatural design history of caseof report Disease development registries document disease forms for Phase II and Phase III programs. Diseaseand and burden of illness, and compare effectiveness registries are important sources of safety information safety of different therapies or devices. They can be (including rare events) and of information regarding used for planning for secondary prevention (detection/ adherence to therapeutic guidelines. screening) and tertiary prevention (reducing the negative impact of disease through rehabilitation and 3.6.4 Other sources reduction of data related complications) and may also be a When study findings are incomplete or inconsistent, or source of subjects for clinical research. if there is a paucity of published data, expert opinion Priorbetosought or earlytoinestimate productthe development, it is important can appropriateness of the to understand the natural history of a disease which data. For example, a Delphi Panel consists of afor number a therapy developed; thatasked is, to document of experts is in being a given field who are to review and elements such as incidence, prevalence, patient risk synthesize available quality of life, clinical, or economic factors, disease progression, clinical management, and data. Then they are interviewed or sent questionnaires case fatality. Often, this of information comes anonymously in order to type resolve discrepancies and only from case series or small cohort studies in provide a consensus agreement to help direct single populations. A disease registry is an effective way to decision-making. ©©Rx RxValues ValuesGroup GroupLtd. Ltd.All Allrights rightsreserved reserved

Introduction to Comparative Effectiveness and Evidence-Based Medicine 27 21 Introduction to Comparative Effectiveness Research

3.7

Summary table of comparative effectiveness study types

Table 3 summarizes all the study types, their main features, and their application in comparative effectiveness. Table 3. Summary of comparative effectiveness study types Type of study

Main features

Systematic reviews • Treatments can be compared against one another by combining reports of single treatments against placebo or another treatment • It is important to assess carefully the strengths and weaknesses of the available evidence and to reconcile conflicting findings Meta-analyses • In RCTs a meta-analysis can determine an overall measure of the efficacy of treatment • In non-experimental studies a meta-analysis will explore the reasons for disagreement between individual studies Randomized controlled • RCTs have the advantage of controlling for confounding by trials (RCTs) the process of randomization • They provide an unbiased estimate of beneficial effects and adverse outcomes • RCTs can be expensive and time consuming Indirect comparison • Provides an alternative means of comparison if direct comparative data are not available • They compare the magnitude of the treatment effects in each individual study to provide a comparison between treatments • MTC utilizes both direct and indirect evidence to compare multiple treatments Pharmacoeconomic • Cost-effectiveness and outcomes analysis (CEA) studies – Compares treatments using costs – Is a ratio of costs to outcomes – ICER reveals the cost per unit of benefit of using one treatment versus another – Measured in natural units (i.e. cost per unit of effect, e.g. mmHg drop in blood pressure)

Introduction to Comparative Effectiveness Research

Application of Study Methods

• Cost-utility analysis (CUA) – Compares treatments using costs and utilities – Can compare different treatments for different diseases – Measured as preferences; QALYs are the most common utility measure Additional data sources Administrative claims • Studies use data that were originally collected for billing database analyses purposes and may be linked to electronic health records • Medical classification systems e.g. ICD-9-CM are used to categorize patient data Observational studies

• May be prospective (cohort) or retrospective (case-control) • Are naturalistic, and can be used for hypothesis testing

Registries

• Registries may be prospective, observational cohort studies • These data collections are typically disease- or treatment-based • Registries can be important sources of safety information

3.8

Possible pitfalls in study methods

Although the methods and study types used in comparative analysis are epidemiologically sound and statistically robust, special attention needs to be paid to the avoidance of error or bias when designing, conducting, analysing, and interpreting the results of the studies. In this section we discuss some of the possible pitfalls that can make the interpretation of study results problematic. 3.8.1 Biases A number of different types of error can occur in comparative effectiveness studies: • Random error is an error simply due to chance • Systematic error or bias: -- selection bias occurs when subjects in the comparison group are not selected in the same manner. This is a fatal flaw in study design and cannot be corrected for in the analysis -- information bias: --

a patient may recall their drug exposure differently depending on whether they have the disease of interest or not (known as recall bias)

Introduction to Comparative Effectiveness Research 29

•

.9.3 utS tem idni itap orf t rof dni nuf kaT itap t rof tiuq ihw oH h ro a sa ikat efni

the interviewer’s approach to questioning may be different if the disease status of the patient is known (known as interviewer or observation bias).

• Misclassification bias may occur if, for example, cases are classified as controls or vice versa in a case-control study. Similarly, a subject’s exposure status (yes versus no, dose, duration) may be misclassified. Misclassification can be random (non-differential) or non-random (differential). Random misclassification occurs when error rates in categorization of exposure or outcome are similar across the groups being compared. • Surveillance bias may occur if disease recognition is better in a closely monitored population than in the general population. • Confounding bias occurs when the presence of a variable (confounder) that is related independently to both the risk factor and outcome under study may create an apparent association or mask a real one. 3.8.2 Confounding by the indication Studies of beneficial effects are subject to the unique methodological problem of confounding by the indication. In clinical practice one would expect patients receiving the drug under study to differ from untreated patients by having the indication for treatment. If the indication for treatment is also independently related to the clinical outcome, it can function as a confounding variable (Figure 12). Take, for example, Case B in Figure 12. The reasons the patients are taking the intervention (e.g. as pain relief for their arthritis, dysmenorrhea, or acute pain) are quite unrelated to the adverse event of gastrointestinal bleeding which may occur as a result of the intervention. However, in Case A, patients with

ssenevitc30 effE evIntroduction itarapmoC oto t nComparative oitcudortnI Effectiveness 82 Research

Application of Study Methods

angina, arrhythmias, or hypertension may incur a myocardial infarction as a result of these symptoms, whether or not they are taking beta-blockers. Thus it is difficult to infer a causal relationship in Case A. When the indication can be measured sufficiently well, traditional epidemiologic statistical techniques of exclusion, matching, stratification, and mathematical modeling adjusting for baseline differences can be applied to control for confounding. In RCTs, the process of randomization can control confounding. Figure 12. Confounding by the indication Case A

Case B

Intervention

beta-blockers to prevent myocardial infarction recurrence

Indication

angina, arrhythmias, hypertension

Outcome

myocardial infarction

Intervention

non-steroidal anti-inflammatory drugs

Indication

arthritis, dysmenorrhea, acute pain

Outcome

gastrointestinal bleeding

Case A has confounding Case B has no confounding

Introduction to Comparative Effectiveness Research 31

3.9

Innovative methods to support CER

There is much ongoing research into innovative methods for studying comparative effectiveness and patient safety23. New methods to avoid bias and extract more information from the results are underway. Some examples of important areas of study are: • Using new types of experimental design, including cluster randomization, delayed design, pragmatic trials, and practice-based investigations23. • Finding efficient ways to extract/evaluate/analyze information from different automated databases, such as Medicaid claims data, state hospital data, Centers for Disease Control and Prevention surveillance files, and Department of Veterans’ Affairs files • Decreasing the threats to the validity of analyses relying on observational data by using tools such as propensity analysis, inverse probability weighting, risk adjustment, and methods for synthesizing comparative effectiveness information23. 3.10 Other issues Comparative effectiveness information can be a valuable tool in making informed medical decisions and in improving healthcare delivery, but there are issues that need to be considered24,25. • Ideally, it should cover all areas of the healthcare delivery system, including prevention, diagnosis, and medical procedures as well as drugs, devices, and biologics. • All relevant aspects of the disease and its treatments should be captured, using the highest standards of evidence. • CER should not be used as a means of controlling healthcare costs, but should be considered as one factor in the overall value of a specific healthcare intervention. • CER can be prone to a range of logistical problems, including problems of data access, making the case that these studies need to be performed only after obtaining ethical approval and informed consent. 3.11 What do we measure in comparative effectiveness studies? In order to understand the results of published studies we need to know the ways that the results are expressed and the types of measurements used. Only then can we make objective comparisons between treatments. This section covers all the main types of results you may encounter in comparative effectiveness studies.

Introduction to Comparative Effectiveness Research

Application of Study Methods

3.11.1 Outcomes used in comparative effectiveness studies

•

Comparative effectiveness studies look for the benefits associated with a particular treatment, which essentially means a positive or favorable outcome. The benefits that might be associated with treatment can be categorized according to whether they measure clinical intermediate endpoints, final outcomes, or patient-reported outcomes (PROs).

C u s v

•

• Surrogate or intermediate endpoints – these are often reported in clinical trials and they are usually important measures of treatment efficacy. However, they tend to focus on clinical measures (e.g. a change in blood pressure or lab tests) and do not reflect patient-centered outcomes; and a lack of a common denominator between studies can make comparisons difficult.

C a o n a

3.10 Ther for s safe info exam

• Clinical outcomes – these can be useful if they reflect issues of importance to the patient such as life-years gained, symptom-free days, reduced disease progression, mortality, bleeding, myocardial infarction, stroke, etc. • Patient-reported outcomes (PROs) – these can be particularly useful as they collect data directly from the patient, so they can focus directly on the treatment effects known only to the patient, or the patient’s perspective about the effectiveness of a particular treatment. • Quality-adjusted life-years (QALYs) – these measure both the quality and quantity of the years of life that a patient would be expected to have. The basic idea is that an extra year of poor health does not have the same value to a patient as an extra year of good health. A QALY is calculated by assigning a value between 0 (poorest health state) to 1 (best health state) for each year the patient lives with an illness.

•

u c t

•

fi i s C s A

•

d r p r c

Introduction to Comparative Effectiveness 33ectiveness a 30 Introduction to Research Comparative Eff

3.11.2 Statistical measures inectiveness comparative patient’s perspective aboutused the eff of a effectiveness studies particular treatment.

Risk = n/N x 100

Incidence rate = Person-time of at risk population/n Incidence rate = Number of new cases/N

Prevalence = Number of existing cases/N

32 34

following sections describe the most common •TheQuality-adjusted life-years (QALYs) – these take into statistical used in comparative accountparameters both the quality and the quantity of the effectiveness years of lifestudies. that the patient would be expected to have. They address the that an additional year Absolute risk (number andfact percentage) of poor health would not be viewed by the patient The number of events occurring in a sample of to be as valuable as an additional year of good subjects is denoted by n. The risk of the event health. occurring can be reported as a percentage, that is, the number of events (n) divided by the population 3.11.2 Statistical measures used in comparative of subjects who are at risk of the event occurring (N) effectiveness studies multiplied by 100. The following sections describe the most common Incidenceparameters rate statistical used in comparative eff ectiveness The incidencestudies. rate of a disease is a measure of how frequently the disease occurs. It is the number of Risk (number percentage) new cases of aand disease that occur during a specified The number ofin events occurring a sample of subjects period of time a population atin risk for developing is denoted by n. The risk of the event occurring canbybe the disease (i.e. previously disease-free), divided reported as a percentage, that is,accumulated the number of the person-time of observation by events the 26 (n) divided by the population of subjects who are at population at risk (N) . Person-time is the amount risk of the event occurring (N) multiplied by 100. of time each person is at risk of getting the disease, usually measured in units such as person-years. Incidence Prevalencerate The incidence rate of a disease is a measure of how The prevalence of a disease of how frequently the disease occurs.isItaismeasure the number of common It isoccur the number existing new casesthe of adisease diseaseis. that during aofspecifi ed cases in a defined population at a given point in time period of time in a population at risk for developing or over a defined period of time, divided by the number the disease (i.e. previously disease-free),26divided by of subjects in theofpopulation at accumulated risk (N) . Prevalence the person-time observation by the is often reported as a percentage (by multiplying the 3 population at risk (N) . proportion by 100). (Person-time is the amount of time each person is at Relative risk risk of getting the disease, usually measured in personThe relative risk (RR) is the ratio of the risk of days). disease in exposed subjects to the risk of disease in non-exposed subjects and can be measured in a cohort study or RCT (Figure 13). If the risk in exposed

Introduction to Comparative Effectiveness and Evidence-Based Medicine Introduction to Comparative Effectiveness Research

Application of Study Methods

subjects equals the risk in non-exposed subjects then the RR will equal 1 (i.e. no association). If the risk in exposed subjects is greater than the risk in non-exposed subjects then the RR will be greater than 1. If the risk in exposed subjects is less than the risk in non-exposed subjects then the RR will be less than 1. Figure 13. Calculation of relative risk (RR)

Exposed Not exposed RR =

Develop disease

Do not develop disease

a c

b d

Risk of disease in exposed subjects Risk of disease in non-exposed subjects

a /(a+b) c /(c+d)

Odds ratio The odds ratio (OR) is the ratio of the likelihood (odds) that cases were exposed to the odds that controls were exposed (Figure 14) and it is measured in case-control studies. If the exposure is not related to the disease, the OR will equal 1. If the exposure is positively related to the disease then the OR will be greater than 1. If the exposure is negatively related to the disease then the OR will be less than 1. The OR is a close estimate of the RR obtained from the cohort study when the disease in question is relatively rare (i.e. a and c are small).

Introduction to Comparative Effectiveness Research 35

Figure 14. Calculation of odds ratio (OR)

Exposed Not exposed OR =

Develop disease (cases)

Do not develop disease (controls)

a c

b d

Odds that a case was exposed Odds that a control was exposed

a/c b/d

ad bc

Hazard ratio The hazard ratio (HR) is a type of RR estimate that indicates the odds of an event (e.g. death) happening faster with a particular treatment intervention. It describes the relative risk of the event happening based on a comparison of event rates. The HR does not indicate the relative speed of change in the patient status. For example, an HR of 2.0 means that at any certain time point, the patients still alive in the placebo group are at twice the risk of death as those in the treatment group by the next time interval.

Hazard function = Pr(die at time 1/survive to time 1) Hazard ratio = hazard function in placebo/hazard function in treatment HRs are often given with Kaplan–Meier survival curves and calculated from Cox regression analysis, and, although they may give the likelihood of one patient group reaching the endpoint first, they do not convey how fast the changes were. 3.11.3 Statistical and clinical significance Understanding the clinical and statistical interpretation of comparative effectiveness study results is imperative to discern the main clinical significance and its relevance in treating patients. Statistical significance is what readers (investigators, researchers, physicians, sponsors, etc.) tend to look for and use as a basis for their clinical conclusions. But clinical significance is key when interpreting and providing clinical context to the data.

Introduction to Comparative Effectiveness Research

CI. If the 95% interval excludes 1, then the estimate is statistically significant. The CI can provide more information than the p-value alone because it is used to indicate reliability and precision of an estimate. Application of

Both p-value and 95% CI are used to indicate statistical significance (Table 4). Study Methods

Statistical significance One must be familiar with two important statistical terms used to define statistical significance. • p-value: This is the probability that a difference, as large as or larger than that observed in a study, could have occurred by chance. The smaller the p-value, the more likely it is that the difference between groups was due to treatment effect. • Significance testing provides an assessment of how strong the evidence is in support of a genuine treatment difference. For example, a p-value of <0.05 indicates that there is a 5% chance that the difference between the groups comparedIntroduction is due totochance. In Comparative Effectiveness and Evidence-Based Medicine © Rx Values Group Ltd. All rights reserved other words, one in every 20 (i.e. 5%) of every truly negative trial will produce a false-positive result. The most widely used p-value is <0.05 indicating a statistical significance.

• 95% confidence intervals: The confidence interval (CI) of an estimate such as a percentage (%), OR, HR, and RR provides an indication of the range of values within which the real value is likely to lie. If an experiment is repeated 100 times, the calculated estimate will lie within the 95% CI 95 times out of 100. It is widely thought that a 95% CI around an estimate means that we can be 95% confident that the true estimate lies in the range between the lower and upper limit of the CI. If the 95% confidence interval excludes 1 for an RR, OR or HR, then the estimate is statistically significant. The CI can provide more information than the p-value alone because it is used to indicate reliability and precision of an estimate. Both p-value and 95% CI are used to indicate statistical significance. When considering confidence intervals, the size of the confidence interval is important. A wider confidence interval means less precision in the estimate, and is sometimes due to a smaller sample size or more variance. © Rx Values Group Ltd. All rights reserved

Introduction to Comparative Effectiveness Research 37

Application of Study Methods

difference in the behaviour of the level of care. However, this eyes of the insurer who sees no

gnificance statistical significance are . Our ultimate goal is to achieve nce that may become the

d clinical significance, four

ent. The study confirmed its

nificance is not. The study did not

re tests you do the more likely

nificance is not. The study did not

m its ht

Clinical significance Criteria to evaluate clinical significance must be defined prior to starting a study. Usually, a group of experts with various backgrounds are involved in establishing clinical significance, depending on the question under investigation. One must be aware of the following attributes when determining clinical significance: clinical improvement, benefit/risk ratio, safety, quality of life (patient reported outcomes), compliance, and other attributes. For example, it is important to know how large a response needs to be compared with placebo or an alternative treatment, in order to convince physicians to use the treatment. Ideally, researchers want to demonstrate that a new therapy is superior to or at least equal to the clinical improvement caused by an existing therapeutic agent. Bear in mind that the evaluation of clinical significance, when considering alternative treatments, will depend on more perspectives than just those of the study investigators. Stakeholders who evaluate clinical significance include investigators, healthcare professionals, patients, regulatory agencies, payers, and pharmaceutical and medical device companies. Each of these stakeholders may have different view of what is significant clinically. For example, a care giver might consider clinical significance to mean enough of a difference in the behavior of a patient with Alzheimer’s disease, to give them a respite in the level of care. However, this difference may not be considered significant in the eyes of the insurer who sees no change in staffing levels.

Effectiveness Evidence-Based Medicine Effectiveness 37 38 andIntroduction to Comparative Research

Application of Study Methods

Relationships between statistical and clinical significance The relationships between clinical significance and statistical significance are vitally important to the interpretation of the results. Our ultimate goal is to achieve accurate and relevant conclusions about the evidence that may become the foundation for changes made in clinical practice. In considering relationships between statistical and clinical significance, four outcomes are possible. 1. Both statistical and clinical significance are present. The study confirmed its objectives. 2. Statistical significance is present but clinical significance is not. The study did not confirm clinical objectives, possibly due to: -- confounding -- the use of too many statistical tests (the more tests you do the more likely you are to find something) -- a very large study population (the larger the population the more likely you are to find small but statistically significant differences between groups that are essentially clinically meaningless). 3. Clinical significance is present but statistical significance is not. The study did not confirm its objective, possibly due to: -- too small a study sample -- use of inappropriate statistical tests. 4. Neither statistical significance nor clinical significance is present. The study did not confirm its clinical objective (but further investigation might be warranted to ensure that this is a real result). 3.12 How are comparative effectiveness results used? Overall, the results of comparative effectiveness should: • Inform healthcare professionals and devise treatment guidelines • Ensure appropriate use of therapies/technologies and not hinder access to new therapies/technologies • Be updated to remain current with scientific evidence • Not be used as a tool to limit access to healthcare to subsequently decrease cost. When discussing the benefits, risks, or costs of an intervention, a number of different perspectives have to be considered, ranging from the societal perspective to that of an individual (Figure 15).

Introduction to Comparative Effectiveness Research 39

••• be current withscientifi scientificccevidence evidence be updated to remain current with scientifi evidence beupdated updatedto toremain remaincurrent currentwith ••• not healthcareto tosubsequently subsequently not to be used as tool to limit access to notto tobe beused usedas asaaatool toolto tolimit limitaccess accessto tohealthcare decrease decrease cost. decreasecost. cost. When of diff erent When benefi ofan anintervention, intervention,aanumber erent discussing the benefi costs Whendiscussing discussingthe thebenefi benefits, ts, ts,risks, risks,or orcosts costsof numberof ofdiff different risks, or perspectives have to be considered, ranging from the societal perspective to that of perspectives considered, ranging from the societalperspective thatof of perspectiveshave haveto tobe beconsidered, considered, ranging perspectiveto tothat that an an individual (Figure 12). anindividual individual(Figure (Figure12). 12). Figure12. 15.Healthcare-related Healthcare-related perspectives perspectives Figure Figure Healthcare-related perspectives Figure12. 12.Healthcare-related Healthcare-related perspectives

Patients Patients Patients Patients Patients

Hospitals Hospitals Hospitals Hospitals Hospitals

Society Society Society Society Society Society

Employers Employers Employers Employers Employers

Whose Whose Whose Perspective? Perspective? Perspective? Perspective?

Pharmaceutical Pharmaceutical Biotech/Pharma Pharmaceutical Pharmaceutical Manufacturers Suppliers Suppliers Suppliers Suppliers

Insurers Insurers Insurers Insurers

Government Government Government Government

Healthcare Healthcare Healthcare Healthcare Healthcare Professionals Professionals Professionals Professionals Professionals

The scientific evidence provided by CER has to be translated into understandable information that can be applied by patients, providers and policy-makers (government) to help in their healthcare decisions. Dissemination of results to healthcare organizations, patients, advocacy groups, and other interested parties is a key element in the implementation ofand CER. Providing information that©©©©Rxis accurate and 38 38 Introduction Introduction to to Comparative Comparative Eff Eff ectiveness ectiveness and and Evidence-Based Evidence-Based Medicine Medicine 38 Introduction Introduction toComparative ComparativeEff Effectiveness ectiveness and Evidence-Based Medicine Group rights RxValues Values GroupLtd. Ltd.All All rightsreserved reserved Rx Values Group Ltd. All rights reserved 38 to Evidence-Based Medicine 38 Introduction to Comparative Eff ectiveness and Evidence-Based Medicine ©Rx RxValues ValuesGroup GroupLtd. Ltd.All Allrights rightsreserved reserved useful to both technical and general audiences presents a significant challenge, particularly if there is an element of uncertainty in the conclusions. However, it is well known that even when information is clear, unambiguous, and widely disseminated, it still might not influence practice patterns. In interpreting the results of comparative effectiveness studies it must be recognized that the results obtained represent an ‘average’ result and as such may not inform the decisions about an individual patient. Furthermore, a study that finds no difference in outcomes between two treatments does not necessarily mean that there is no difference. The lack of a statistical difference between the treatments may be due to a patient sample size that is too small or some other flaws in the study design.

Introduction to Comparative Effectiveness Research

Evidence-based Medicine

Evidence-based medicine

4.1

What is evidence-based medicine?

4.1 Ev

Evidence-based medicine (EBM) - has been defined as: “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating clinical expertise with the best available external evidence from systematic research.”27 4.2

Collecting the evidence

The phases of drug regulation, from discovery of a new molecular entity (NME) to regulatory approval and post-approval, are shown in Figure 16. The vast majority of compounds that pharmaceutical companies identify as having therapeutic potential never actually make it through this lengthy process to reach the market. Of every 5000 new compounds identified at screening, after 3–6 years of testing only five will reach preclinical testing. Only one of these five will ultimately be approved by regulatory agencies for clinical use to treat a specific disease.

Introduction to Comparative Effectiveness Research 41

“Th be the ex fro

4.2 Th ch ap co as thr ev aft clin ap as

Figure 16. From drug discovery to post-marketing

Discovery Identifying new molecular entities (NMEs)

Pre-clinical testing Lab and animal research

Phase I

Safety and dosage studies

Phase II

Efficacy and safety studies

Phase III

Long-term efficacy and safety studies

Review and approval Post-approval period (life cycle management)

0 2 4

8 10 12 14 16 18 20 22 24 Years

After pre-clinical testing to ensure adequate safety, the clinical trials carried out in human subjects are designed to gather information on pharmacology, pharmacokinetics, efficacy, and safety. This information forms the basis of the evidence that doctors use in making treatment decisions. Phase I studies involve testing an experimental compound in a small number of healthy volunteers, or, in some cases, in patients with the target illness. This is the first testing in humans and so aims to determine the pharmacology and pharmacokinetics of the compound, its side effects and evidence of effect. Phase I trials can take 1–3 years to complete and generally one-third of all compounds fail at this stage. Phase II studies are controlled trials that involve patients with the disease to further evaluate the efficacy and safety of the new compound. The experimental drug must show a unique benefit and/or a unique safety profile at this stage. In some cases, even minimal benefits may be important; for example, in situations where the target disease is currently untreatable. This phase can take 1–2 years, and only about half of all drugs that enter this phase complete it successfully.

Introduction to Comparative Effectiveness Research

Evidence-based Medicine

Phase III studies are large, involving hundreds to thousands of patients. These are time consuming and very costly to perform. They are designed to provide data that are required by the regulatory authorities prior to a product gaining formal approval for use. These studies are RCTs in which the control group is either a placebo or an alternative treatment already available for the indication under study. In general, the studies last several years, and most drugs that reach this phase are successfully launched onto the market. Phase IV studies are carried out after regulatory approval has been obtained, in order to find out more about the medicine in clinical practice. These trials may be initiated for many different reasons; for example to address any questions that arose during phases I to III, to investigate drug interactions, or to better understand finer points about efficacy and safety, and patterns of drug use. Methods of surveillance include spontaneous adverse event reports, observational studies, registries, and surveys. Comparative effectiveness, health economics, and outcomes research studies are conducted in the post-approval period to support the value proposition of a therapeutic approach. This post-marketing surveillance continues throughout the life cycle of the drug. In 2005, the FDA published a guidance document on pharmacovigilance (PV) and pharmacoepidemiological assessment in the postapproval period28.

and com

Phas thou very data to a stud plac for th last s this

Phas appr abou be in to ad I to I unde patte

Meth even surv and in th prop mark life c guid phar appr

Introduction to Comparative Effectiveness Research Eff43 42 Introduction to Comparative ectiveness a

4.3 Making the evidence available for consideration by decision-makers The information from clinical trials and observational studies is presented at scientific meetings, and reported in peer-reviewed and specialist medical publications. In addition, the FDA requires all clinical trials to be registered on a public website (www. clinicaltrials.gov) and, from September 2008, primary and secondary endpoint results, plus adverse events from clinical trials, must be tabulated on www.clinicaltrials.gov29. Once the studies are published they are often summarized and reviewed in the form of systematic reviews or meta-analyses. Published evidence varies in terms of quality and credibility and, as a result, ranking systems or hierarchies of evidence have been established to guide decision-makers as to which types of evidence are the best. The hierarchy of the “level of evidence” associated with the type of study methods is often depicted in a pyramid as shown in Figure 17. While most ranking systems agree that systematic reviews, meta-analyses and RCTs provide the most credible evidence, the growth in the development of comparative effectiveness studies has generated some new thinking on this subject30. In the current era of CER, the traditional concept of the ‘evidence pyramid’ has been under debate. Specifically, the appropriateness of the study design and methodology selected for a given CER study depends largely on the research question being asked. In CER, not all health care questions can be adequately or appropriately addressed by RCTs. For example, a study evaluating treatment effectiveness and long-term safety in the general patient population may be best evaluated using a prospective observational (registry) study. In summary, there is no single study design or method that is the best in answering all research questions. Figure 17. The traditional ‘evidence pyramid’ Metaanalysis Systematic review Randomized controlled trial

In CER the appropriate method depends on the research question

Cohort studies Case-control studies Case series/Case reports Animal research/Laboratory studies

Introduction to Comparative Effectiveness Research

Evidence-based Medicine

4.4 4.4

Using the evidence

4.4.1

4.4.1 How do healthcare practitioners make sense of all the evidence?

The mak paye conc situa orga appr on o deci open and prac thera revie reco supp the i Assu and ‘

The increasing amount of information now available makes it extremely difficult for prescribers and payers to assimilate the key evidence, to draw valid conclusions, and to apply the evidence to the clinical situation. Historically, ‘practice guidelines’ from medical organizations and healthcare agencies regarding the appropriate use of various treatments have relied on opinions from experts rather than objective decisions based on data review. Such methods are open to criticism because they are subject to bias and misinterpretation. More recently, evidence-based practice guidelines have been developed for many therapeutic areas. These are based on a systematic review of the evidence to produce a series of graded recommendations that are directly linked to the supporting data. However, guidelines are only developed as tools to help physicians and patients make individual treatment decisions based on the available information. Importantly, healthcare professionals are increasingly encouraged to involve patients in their treatment decisions. The medical world is beginning to recognize patients as experts with a unique knowledge of their own health, and driven by patient empowerment initiatives it is now considering their preferences for treatments.

How help deci Impo enco deci patie own initia treat

4.4.2 How do payers make sense of all the evidence?

Ther EBM (Figu

Evidence-based decision-making involves payers and quality improvement organizations, as well as prescribers. Economic studies looking at issues related to the cost of an intervention should be considered along with the clinical evidence. 44 © Rx Values Group Ltd. All rights reserved

Introduction to Comparative Effectiveness a

Introduction to Comparative Effectiveness Research 45

• t s oc / y t i l au q = e u l aV

•

Value = quality/cost • yaP r ni g ot hW ohs p ot .4.4 roF ppa eht oc slA b ot opu kcal t fo ehT iled orF arp rac rah rac BE am owt

Economic analyses can be integrated into a study protocol looking at efficacy, effectiveness, and safety, or can be carried out in a ‘usual care’ setting. These two situations will provide different measures of the economic cost of treatment. The costs that can be considered in economic analyses are: • Direct medical costs for provision of care, including drug costs, staff costs, and cost of diagnostic tests • Direct non-medical costs, including travel expenses, and accommodation costs incurred by the need to seek medical care • Productivity costs related to the morbidity resulting from the illness, including time off work, or mortality due to premature death • Intangible costs, including those of pain, suffering, and grief. Payers need to consider the economic evidence in relation to budget constraints and value, and to guide pricing and reimbursement decisions. When they assess the value of an intervention, they should consider both the short- and long-term value to patients. 4.4.3 How do policy-makers and government use the evidence? For policy-makers, evidence-based medicine (EBM) is a scientific and systematic approach that enables them to reduce waste within the healthcare system at different levels without compromising the safety or quality of medical care. Also, the government’s role is to help decision-makers to base the crucial decisions that affect patient care upon solid evidence, and if systematic evidence is lacking in a particular area, to accelerate the generation of that evidence to meet the gaps.

ssenevitc46 effE evIntroduction itarapmoC oto t nComparative oitcudortnI Effectiveness 64 Research

Evidence-based Medicine

The main reason for the use of EBM is to optimize the delivery of health services. From the published literature, it is clear that ‘real-world’ practices differ from what ‘should be’ evidence-based care, which may lead to high waste, low value, and harm. Hence, policies to ensure that evidence-based care is followed should help. The rationale for using EBM is that patients and populations in the real-world may differ from the populations included in one or two published clinical trials. Thus, EBM guidelines incorporate all the available data for a particular condition through comparative effectiveness and thorough research, to correct for these variations in populations to ensure the right care for the right patient at the right time. In the USA, the governmental utilization of EBM in action includes use by Medicare in National Coverage Determinations (NCD) and these activities are overseen by the Centers for Medicare and Medicaid Services (CMS). The assessment of health technologies within the Medicare and Medicaid programs is both ‘complicated and fluid’31 and beyond the scope of this booklet. Many national agencies in Europe and the rest of the world issue guidelines for treatment using EBM. For example, NICE in the UK issues guidelines for the treatment of diseases at regular intervals, based on the changing therapeutic landscape and the importance of the disease to the UK healthcare system. Some countries do not allow the use of medicines outside those listed in the guidelines unless patients are participating in clinical trials.

Introduction to Comparative Effectiveness Research 47

5. How atcomparative evitatoraevaluate pmoc a e aulave ot woH effectiveness study or article elcitra ro yduts ssenevitceffe

Whenelreading alpublished citra dehand silbevaluating up a gnitau ave dna gnarticle idaer nehW about comparative effectiveness, it is important ot tnatropmi si ti ,ssenevitceffe evitarapmocto tuoba understand the study design that was used and to ot dna desu saw taht ngised yduts eht dnatsrednu recognize any deficiencies, how they were addressed, desserdda erew yeht woh ,seicneicfied yna esingocer and the effect they may have had on the results. No oN .stluser eht no dah evah yam yeht tceffe eht dna study is perfect and the best intention of the authors srohtua eht fo noitnetni tseb eht dna tcef rep si yduts should be assumed. Never take anything for granted detnisarnot g rospecifically f gnihtynastated ekat reand veNalways .demuask ssa questions eb dluohs that swhere noitseuthere q ksaissydoubt. awla dBe naobjective detats ylland acfiitake cepsatobalanced n si taht deview cnalof abthe a einformation kat dna evitprovided. cejbo eB .tbuod si ereht erehw

.dedivorp noitamrofni eht fo weiv

The following elements should be taken into a gnitaulawhen ve neevaluating hw noitareadcomparative isnoc otni nekat eb dluohs stnemele gniwollof ehT consideration .elcitra ro yduts ssenevitceffe evitarapmoc effectiveness study or article. 5.1

Data collection

• What were the study objectives? • Were the hypotheses clearly defined?

noitcelloc ataD 1.5 ?sevitcejbo yduts eht erew tahW • ?denfied ylraelc sesehtopyh eht ereW

• What was the outcome and was it accurately ?derusaem yletarucca ti saw dna emoctuo eht saw tahW measured? • What was the exposure ?derand usaewas m ylitetaccurately arucca ti saw dna erusopxe eht saw tahW measured?

• •

?morf nekat elpmas yduts eht saw noitalupop tahW

•

?devired ezis elpmas eht saw woH

•

?saib noitceles yna ereht saW

•

• What population was the study sample taken from? • How was the sample size derived? • Was there any selection bias?

?t-s- erwas etni selection fo erusopof xesubjects ro emocbased tuo ehon t nthe o doutcome esab stcejbus fo noitceles saw

–

?sloor rtnexposure oc dna seofsainterest? c neewteb dereffid evah noitceles dluoc yaw tahw ni

–

in what way could selection have differed ?saib noitamrofni yna ereht saw between cases and controls?

–

was information collected in the same manner ?slexposure ortnoc dnora outcome sesac neefor wtall ebpatients? dereffid evah ti dluoc yaw tahw ni on the

–

in what way could it have differed between cases and controls?

emoctuo ro erusopxe eht no rennam emas eht ni detcelloc noitamrofni saw – -- was there any information bias? ?stneitap lla rof

•

Introduction to Comparative Effectiveness Research

How to Evaluate Studies or Articles

• Was there any potential confounding? • Were there efforts to minimize the influence of external factors prior to analysis, for example, by the study design or restriction of subjects? 5.2

Data analysis

• Were data appropriately reported? • Were the statistical analyses appropriate and well described? • Were potential confounders addressed in the analyses? • Were the data discussed in the discussion section reported in the results section? • Were differences in the baseline characteristics of patient groups addressed in the analyses and results? 5.3

Interpretation

• What was the major result of the study? • How would the interpretation of this result be affected by previously described biases (consider direction and magnitude)? • How would the interpretation of this result be affected by any random or non-random misclassification of exposure or outcome (consider direction and magnitude of bias)? • To what larger population may this study be generalized? • Were the main study findings discussed and compared with the published literature? • Did the discussion adequately address the limitations of the study? • Did the final conclusion accurately reflect the study findings? • Who funded the study?

Introduction to Comparative Effectiveness Research 49

Glossary of terms and acronyms

Absolute risk

•

Association • Benefit

•

The likelihood that something will happen An observed relationship between a risk factor and a particular outcome, linked either through a causal or non-causal relationship A positive or favorable outcome of the treatment

Bias •

A non-random error introduced into a study by its design, conduct, or analysis, that leads to distorted results

Case-control study •

An observational retrospective study that involves identifying patients who have a disease or outcome of interest (cases) and a suitable control group without the disease or outcome, and examines the relationship of an intervention, exposure, or risk factor to the outcome of interest and how it differs between the two groups

Case-crossover •

A study that compares cases with a disease to a different time study period, in the same individuals, looking for differences in antecedent exposures

Cohort study •

An observational prospective study that identifies subjects with a particular risk factor and a control group without the risk factor, and follows up both groups over time to determine the development of a disease or outcome of interest

Comparative • effectiveness research

The generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition, or to improve the delivery of care.

•

A range of values between which the true population value probably lies

Confidence interval

Confounder •

A variable other than the exposure (risk factor) and outcome under study that is independently related to both the exposure and the outcome

Confounding by the • indication

Can occur when the underlying diagnosis or other clinical features that result in the use of a certain drug are also related to the patient outcome

Cost effectiveness •

Compares the cost of a healthcare intervention in monetary units to its effectiveness determined independently using an appropriate, clinically meaningful unit

•

A study that examines exposures and outcomes at one point in time

Cross-sectional study

Efficacy •

The measured extent to which a medicine works under ideal circumstances in a clinical trial

Epidemiology •

The study of the distribution and determinants of disease frequency in populations

Introduction to Comparative Effectiveness Research

Glossary

Evidence-based • medicine

The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating clinical expertise with the best available external evidence from systematic research

Exposure •

Characteristics of the subject that are hypothesized to have caused the outcome of interest

Hazard ratio •

A type of relative risk, that indicates the odds of an event (e.g. healing) happening at one point in time with a particular treatment intervention

Incidence •

The number of new events/cases that develop in a population at risk during a specified period of time

Meta-analysis •

A statistical analysis that combines the results of several studies that address a set of related research hypotheses

Observational study •

A study where the investigator does not control the treatment given to the patient, but monitors, observes, and analyzes the study results

Odds ratio •

Odds of exposure in diseased group divided by the odds of exposure in the non-diseased group

Outcome

•

Study endpoint, for example, death

p-value •

The probability that a difference as large as, or larger than, that observed in the study could have occurred purely by chance

Pharmacovigilance •

The science and activities relating to the detection, assessment, understanding, and risk management of adverse effects or any other drug-related problem

Prevalence •

The proportion of individuals in a population who have the disease at a specific time point

Probability •

The likelihood or chance that something is the case or that something will occur

Random error

•

An error occurring by chance, also known as non-systematic error

Relative risk •

Ratio of the incidence rate of an outcome in an exposed group over the incidence rate of the outcome in a non-exposed group

Selection bias •

Error in a study due to systematic differences in characteristics between those who are selected for the study and those who are not

Systematic error

•

See Bias

Introduction to Comparative Effectiveness Research 51

ADR

Adverse drug reaction

AHRQ

Agency for Healthcare Research and Quality

CEA

Cost-effectiveness analysis

CER

Comparative Effectiveness Research

Confidence interval

CMS

Centers for Medicare & Medicaid Services

CUA

Cost-utility analysis

DERP

Drug Effectiveness Review Project

DHHS

US Department of Health and Human Services

EBM

Evidence-based medicine

EPC

Evidence-based Practice Center

ICD-9-CM

International Classification of Disease-9th revision-Clinical Modification

ICER

Incremental cost-effectiveness ratio

MTC

Mixed treatment comparison

NICE

National Institute for Health and Clinical Excellence

NIH

National Institutes of Health

NHS

National Health Service

NME

New molecular entity

Odds ratio

PCORI

Patient centered outcomes research institute

PPACA

Patient Protection and Affordable Care Act

PRO

Patient-reported outcome

Probability

Pharmacovigilance

QALYs

Quality-adjusted life-years

RCT

Randomized controlled trial

Relative risk

Veteran Administration

Introduction to Comparative Effectiveness Research

References

1. Institute of Medicine of the National Academies. Initial National Priorities for Comparative Effectiveness Research. June 30, 2009. Available at: http://iom.edu/Activities/Research/ CERPriorities.aspx. Accessed 9 December 2010. 2. Gold MR, Siegal JE, Russell LB, Weinstein MC (Eds). Cost Effectiveness in Health and Medicine. New York, NY, Oxford University Press, 1996: 392–411 3. Ip S, Bonis P, Tatsioni A, Raman G, Chew P, Kupelnick B, et al. Comparative Effectiveness of Management Strategies for Gastroesophageal Reflux Disease. Comparative Effectiveness Review No. 1. (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under Contract No. 290-02-0022.) Rockville, MD: Agency for Healthcare Research and Quality. December 2005. Available at: www.effectivehealthcare.ahrq.gov/reports/final.cfm. 4. ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group. The Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial. Major outcomes in high-risk hypertensive patients randomized to angiotensin-converting enzyme inhibitor or calcium channel blocker vs diuretic: The Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT). JAMA 2002; 288: 2981–2997 5. Cannon CP, Braunwald E, McCabe CH, Rader DJ, Rouleau JL, Belder R, et al; Pravastatin or Atorvastatin Evaluation and Infection Therapy-Thrombolysis in Myocardial Infarction 22 Investigators. Intensive versus moderate lipid lowering with statins after acute coronary syndromes. N Engl J Med 2004; 350: 1495–1504 6. William E. Boden, Robert A. O’Rourke, Koon K. Teo, Pamela M. Hartigan, David J. Maron, William J. Kostuk, et al, for the COURAGE Trial Research Group. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med 2007; 356: 1503–1516 7.

Agency for Healthcare Research and Quality (AHRQ). ARHQ Mission. http://www.ahrq.gov/about/budgtix.htm. Accessed 7 January 2011

8. National Guideline Clearing House. www.guideline.gov. Accessed 7 January 2011 9. The American Recovery and Reinvestment Act of 2009. Available at: http://www.recovery.gov/About/Pages/The_Act.aspx; and http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=111_cong_ bills&docid=f:h1enr.pdf. Accessed 10 December 2010. 10. The Patient Protection and Affordable Care Act. Available at: http://democrats.senate.gov/reform/patient-protection-affordable-care-act-aspassed.pdf. Accessed 10 December 2010. 11. ISPOR 2nd Asia-Pacifi c Conference March 2006. First Plenary Session: Pharmacoeconomics and Outcomes Research in Asia-Pacifi c: China, Japan, South Korea, Singapore, Thailand, Pakistan, Malaysia and India. http://www.ispor.org/conferences/shanghai0306/Plenary1_tbl.pdf

Introduction to Comparative Effectiveness Research 53

12. ISPOR Ist Latin America Conference September 2007. First Plenary Session Matrix: Pharmacoeconomics and Outcomes Research in Latin America: Argentina, Brazil, Chile, Colombia, Mexico and Venezuela. http://www.ispor.org/conferences/Colombia0907/ISPOR_ LAProgramMatrix.pdf 13. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB (Eds). Designing Clinical Research. 3rd edition. Philadelphia, PA, Lippincott, Williams & Wilkins, 2007 14. Comparative effectiveness of second-generation antidepressants in the pharmacologic treatment of adult depression. AHRQ CER No.7, January 2007 15. The Cochrane Library. http://www.thecochranelibrary.com/view/0/index.html. Accessed 7 January 2011 16. Bucher HC, Guyatt GH, Griffi th LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomised controlled trials. J Clin Epidemiol 1997;50:683-91. 17. Vandermeer BW, Buscemi N, Liang Y, Witmans M. Comparison of meta-analytic results of indirect, direct and combined comparisons of drugs for chronic insomnia in adults: a case study. Med Care 2007;45:S166-S172. 18. Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Statist Med 2004;23:3105-24 19. Higgins JPT, Green S (Eds). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.2 (updated September 2009). http://www.cochrane-handbook.org/. Accessed 18 December 2010 20. J Lyle Bootman, Raymond J Townsend, William F McGhan (Eds). Principles of Pharmacoeconomics. Harvey Whitney Books Company, Cincinnati, Ohio, USA; 3rd Edition, July 2004 21. Schneeweiss S, Patrick AR, Sturmer T, Brookhart MA, Avorn J, Maclure M, et al. Increasing levels of restriction in pharmacoepidemiologic database studies of elderly and comparison with randomised trial results. Med Care 2007; 45 (10 Suppl 2): S131–S142 22. Gliklich RE, Dreyer NA (Eds). Registries for evaluating patient outcomes: A User’s Guide. 2nd edition. Rockville, MD: Agency for Healthcare Research and Quality. September 2010. 23. Lohr KN. Emerging methods in comparative effectiveness and safety: symposium overview and summary. Med Care 2007; 45 (10 Suppl 2): S5–S8 24. Vandermeer BW, Buscemi N, Liang Y, Witmans M. Comparison of meta-analytic results of indirect, direct and combined comparisons of drugs for chronic insomnia in adults: a case study. Med Care 2007; 45 (10 Suppl 2): S166–S172 25. Horn SD, Gassaway J. Practice-based evidence study design for comparative effectiveness research. Med Care 2007; 45 (10 Suppl 2): S50–S57

Introduction to Comparative Effectiveness Research

References

26. Buckley T. The complexities of comparative effectiveness. BIO (Biotechnology Industry Organization), October 25, 2007 27. Sackett DL, Rosenberg WMC, Gray JA, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isn’t. BMJ 1996; 312: 71–72 28. FDA Guidance for Industry. Good pharmacovigilance practices and pharmacoepidemiological assessment. March 2005. http://www.fda.gov/Cder/Guidance/6359OCC.htm. Accessed 1 July 2008 29. FDA Amendment Act. http://www.fda.gov/oc/initiatives/hr3580.pdf. Accessed 1 July 2008 30. Atkins D. Creating and synthesizing evidence with decision makers in mind: integrating evidence from clinical trials and other study designs. Med Care 2007; 45 (10 Suppl 2): S16–S22 31. Sullivan SD, Watkins J, Sweet B, Ramsey SD. Health technology assessment in health-care decisions in the United States. Value Health 2009; 12(2): S39–S44

Introduction to Comparative Effectiveness Research 55

Introduction to Comparative Effectiveness Research

An Rx Values Group Ltd Publication Reprinted by Genentech with permission from Rx Values Group Ltd.