Regulators’ Experimentation Toolkit • Annex: Glossary and Suggested resources
Annex Glossary Causal inference Reliably demonstrating a connection between an intervention and an effect (an observable change). This results in strong evidence that the intervention is responsible for a certain outcome.
Causal power The extent to which an experiment can approximate the counterfactual by establishing a clear link between the intervention and any impacts that are observed. Since randomized experiments get closest to approximating the counterfactual, they have the highest causal power while pre-post experiments have the lowest causal power.
Comparison group When randomly assigning participants to either a treatment or control group is not possible or ethical, nonrandomized experiments or quasi-experimental designs can instead use statistical models to define a comparison group that is as similar to the treatment group as possible, but is not exposed to the intervention. The comparison group is then compared with the treatment group.
Confounders Factors other than an intervention that participants in an experiment might be exposed to. An experiment must have enough participants (i.e., a large enough sample size) and enough similarity between the control or comparison and treatment groups to ensure that the likelihood of exposure to any of these other influencing factors is equal across the groups. If not, these
85
confounders might result in observable differences in the results of the groups being compared, which could make it more difficult or impossible to understand whether the intervention has had an effect or not.
Control group In a randomized experiment, participants are randomly assigned to either the treatment group or the control group. The control group is the one that does not receive the intervention (product, service, approach, policy, or process) that is being tested.
Counterfactual In an experiment, this can be understood as an estimation of what would have happened to a treatment group if the experiment hadn’t taken place and they hadn’t been exposed to an intervention. As it’s not possible to observe this directly, experiments are designed to approximate this as closely as possible by establishing a control group in a randomized experiment, a comparison group in a non-randomized experiment, or, in the case of a pre-post experiment, observing a baseline.
Innovation The process by which new ideas turn into practical value in the world: new products, services, or ways of doing things.
Innovator An individual or organization developing innovation.
