Glossary and Suggested resources

Annex

Glossary

Causal inference

Reliably demonstrating a connection between an intervention and an effect (an observable change). This results in strong evidence that the intervention is responsible for a certain outcome.

Causal power

The extent to which an experiment can approximate the counterfactual by establishing a clear link between the intervention and any impacts that are observed. Since randomized experiments get closest to approximating the counterfactual, they have the highest causal power while pre-post experiments have the lowest causal power.

Comparison group

When randomly assigning participants to either a treatment or control group is not possible or ethical, nonrandomized experiments or quasi-experimental designs can instead use statistical models to define a comparison group that is as similar to the treatment group as possible, but is not exposed to the intervention. The comparison group is then compared with the treatment group.

Confounders

Factors other than an intervention that participants in an experiment might be exposed to. An experiment must have enough participants (i.e., a large enough sample size) and enough similarity between the control or comparison and treatment groups to ensure that the likelihood of exposure to any of these other influencing factors is equal across the groups. If not, these confounders might result in observable differences in the results of the groups being compared, which could make it more difficult or impossible to understand whether the intervention has had an effect or not.

Control group

In a randomized experiment, participants are randomly assigned to either the treatment group or the control group. The control group is the one that does not receive the intervention (product, service, approach, policy, or process) that is being tested.

Counterfactual

In an experiment, this can be understood as an estimation of what would have happened to a treatment group if the experiment hadn’t taken place and they hadn’t been exposed to an intervention. As it’s not possible to observe this directly, experiments are designed to approximate this as closely as possible by establishing a control group in a randomized experiment, a comparison group in a non-randomized experiment, or, in the case of a pre-post experiment, observing a baseline.

Innovation

The process by which new ideas turn into practical value in the world: new products, services, or ways of doing things.

Innovator

An individual or organization developing innovation.

Minimum detectable effect size (MDES)

The smallest true effect size that has a good chance of being found to be statistically significant.

Power calculation

An estimate of the probability that a trial with a specified number of participants will detect a statistically significant intervention effect of a certain size.

Prototype

A prototype is a draft manifestation of a concept or idea that makes it tangible, shareable, and testable. Depending on the concept or idea that needs testing, this could be anything from a drawing, a hypothesis, a written description of a program, a diagram of a process, a scale model, a rehearsal or dry run, or a demonstrator technology before it has been produced at scale. Prototypes provide a basis for further development to improve a concept or idea before it takes its final form.

Regulatory experimentation

A regulatory experiment is a test or trial of a new product, service, approach or process designed to generate evidence or information that can inform the design or administration of a regulatory regime.

Regulatory sandbox

A regulatory sandbox is a facility, created and controlled by a regulator, designed to allow the conduct of testing or experiments with novel products or processes prior to their full entry into the marketplace.

Selection bias

If participants in an experiment are assigned to an experimental group (treatment, control, or comparison) based on factors that make this group distinct from the other experimental group, then this can result in selection bias. For example, if a regulator was looking to measure the effect of an opt-in program, it might conclude that the program had an effect because the outcomes for the participants opting in and those not opting in could look quite different. However, the participants who proactively signed up could be more motivated or distinct from the control or comparison group in other ways, so it might be these factors that account for the difference between the groups rather than the intervention.

Treatment group

Also known as the ‘intervention’ group, this designates the participants in an experiment who are exposed to the product or service, new approach to regulating, or policy or regulatory process that is being tested.

Suggested resources

Experimentation Works

An initiative led by Treasury Board Secretariat to build the capacity of public servants in experimentation skills and practice. It applies a unique learning-bydoing model that supports and showcases small-scale experiments in the open. You can find more information about Experimentation Works at: www.canada.ca/en/ government/publicservice/modernizing/experimentationworks.html