Regulators’ Experimentation Toolkit • 2: Regulatory experiments
Part C: Which type of experiment should you use? Use the following sections to complete the Experimental design section of the Is regulatory experimentation right for you? worksheet. Experiments can be run in different ways. There are three core types of experimental design: Randomized experiments Often considered the ‘gold standard’ of experiments, these experiments separate participants into a treatment group which receives an intervention and a control group which doesn’t in order to understand the impact of the intervention. These groups are randomly assigned to ensure there is no bias in how they are allocated. Non-randomized and quasi-experimental designs Randomization isn’t always possible, and in these types of experiments a comparison group is created using statistical models to ensure it is as similar to the treatment group as possible. The treatment group receives the intervention and the comparison group doesn’t, and the results are compared. Pre-post experiments If there is no way to create a comparison group, the same group can be measured before and after receiving an intervention. In their own way, each of these approaches aims to create what’s called the counterfactual, an estimation of what would have happened if the experiment hadn’t taken place. This in turn influences the causal power (a clear link between the intervention and the changes observed) that can be inferred or established as a result of the experiment, with a randomized experiment giving high causal power and a pre-post experiment low causal power. Note that because experimentation is still a relatively new approach within regulatory practice, there are not yet many examples of regulatory experiments that match these categories exactly as they are described. While establishing a counterfactual is best practice, not all regulatory experiments may do this. However, any type of well-designed experiment has the potential to help regulators improve the rigour and efficiency of their evidence and insight-gathering activities. While a randomized experiment will always give experimenters the highest causal power, both non-randomized/quasi-experimental designs and pre-post experiments are good options for regulators and in some cases may be more feasible to implement.
28
