3 ITT treatment effects on learning

Table 3 shows results from student tests at baseline and at follow-up one year later. The first two columns show differences between control and treatment schools’ test scores at baseline (September/October 2016), while the last four columns show the difference in May/June 2017. In our preferred specification (Column 5) the treatment effect of PSL after one year is .18σ for English (p-value < 0.001) and .18σ for math (p-value < 0.001). Table A.4 in Appendix A shows both the ITT and the treatment-on-the-treated (ToT) effect (i.e., treatment effect only for students that actually attended a PSL school in 2016/2017), while Table A.5 shows the ITT effect using different measures of student ability.

Table 3: ITT treatment effects on learning

Baseline One-year follow-up Difference Difference Difference Difference Difference Difference (F.E.) (F.E.) (F.E. + Controls) (ANCOVA) (1) (2) (3) (4) (5) (6) English 0.06 0.09∗∗ 0.17∗∗ 0.17∗∗∗ 0.18∗∗∗ 0.13∗∗∗ (0.08) (0.05) (0.08) (0.04) (0.03) (0.02) Math 0.08 0.08∗ 0.18∗∗∗ 0.19∗∗∗ 0.18∗∗∗ 0.14∗∗∗ (0.07) (0.04) (0.06) (0.04) (0.03) (0.02) Abstract 0.05 0.05 0.05 0.05 0.05 0.03 (0.06) (0.05) (0.05) (0.04) (0.04) (0.04) Composite 0.08 0.09∗ 0.18∗∗∗ 0.19∗∗∗ 0.19∗∗∗ 0.14∗∗∗ (0.07) (0.05) (0.07) (0.04) (0.03) (0.02) New modules 0.18∗∗ 0.20∗∗∗ 0.19∗∗∗ 0.16∗∗∗ (0.07) (0.04) (0.04) (0.03) Conceptual 0.12∗∗ 0.14∗∗∗ 0.12∗∗∗ 0.10∗∗∗ (0.05) (0.04) (0.04) (0.04) Observations 3,499 3,499 3,498 3,498 3,498 3,498

Columns 1-2 use baseline data and show the difference between treatment and control (Column 1), and the difference taking into account the randomization design—i.e., including “pair ” fixed effects—(Column 2). Columns 3-6 use May/June 2017 data and show the difference between treatment and control (Column 3) in test scores, the difference taking into account the randomization design—i.e., including “pair ” fixed effects—(Column 4), the difference taking into account other student and school controls (Column 5), and the difference using an ANCOVA style specification that controls for baseline test scores (Column 6). ∗ p < 0.10, ∗∗ p < 0.05, ∗∗∗ p < 0.01

To ease any concerns that differences in pre-treatment student ability drive the difference on test scores after one year, we estimate the treatment effect separately for students tested during the first and the second half of baseline field work (see Figure A.2). As discussed above, the imbalance in baseline test scores is only apparent for students tested later at baseline. Yet the difference in test scores at the oneyear follow-up is almost identical regardless of which students are included in the sample. Mechanically, the treatment effects using an ANCOVA style specification become smaller for students tested later at baseline.

An important concern when interpreting these results is whether they represent real gains in learning or better test-taking skills resulting from “teaching to the test”. We show suggestive evidence that these results represent real gains. First, the treatment effect over new modules that were not in the baseline

25