Annex 6 - Sampling Frame and Sample Sizes (Quantitative Data Collection)

The starting point for learning outcomes in this research was the data collected as part of the midline evaluations of the GEC programs in Kenya and Nepal. This data, collected in July 2019, were the most recent before the school closures due to Covid-19 in 2020. This was our ‘before’ – and to estimate the ‘after’, we conducted primary data collection between February-March 2021 using the same learning assessment instruments used by the external evaluators in the same schools to estimate the learning loss. As such, we inherited the schools for the primary data collection rather than resampling, with our sampling frame being the list of schools at midline – with a view to collecting in all schools included in the midlines. We included both treatment and control schools to maximise our sampling power.

In Kenya, the midline external evaluation was conducted by Women Educational Researchers of Kenya. The midline dataset included a total of 8,942 observations with learning assessment data across 197 primary and 70 secondary schools.

Within the 70 secondary schools, there were 1,481 and 1,274 learning assessment cases for girls attending Form 1 and Form 2 of secondary schools. After excluding those who were ‘benchmarking’ schools – that is, had only a small number of girls tested in later grades, we sought access to 66 schools. Due to Covid restrictions, we could not access Kilifi region, which left us 52 schools. Of these, the data collection was conducted successfully in 50 schools – with issues accessing the other two schools due to Covid concerns from school leaders. Of these, 41 were previously treatment schools and 9 were previously control schools.

In Nepal, the midline external evaluation was conducted by Foundation for Development Management (FDM). The midline dataset included a total of 579 observations with learning assessment data across 47 1 secondary schools. 2 A sample of 45 secondary schools was selected for the primary data collection in 2021. 3 Of these, 30 schools were treatment, while the remaining 15 schools served as control during the midline evaluation.

In Kenya, a total of 2,313 girls were tested Post-Covid-19 compared to 2,289 girls at Midline across Forms 1 and 2 (the first two years of secondary education) across the 50 sampled schools The study aimed to replicate the sampled numbers from the Midline data in each school, yielding 46 pupils on average per school.

At Midline, 54% of pupils attended Form 1, while 46% of pupils attended Form 2. In Post-Covid-19, pupils were split equally between the forms as shown in Table 1 below. We find our sample to be relatively balanced, aside from more children reporting certain disabilities and speaking Kiswahili at home than at Midline (see Table 2Table 1 below).

In Nepal, a total of 1,685 girls across grades 8-12 were tested post-covid, while there were 577 girls tested at Midline across the 45 sampled schools. The remaining sample of girls (1,572 Post-Covid-19 and 464 Midline) are referred to as ‘cross-cohort’ sample. The cross-cohort sample consists of different girls, but a comparison is possible as a sample of girls attending the same grades at two different points in time is collected.

Alongside this sample, 113 of these girls tested at Midline were identified and tested again post-Covid-19 – these are referred to as a ‘panel cohort’ sample. These girls typically progressed two grades between the two data collection points.

The midline data collected included insufficient number of observations on grades 8 and 11, but because we were already in the schools, we decided to collect data on grade 8 girls (with a view to seeing how learning levels vary between years in the same cohort) – but only the sample of cross-cohort girls attending grades 9 and 10 are included in the analysis to answer RQb in Nepal. This is a cross-cohort sample of 1,046 and 432 girls in Post-Covid19 and Midline

As the sample was small in Nepal, the IE team aimed to increase the sample numbers in each school, yielding 12 pupils on average for each grade in each school instead of an average of 3-5 pupils at Midline.

For the midline sample, 59% were drawn from Grade 9, and 41% Grade 10 at Midline – this is slightly unbalanced, so we split the sample equally between the two forms in our data collection. This is shown in

Panel sample Cross-cohort sample

3