Improving comparability of SDI surveys over time

During the past decade, SDI surveys have evolved steadily. Each survey has been tailored to address specific policy and research questions in its setting of implementation. Although such specificity means that the surveys are well designed to respond to pressing national policy concerns, it also poses challenges for the direct comparability of some surveys from the same country over time. In addition, the SDI surveys are designed to provide just-in-time findings in fragile and conflict-affected settings, which means that logistical and safety considerations may also affect comparability. Experiences from multiple surveys in the same country are now being used to inform the design of future repeated surveys. This section outlines challenges, solutions, and lessons learned from repeated SDI surveys in Kenya, Mozambique, and Tanzania.7

The methods used in the Tanzania SDI education surveys in 2014 and 2016 allow for direct comparisons across the two years. Over this period, student learning outcomes for fourth-grade students, as measured by the English, math, and Swahili SDI tests, improved. The mean student test score across all subjects improved by an average of 11 percentage points. Other SDI indicators for which a significant change was observed in this two-year period included the share of students with pens, pencils, and an exercise book (increased from 84 percent to 92 percent) and the proportion of schools with a functioning blackboard (increased from 74 percent to 83 percent). Interestingly, the proportion of students with a textbook declined significantly (from 25 percent to 19 percent), while the observed student-teacher ratio increased (from 43 percent to 47 percent), perhaps indicating that system resources have not kept pace with enrollment. Smaller changes were observed in other SDI indicators, such as teachers’ absence from the classroom (declined by 5 percentage points) and teachers’ subject knowledge in language (deteriorated slightly) and mathematics (increased significantly, but only on certain tasks, such as Venn diagram and graph interpretation). Pedagogical knowledge also improved modestly. Trends in basic infrastructure showed no noticeable improvement.

The 2014 and 2016 education surveys for Tanzania are so far the only SDI surveys that track a complete panel of schools over time. The direct comparability of data is the result of a careful panel sampling strategy and the use of identical survey tools. These methodological and logistical choices allow for accurate measurement of changes over time to track progress in education outcomes and process indicators. Nonetheless, the trends observed in Tanzania also suggest that, whereas certain indicators can be improved substantially in a two-year period, others may take longer. In addition, the measurement error associated with some indicators may make direct comparisons harder to interpret. For instance, many factors may affect the performance of students and

teachers in SDI assessments, including how they are feeling on the day of the exam, whether they had breakfast, their environment, and even the weather. In contrast, directly observing whether there is a blackboard or bathroom in a school is subject to much less measurement error. In the future, teams interested in carrying out repeated SDI surveys will need to identify a relevant interval between surveys, consider specific project needs, and understand which indicators are likely to show meaningful changes over the defined period.

Mozambique implemented repeated SDI education surveys in 2014 and 2018. Over this longer interval, researchers observed significant changes in both student knowledge (increased from 21 percent to 31 percent of correct answers) and teacher knowledge (increased from 31 percent to 42 percent of correct answers). At the same time, whereas teacher absence was greatly reduced (from 45 percent to 30 percent), the percentage of schools with minimum equipment dropped significantly (from 77 percent to 68 percent). Differences in all of the remaining SDI indicators are less than 2 percentage points across both years. The group of variables that exhibit significant differences is similar to that in Tanzania. Some level of caution is warranted in directly comparing the two sets of SDI results from 2014 and 2018 in Mozambique, because logistical factors resulted in important differences in sampling. Among other limitations, ongoing conflict made some regions impossible to survey in 2018, so these areas were excluded from the sampling process. Additionally, in 2018, the survey team faced several logistical constraints that prevented enumerators from reaching all schools in the sample and from carrying out the second (unannounced) visit in many others. Moreover, fieldwork for the 2014 survey started approximately three months earlier in the calendar year than the fieldwork in 2018. For this reason, data were collected at different moments of the school year, which could affect (1) student learning, because students in one survey had more schooling, and (2) absence indicators, because both teachers and students may be more likely to be absent as the school year draws to an end, for example. Although the final analysis employed adjusted weights and explored differences using various sensitivity analyses and scenarios, the magnitude of the differences observed could be attributed partially to some of the above circumstances. For instance, one sensitivity analysis suggested that teacher absence for 2018 was between 30 percent and 40 percent, which would amount to a reduction of anywhere between 5 percentage points and 15 percentage points, relative to 2014. Lessons learned from the repeated Mozambique surveys underscore the challenges of maintaining comparability over time, particularly during periods of conflict, and emphasize the need for additional research on the role of seasonality in education survey findings.

In Kenya, an SDI health survey was conducted in both 2012 and 2018. As discussed in the 2018 report for Kenya (World Bank and Government of Kenya 2019), important differences between the 2012 and 2018 survey rounds may