Impact Evaluation of Education Quality Improvement Programme in Tanzania: Endline Quantitative Technical Report, Volume I

Below this high-level results chain sit the expected results chains for each of the programme components, running from inputs to outputs to intermediate outcomes. The four chapters which follow the current chapter present quantitative evidence related to these component results chains. As noted in Chapter 2 (Section 2.3) the quantitative evidence is more comprehensive for the teacher performance and SLM components, than for the others.29 It is important to bear this in mind, when reading this chapter, which draws findings from the component analyses to help explain programme impact.

The main objective of this chapter is to answer the key evaluation question of whether EQUIP-T has had an impact on early grade learning achievement, after four years of implementation. It also seeks to explain the most likely reasons for any impact, drawing on descriptive evidence related to the theory of change that is discussed in detail in the component chapters (Chapters 4 to 7). The other key results relate to trends in overall levels of early grade learning in the programme regions, and changes (if any) in learning disparities by gender, poverty status of the pupils’ household, and main language spoken at home. This is supplemented by results discussed in Volume II Chapter 5 covering trends in learning outcomes in programme schools for various sub-populations of EQUIP-T beneficiaries (girls, boys, pupils from poorer and richer homes, pupils whose home language is or is not Kiswahili) over the four year period. It also includes a description of changes in learning outcomes between midline and endline for pupils in programme schools with disabilities. Given EQUIP-T’s increasing focus on inclusive education in the extension period up to January 2020, these results should be of particular interest to the MA.

To give some context to the results, Box 4 contains information on the scale of expected change in learning achievement in the programme regions over the programme lifetime.

Box 4: Learning achievement targets from programme logframes, and original intervention summary

Source 1: Original Intervention Summary for EQUIP-T (DFID, 2013, p. 38)

Indicator: proportion of early standard pupils with basic literacy skills

Target: to more than double the value of this indicator over the original programme period (2014 to 2018)

Source 2: EQUIP-T logframes (EQUIP-T MA, various years)1

Indicator: proportion of Standard 3 pupils achieving Standard 2 literacy and numeracy skills in the five original programme regions (those covered by the impact evaluation)

Baseline: 12% literacy, 4% numeracy

Target (2018): 21% literacy and 10% numeracy, with no gender disparity

Target (2019): 25% literacy and 10% numeracy, with no gender disparity

Note (1): The target for 2019 comes from the latest logframe made available to the evaluation team (dated November 2017), but as there is no target for 2018, this was taken from an earlier logframe (November 2015).

This chapter is structured into six further sections, starting with Section 3.2 which presents trends in pupil background conditions since baseline, followed by trends in early grade learning results in Section 3.3. This section also contains the impact estimates, which demonstrate whether part of the gains in learning achievement can be robustly attributed to the EQUIP-T programme or not. All impact estimates in this report (which are based on a comparison of EQUIP-T programme schools against a counterfactual set of control schools) are presented in shaded boxes, in the interests of clarity. These key results are followed by a brief summary of the most likely reasons for any observed impact based on the main findings from the subsequent EQUIP-T component chapters (Chapters 4 to 7).

29 This is particularly true for Component 3A, district planning and management, where only the WEOs’ support to school is assessed, and hence this report omits the majority of the interventions under this component.

Sections 3 4 to 3 6 discuss changes in learning gaps by gender, home language, and household poverty in the programme schools since baseline. The discussion of these descriptive trends is complemented by summary evidence from the component chapters which point to likely reasons for some of the changes observed. The final section, 3.7, summarises the key points regarding changes in pupil learning

3.2 Trends in pupil backgrounds and home support for education in treatment areas

As highlighted in Figure 3 above, pupil backgrounds and home support factors can inhibit (or enhance) the link in the theory of change between improving schooling and achieving the desired learning gains.

Before presenting trends in learning achievement of Standard 3 pupils since baseline, it is first useful to see if there are any changes in the profile of Standard 3 pupils over time, as these could be contributing to changes in learning levels. Changes in pupils’ profile cover personal characteristics, such as gender and age; economic circumstances; support for education at home; and the demand for schooling. It is worth noting that the majority of the current cohort of Standard 3 pupils in 2018 entered school in Standard 1 in 2016, shortly after the fee-free basic education announcement, so it might be expected that the profile of this cohort differs from previous cohorts.

EQUIP-T is seeking to directly influence some of the background factors discussed below, mainly via its community participation and inclusive education components (4A and 4B, respectively) (discussed in Chapters 6 and 7). In addition, EQUIP-T piloted, and has subsequently expanded, a community-based short pre-school programme, called the SRP. This is now part of component 1B (improving rural access to quality pre- and early primary) under the programme extension. Although this evaluation does not cover this element, Box 5 below explains more about the intervention and summarises findings from EQUIP-T’s commissioned research

3.2.1 Has the personal and economic profile of Standard 3 pupils changed since baseline?

The near equal balance between boys and girls attending school in Standard 3 (52% girls at endline), as well as their average age of 10 years, has not changed significantly since baseline. The distribution of pupils’ ages has changed significantly, however, such that the share of over-age pupils (those greater than 11 years) increased from 36% to 46%. This appears to be due to a greater share of over-age pupils entering school for the first time three years earlier, shortly after the fee-free policy announcement, rather than due to high repetition rates. Only 1% of Standard 3 pupils report that they are repeating the standard. Over-age enrolment is problematic for a number of reasons: it can affect demand for schooling (premature dropout), as well as the effective delivery of education and the efficiency of the system.30 One of EQUIP-T’s logframe targets is to reduce the share of over-age children entering Standard 1 in its programme regions from 20% in 2014 to 12% by 2018. Of course, this may be a temporary problem if the fee-free policy caused a one-off surge in older children returning to school or entering for the first time.

Pupils’ economic circumstances have worsened to some extent since baseline. Close to 40% of Standard 3 pupils come from households that are predicted to fall below the national poverty line,31 up from 33% at baseline (Figure 4), and this change is weakly significant. The programme deliberately selected remote and economically disadvantaged districts into the programme, and so this extremely high share of pupils coming from low socioeconomic backgrounds was expected from the outset. Figure 4 also illustrates how the worsening economic circumstances affect pupils directly. Particularly striking is the large and strongly significant increase in the share of pupils doing unpaid work outside the household, which reached 13% by endline.32 There is also a drop of five percentage points in the already low share of pupils who ate some food before school, on the day of the survey, to under 30%, and this change is weakly significant. Being hungry at school can affect the ability of pupils to concentrate on learning (Pivak et al, 2012, and Hoyland et al, 2009), and about 10% of schools have a feeding programme to try to mitigate this (see Chapter 7).

30 See, for example, evidence from studies which find that over-age pupils are more likely to repeat grades and drop out of school before completing a cycle (UN Economic, Scientific and Cultural Organization (UNESCO), 2007.

3.2.2 Has the home support for education and demand for schooling for Standard 3 pupils changed since baseline?

The vast majority of pupils (81% at endline) do not speak Kiswahili, the language of instruction, as their main language at home, and this share has not changed significantly since baseline. 33 While this is an important environmental factor, and a marker of disparity in learning achievement (see Section 3.5), there are some signs of positive change in home support for education. Just under half of pupils’ homes have books, newspapers, or other reading materials available at endline, a significant and large gain of nine percentage points since midline. Another positive change is in the availability of someone at home to help pupils with homework. Some 70% of pupils have support for homework now, significantly up from 63% at midline. Direct support for reading, however, is unchanged since midline. A sizeable minority of pupils never read out loud to someone at home (about 20%) or never have someone at home read out loud to them (about 35%).

31 See Volume II, Chapter 3 for further details on how this was measured.

32 The survey did not probe for examples. In rural Tanzania there is a communal culture of helping each other with activities such as collecting water, firewood, taking care of younger children, and grazing cattle.

33 This should not be taken to mean that this group of pupils do not understand Kiswahili, but that they have less opportunity to practise speaking Kiswahili than their peers who come from homes where Kiswahili is the main language.

Nearly three-quarters of Standard 3 pupils say they attended pre-school, and there is no significant change in the reported rate since midline. 34 Of those with pre-school experience, about 80% attended a Government pre-school, while 20% attended a nursery or kindergarten. Only one pupil reported attending an EQUIP-T SRP, which most likely would have been when the SRP was being piloted in 2015 (see Box 5 for more details). Both at midline and endline, parents reported that 5% of Standard 3 pupils attend extra paid classes, so having this extra support outside the home is fairly rare.

Box 5: EQUIP-T’s SRP SRP implementation summary

The SRP started in October 2015 with a 12-week programme in 25% of programme districts. This included 1,000 centres and 50,000 enrolled children. The SRP is based in community buildings and is delivered by community teaching assistants (now called SRP facilitators), who are trained in a five-day programme prior to teaching starting. To support this, the programme developed an SRP competency framework to guide learning (focusing on building competencies that support rapid learning), including 12 SRP story-books. The framework is based on active pedagogy and a story-based approach. SRP facilitators further learn in training to make and use locally produced teaching aids. The number of SRP centres expanded to 2,744 in October 2016, when another 12-week programme was delivered. In 2017, the duration of the SRP was expanded to 16 weeks.

Three main aims of SRP

Access: To provide some pre-schooling to children who are out of school and likely will not attend preschool at all.

Equity: To target areas where there is no existing pre-school (more remote and typically more disadvantaged areas) and where children do not speak Kiswahili (since these children perform much worse in the early grades of primary school and their need for school readiness interventions is greater). The programme has a focus on inclusive pedagogy, particularly with regard to addressing gender imbalance. Quality: By introducing an active pedagogy using a story-based approach, the theory is that this will enable the children to get prepared for school faster (i.e. this pedagogy will be more effective and more engaging for young children).

Findings on SRP provision from the impact evaluation quantitative surveys

Between the midline and endline surveys, the share of head teachers in programme schools reporting that there is an SRP centre in the community increased significantly from 18% to 51%, in line with the scale-up from a pilot programme. This highlights both impressive growth in SRP centres over a short period, but also the limited coverage of the SRP, which is not present in just under half of the communities serving programme schools.

Findings from EQUIP-T’s SRP research study

Results from an international development and early learning assessment (IDELA)1 conducted in March 2017 in the seven EQUIP-T regions, found that Standard 1 pupils who had attended an SRP had significantly stronger skills than those who had no access to early childhood education, and moderately stronger skills than those who had attended formal pre-school. These findings are based on a sample of 1,119 Standard 1 pupils from 42 schools in 14 districts across the seven regions. Within each school, pupils were grouped according to their pre-school experience, and then sampled randomly. The study does not control for background characteristics of pupils, so the differences observed in performance cannot be interpreted as being caused by the different types of pre-school exposure.

Source: EQUIP-T annual monitoring reports (various years), and interviews with EQUIP-T programme staff in January 2018. EQUIP-T MA (2017b). Note: (1) IDELA was developed by Save the Children and contextualised with technical assistance from the Tanzania Institute of Education, the University of Dodoma, and Aga Khan.

34 This was not collected at baseline. At endline (only), parents were also asked whether their child had attended preschool. The parental reported rate of pre-school attendance for Standard 3 pupils was slightly higher, at 78%.