Impact

The figure above shows the ATT (a measure of programme impact) on the proportion of Standard 3 pupils in the top and bottom performance bands for Kiswahili and maths (in percentage points) and, separately, the ATT detected on Standard 3 pupils’ scale score for Kiswahili and maths (in logits). It compares changes in EQUIP-T schools with changes in control schools between baseline and endline.

Positive impact on pupils’ performance in Kiswahili

There is strong evidence that EQUIP-T has increased the proportion of pupils in the top performance band and reduced the proportion of pupils in the bottom performance band for Kiswahili in programme schools. These results remain strong and significant across an array of estimation models and robustness checks. Pupils in EQUIP-T schools are found to be about five percentage points more likely to be in the top performance band and 18 percentage points less likely to be in the bottom performance band compared to the counterfactual situation. Similar to the impact assessments at midline, the counterfactual situation is not simply the absence of EQUIP-T treatment, but includes other, mainly training, interventions in control schools delivered by other programmes. Hence the treatment effect measures the additional effect of EQUIP-T over and above any other existing interventions taking place in control schools. For simplicity, the counterfactual situation is described as ‘the absence of EQUIP-T’ in the results that follow.

Given the results presented in the descriptive analysis in Figure 5 above, which indicates that 18% of pupils are in the Kiswahili top band and 16% are in the Kiswahili bottom band at endline, it can be inferred that in the absence of EQUIP-T the proportion in the top band would have been around 13%42, while the proportion in the bottom band would have been well over 30%. This estimated programme impact is illustrated in the chart below, alongside the overall descriptive trends in pupils’ Kiswahili skills found in programme schools.

The positive impact on Kiswahili is also confirmed by the strong evidence that EQUIP-T has increased the average scale score for Kiswahili in programme schools by 0.6 logits (0.5 SD). This impact analysis at endline compares positively to the midline results, which only showed a statistically significant reduction in the proportion of pupils in the bottom performance band, but no other impact on Kiswahili skills.

42 This figure (13%) is calculated by subtracting 5 percentage points, which is the estimated measure of impact attributable to EQUIP-T, from 18%, which is the proportion of pupils found to be in the Kiswahili top band at endline. The same calculation is employed as part of the narrative on impact used for all other impact indicators.

Positive impact on pupils’ performance in maths

There is strong evidence that EQUIP-T has had a positive impact on pupils’ maths skills. The programme has increased the proportion of pupils in the top performance band and reduced the proportion of pupils in the bottom performance band for maths in programme schools. Pupils in EQUIP-T schools are found to be three percentage points more likely to be in the top performance band and eight percentage points less likely to be in the bottom performance band compared to the counterfactual situation.

Pupils’ average maths score in programme schools is also higher by 0.5 logits (0.3 SD) than it would have been in the absence of the EQUIP-T intervention. On the basis of the descriptive findings in Figure 6 above, showing the distribution of pupil learning in maths, it can be inferred that in the absence of the EQUIP-T intervention the proportion of pupils in the top band would have been around 6%, whilst the proportion in the bottom band would have been around 17% (almost double the measured 9%). These results are significant across an array of estimation models and robustness checks, which gives confidence in the strength and reliability of the findings. This estimated programme impact is illustrated in the chart below, alongside the overall descriptive trends in pupils’ maths skills found in programme schools.

These results are highly significant in statistical terms, as indicated by the confidence intervals not overlapping with zero in the top figure in this box, and present a consistent picture across the different dimensions through which programme impact on maths skills was measured, including maths scale score and performance bands. EQUIP-T is pushing pupils upwards in terms of maths score, which translates into pupils being lifted out of the bottom performance band and more pupils falling within the top band, with improvements visible at both ends of the numeracy outcome. These findings differ greatly from the midline analysis, which showed no clear impact on pupils’ performance in maths. This seems to indicate that the programme needed the full length of the evaluation period (from baseline to endline) to achieve its impact.

Comment on impact size

A logical question which flows from this analysis of programme impact on early grade learning, is whether a 0.5 SD impact on Kiswahili scores, and a 0.3 SD impact on maths scores over four years, can be considered a low, moderate or large impact? It is very common in the education impact literature to cite impact sizes in SD units and to compare these across different impact evaluations. For example JPAL (2014) describes an effect size of 0.5 SD or more on student learning as ‘very large’. Another approach is to be more specific, and to find comparators from impact evaluations of programmes that are targeting early grade literacy and numeracy in similar contexts. For example, EQUIP-T’s impact results for Kiswahili are not dissimilar to the impact size of 0.3 SD found for Liberia’s Teacher Training Program II on oral reading fluency or to the results from Uganda’s School Health and Reading Program which found an estimated impact on oral reading fluency of between 0.2 SD and 1.2 SD for different districts (Gove et al, 2017). But there is a question of how valid such comparisons are.

Singh (2015a) highlighted a number of flaws with using SDs to compare effect sizes across impact evaluations in education. One key problem is that a SD is a measure of dispersion, and this is not the same in different samples. This means that an intervention delivering the same absolute gain in learning would look more effective (have a higher effect size in SDs) in a context of a wide spread in test scores than in one with a narrow spread of test scores. Differences in test design, scoring, and analysis methods can also greatly affect the level of impact expressed in SD units. The message from this critique is that comparing effect sizes in SDs across different studies is not a very reliable approach. (For a more detailed discussion see Volume II Chapter 4 Section 4.6).

The impact of EQUIP-T is perhaps best understood by reflecting on the local context, on the increased share of pupils that are achieving at higher band levels as a result of EQUIP-T, and the estimated additional number of children that are reaching the required curriculum standards.

3.3.3 Why have there been gains in pupil learning overall, and what are the most likely explanations for EQUIP-T’s impact?

The overall trend in learning achievement in Kiswahili and maths in the programme schools over the four years can be summed up as improving between 2014 and 2016, and then increasing only modestly (Kiswahili) or remaining stable (maths) between 2016 and 2018. Only some of the gains made over the period can be attributed to EQUIP-T, in line with the findings in Box 6, while the rest are at least partly explained by common trends experienced across the country.

At the time of the midline report, the findings from a nationally representative study (3Rs-EGRA and 3Rs-EGMA43), which uses similar instruments to the tests used in the impact evaluation, found significant learning gains in Kiswahili skills, and in some (but not all) maths skills across the country over the 2014–2016 period for early grade pupils. Broadly in line with this national trend, estimates from the impact evaluation control sample also reveal significant improvement in both Kiswahili and maths skills for pupils.44

The midline report suggested that the national trend in early grade learning improvement between 2014 and 2016 was likely to be related to the introduction of a new curriculum for Standards 1 and 2, which drastically reduced the number of subjects being taught, and markedly increased the prescribed number of instructional hours per week for Kiswahili in particular, but also maths. The new curriculum is competency-based, and it requires teachers to use a phonics approach to teaching reading, which was new to many teachers in Tanzania. The other national factor which emerged from the midline research was hapa kazu tu (interpreted as ‘just work’) the slogan introduced by the then new government, which was credited by respondents across the midline qualitative work with influencing teachers’ work ethic, and also increasing the level of oversight and monitoring of schools and teachers.

On top of the improvement in learning achievement that EQUIP-T programme schools had in common with other schools in the country over the 2014–2016 period, EQUIP-T had an additional positive impact on the achievement of the lowest performing pupils in Kiswahili (but not in maths). To explain this, the midline evidence pointed to the emphasis of early programme implementation on in-service training for teachers in Kiswahili, which had high participation rates among early grade teachers. Consistent with expectations in regard to the training, some important aspects of teachers’ classroom performance in programme schools also improved significantly over the period, notably the use of inclusive practices. EQUIP-T has also had a positive impact on reducing teachers’ absenteeism from the classroom, and thus increasing actual instructional hours.

43 Research Triangle International (RTI) (2014) and RTI (2016).

44 The control estimates are unweighted, because of the way the control group was selected, and cannot be directly compared with trends in the treatment group to infer impact (as explained in Volume II, Chapter 4, on impact estimation methods).

Turning to the midline to endline period, from 2016 to 2018, results from the national 3Rs-EGRA and 3Rs-EGMA survey, conducted in late 2017, reveal significantly poorer national performance since 2016 in Kiswahili and maths skills among early grade pupils.45 Both average Kiswahili reading speed and average score in addition and subtraction problems fell back to baseline levels. Consistent with this trend, there was a significant fall in Kiswahili and maths skills for pupils in the impact evaluation’s control schools between midline and endline. The evidence suggests that EQUIP-T has helped to prevent a similar learning decline in its programme schools.

The pattern of EQUIP-T impact on early grade learning achievement over time, set out in Box 6, is consistent with the staged implementation of the teacher performance component of EQUIP-T. It is clear from the structure of the programme theory of change that this component, which principally aims to improve teaching skills in regard to the early grade curriculum competences, is the most direct link to improving pupil learning. The other components provide supporting interventions to help deliver changes at classroom level, and, to a lesser extent, to mitigate some of the demandside barriers.

The impact of EQUIP-T on pupils’ Kiswahili skills was only detectable for the lowest performing pupils at midline, while by endline there was impact on pupils with weaker and stronger skills, and on average scores. EQUIP-T’s in-service training for early grade teachers on Kiswahili was largely completed between baseline and midline, but there was some continuation into the midline to endline period to complete the modules. The emphasis of the in-service model is on continuous professional development via school-based sessions and classroom practice (with peer support and feedback) in order to put theory learnt in the modules into practice. The results from the endline survey, described in detail in Chapter 4, suggest that schools are sustaining the schoolbased sessions over time, albeit not quite as intensively as recommended. Participation in the inservice training among early grade teachers is also reasonably high.

At midline, there was no detectable impact of EQUIP-T on early grade pupils’ maths skills, yet by endline the results show impact on pupils with weaker and stronger maths skills, and on average scores. This is logical when considering the timing of the roll-out of EQUIP-T’s maths in-service training for early grade teachers, which only started after midline. All nine early grade maths modules were covered by the teacher in-service training programme after midline.

While the impact of EQUIP-T on early grade learning is consistent with the roll-out of the key teacher-related interventions, the results from the endline lesson observations in treatment schools paint a mixed picture of improved and worsening teaching practices since midline. This may not be inconsistent with the modest growth (Kiswahili) or no change (maths) in learning achievement seen in treatment schools over the same period, considering the huge growth in class sizes, in particular, which makes some types of participatory teaching practices very challenging. It is notable that the significant gains in inclusive teaching practices seen between baseline and midline in programme schools were sustained at endline, despite the sharp increase in classroom overcrowding.

3.4 Trends in early grade learning gaps by gender in programme areas

Relative to boys, girls have improved their Kiswahili and maths skills at a faster rate over the last four years, and girls now significantly outperform boys in Kiswahili At baseline there was no significant difference in average scale scores in Kiswahili by gender, but by endline there is a strongly significant gap in favour of girls. This growing performance advantage for girls in Kiswahili is also reflected in Table 4, which shows that by endline the share of girls who fall in the top performance band is eight percentage points higher than the share of boys achieving at this top level. This is mirrored at the bottom end of Kiswahili performance, where the share of girls achieving in the lowest band is eight percentage points lower than the share of boys falling in the same band. At baseline there was no significant difference in the share of pupils achieving at the top and bottom Kiswahili band levels by gender, so there has been a very marked shift over time.

Table 4: Proportion of pupils in bottom and top performance bands for Kiswahili and maths at baseline, midline, and endline by gender (%) (trends in programme areas)

Sources: Evaluation baseline, midline, and endline surveys (Standard 3 pupil tests).

Notes: (1) Asterisks indicate statistical significance levels: *** p<0.01, ** p<0.05, * p<0.1.

Girls started out at baseline performing significantly worse than boys in maths on average, and while this gap in average scale scores has persisted, it narrowed by midline but then did not change further by endline. In line with this overall trend, Table 4 shows that the share of girls falling into the bottom performance band at baseline was six percentage points higher than the comparative share of boys (a significant difference), but this gender disparity had been eliminated (no significant difference) by endline. This is not the case for top performers, where the gender disparity from baseline is still evident at endline. The share of boys achieving at the top performance level is 3–4 percentage points higher than the comparative share of girls at both baseline and endline, although by endline this difference is only weakly significant.

National trends in early grade learning disparities by gender between baseline and midline, based on the 3Rs-EGRA surveys (discussed above), are consistent with trends, over the same period, in the programme schools for Kiswahili, and to some extent for maths. The national results show girls gaining a significant performance advantage over boys in Kiswahili, in line with the trend in programme schools. For maths, while both the national results and the evaluation in programme schools reported boys significantly outperforming girls at baseline, the disparity had been eliminated nationally by midline, while there was still a persistent, but narrower, gap for pupils in programme areas. Common trends across the country may well explain at least some of the movement in gender gaps in early grade learning over the first two years of the programme. As yet, there is no further nationally available data on early grade learning trends by gender covering the second two years of the programme.

Setting national trends aside, there is some evidence that EQUIP-T has contributed to the comparatively larger gains in the performance of girls compared to boys since baseline, in line with programme goals. Teachers’ interactions with pupils during lessons became significantly more gender-balanced between baseline and midline, and this improvement has persisted to endline, despite the large increase in class sizes, which presents substantial challenges for effective teaching. While gender-responsive pedagogy was part of the broader in-service training for early grade teachers during the first two years of the programme, a specific gender-responsive pedagogy module was delivered after midline. A high proportion of early grade teachers report having completed this module.

3.5 Trends in early grade learning gaps by home language in programme areas

Disparities in learning achievement that favour pupils who speak Kiswahili (the language of instruction) at home, compared to those who do not, are large and strongly significant across all three survey rounds. Pupils who come from homes where Kiswahili is not the main language spoken have far lower average scale scores in both Kiswahili and maths than pupils who speak the language of instruction at home.

There has been considerable progress in narrowing the home language learning gap in Kiswahili since baseline. The gap in average scale scores for Kiswahili by home language fell by more than 50% between baseline and endline, following a consistent downward trend. In line with this finding, Table 5 shows that while the home language advantage is still significant in terms of the share of pupils falling into the lowest performance band, the gap has narrowed from 18 percentage points at baseline (44% vs 26%) to 10 percentage points by endline (18% vs 8% at endline). For the best performing pupils, contrary to the situation at baseline, there is no longer a significant difference in the share of pupils achieving in the top performance band by home language.

There has been inconsistent, and more modest, progress in narrowing the home language learning gap in maths since baseline. The gap in average scale scores for maths grew between baseline and midline, but then this gap fell to below the baseline estimate by endline. Sizeable and significant differences in the share of pupils from different language backgrounds falling into the top and bottom performance bands are still evident at endline (Table 5), which clearly illustrates the persistent disparity in maths performance.

Sources: Evaluation baseline, midline, and endline surveys (Standard 3 pupil tests).

Notes: (1) Asterisks indicate statistical significance levels: *** p<0.01, ** p<0.05, * p<0.1.

EQUIP-T regions are not typical of the national situation in terms of linguistic diversity. As discussed in Section 3.2.2, 80% of Standard 3 pupils in programme districts come from homes where Kiswahili is not the main language spoken. By contrast, the national prevalence of nonKiswahili speaking households is about 33% (Uwezo, 2011). Hence the challenge of supporting pupils whose first language is not Kiswahili is much greater in the programme regions than in many other parts of Tanzania.

EQUIP-T’s main strategy for narrowing home language gaps in attainment is strengthening school readiness for pre-school-age children, partly via the SRP, and, under the extended programme, via a broader set of activities, to support expansion of Government pre-primary schooling. It is beyond the scope of this evaluation to assess the effectiveness of the SRP in supporting children who do not speak Kiswahili as a mother tongue to acquire basic Kiswahili language skills earlier than they otherwise would have done. Coverage of SRP is also a factor, as SRP centres only appear to be present in about 50% of school communities in programme areas (Box 5).

Having teachers who can speak the same language as pupils who speak other local languages at home (not Kiswahili) seems likely to be a supportive factor for this disadvantaged group. The prevalence of teachers with these linguistic skills increased significantly between midline and endline. Lesson observations, however, found that teachers were rarely observed providing extra support to pupils who do not speak Kiswahili.

3.6 Trends in learning gaps by household poverty status in programme areas

Pupils from richer backgrounds have continued to outperform pupils from poorer backgrounds since baseline, and while the gap has narrowed modestly for Kiswahili, it has grown for maths. The difference in average scale scores for Kiswahili between pupils from different economic backgrounds has narrowed slightly between baseline and endline, while the comparable difference in maths has increased over the four years. Regardless of the trend, though, the absolute disparities in average scale scores are still large.