Impact Evaluation of Education Quality Improvement Programme in Tanzania: Endline Quantitative Technical Report, Volume I

As an indicator of the vastly increased demand for schooling since baseline, Table 3 shows that mean enrolment in Standard 3 grew from 65 to 95 pupils from baseline to endline, a strongly significant change of nearly 50%. At the same time, mean enrolment across all standards expanded by 111 pupils on average, a 23% expansion, demonstrating that growth is not confined to Standard 3 (the cohort that entered school shortly after the fee-free education policy). High enrolment growth puts pressure on the schooling system, and the near parallel growth in Standard 3 enrolment and class sizes over the period suggests that the system has not been able to increase basic resources (teachers and classrooms) to cope. The mean class size in Standard 3 at endline is 88 pupils.

Parents are sending their children to school somewhat more frequently than at baseline, but pupil absenteeism is still high. For pupils in Standards 1 to 3, absenteeism fell significantly from 34% to 28% between baseline and endline, and there has been a slightly larger decline for girls than boys (Table 3).

Table

Sources: Evaluation baseline, midline, and endline surveys (school records and head count instruments).

Notes: (1) Asterisks indicate statistical significance levels: *** p<0.01, **

3.2.3 Summary of trends in the profile of Standard 3 pupils

Compared with baseline, the endline cohort of Standard 3 pupils are more likely to be over-age; their economic circumstances are worse to some extent; and they are much more likely to be doing unpaid work outside the household on top of their schooling. On the other hand, there is some positive change in support for education at home, and there is vastly increased demand for education, although this has led to very large class sizes. At the same time, pupil absenteeism has fallen slightly since baseline

3.3 Trends in early grade learning and impact of EQUIP-T

The purpose of testing Standard 3 pupils is to measure changes in learning achievement over time, as a means of assessing the overall impact of EQUIP-T in relation to its goal. As in previous rounds of the evaluation, the estimates of pupil learning achievement in Kiswahili and maths are measured on scales that are directly linked to early grade curriculum competencies. This provides insight into whether pupils are performing at, above, or below the curriculum level expected, and gives information on the skills different groups of pupils have (Volume II, Annex E, contains more details on the method of Rasch analysis used to estimate the scale scores35). The current chapter also presents selected ‘raw’ score results, including Kiswahili reading speed, because similar indicators are used to monitor other large-scale programmes in Tanzania. An analysis of trends in raw score pupil learning indicators is given in Volume II, Chapter 5.

The evaluation uses the same adapted-Early Grade Reading Assessment (EGRA) and adaptedEarly Grade Mathematics Assessment (EGMA) instruments as at baseline and midline, to test Standard 3 pupils on Standards 1- and 2-level skills. Although the Standards 1 and 2 curricula changed between the baseline and midline, the instruments are competency-based and, with some assumptions, it is reasonable to consider them valid for comparative purposes (see Volume II, Annex E, for further explanation).

3.3.1 How has early grade pupil learning changed overall in treatment areas?

Consistent with the approach taken at baseline and midline, pupils have been estimated to be achieving at the level of one of five curriculum-linked performance bands: Band 0: below Standard 1 level; Band 1E: emerging Standard 1 level; Band 1A: achieving Standard 1 level; Band 2E: emerging Standard 2 level; and Band 2A achieving Standard 2 level. Pupils who are ‘achieving at band level’ are more likely than not to be able to demonstrate the skills required for the relevant performance band. A list of competencies linked to each band is given in Volume II, Annex E 36 37

3.3.1.1 Kiswahili results in treatment areas

Pupils’ literacy skills in Kiswahili have significantly improved on average between baseline and endline, and the improvement is particularly strong for the poorest performing pupils The bulk of the skills improvement happened between baseline and midline, with a strongly significant rise in average scale scores over the first two years of the programme of 0.9 logits (0.8 SD). There was a further gain between midline and endline, but it is smaller at 0.2 logits (0.2 SD) and only weakly significant.

Figure 5 illustrates the dramatic positive shift in the distribution of pupils across the performance bands, between baseline and endline. It is clear that pupils have, on average, moved up the performance bands over the four years and there is a strongly significant decline in the share of pupils falling into the bottom performance band from 39% to 16%. There is also significant change at the top of the distribution, with the proportion of pupils performing at either emerging Standard 2 level or achieving Standard 2 level rising from 36% to 50% over the period. By endline the share of pupils achieving in the top band (18%) is close to the logframe target for 2018 of 21%.

Put in terms of the absolute number of pupils achieving at each level, the change over time is even more marked because of the sharp growth in Standard 3 enrolment between baseline and endline. Estimates of the number of pupils in government primary schools achieving at Standard 2 level Kiswahili in the 17 districts represented by the evaluation survey sample, reveal an increase from

35 Rasch analysis uses a probabilistic model of item response that generates estimates of pupil scores on an interval scale that capture more precisely differences in performance by weighting items by difficulty, compared to a simple raw score analysis

36 For a minority of items (questions) in the tests and hence the related competencies, there have been changes to their estimated location on the scale over time, these are explained in Volume II, Annex E.

37 The band boundaries scores are the same as at the baseline, for reasons which are explained in Volume II, Annex E.

16,510 pupils at baseline to 35,596 pupils at endline more than a two-fold increase over the four years. Chapter 5 in Volume II contains the details of this analysis.

While the gain in early grade Kiswahili skills for pupils over four years is remarkable, it is important not to lose sight of the extent to which pupils are still behind curriculum expectations. About half of Standard 3 pupils are achieving at Standard 1 level or below, and are thus at least one year behind in skills acquisition, and therefore need further support to catch up.

Achieving sufficient reading fluency for reading comprehension is one of the key differences between Standard 2 and Standard 1 curriculum standards. The Government’s benchmark for Standard 2 pupils is to read at a speed of 50 words per minute (in line with international research on the minimum rate needed for comprehension (Abadzi, 2006)).38 Pupils in programme schools read a simple story passage at a rate of 21 words per minute on average at baseline. This increased to 30 words per minute at midline, but then did not change significantly in the two years between midline and endline (see Volume II, Chapter 5). While the overall gain between baseline and endline is significant and sizeable, it is still far short of the benchmark

That the gap between current achievement and curricular expectations remains large for such a high proportion of pupils raises the question of whether the expected achievement levels at each Standard may themselves be contributing to poorer performance. Pritchett and Beatty (2012) argue, using data from three studies in South Asia and Africa, that low learning gains over time are partly a result of curricular paces moving faster than the pace of learning (children who are left behind find it increasingly difficult to catch up as the curricula moves on), and that by slowing curricular delivery, cumulative learning can increase.39

38 MOEST and PO-RALG, 2018b, p153.

39 The paper’s findings are based on a simulation model of learning related to the gap between pupil skills and instructional level. The results of the simulations also replicate existing experimental findings.

3.3.1.2 Maths results in treatment areas

Pupils’ numeracy skills significantly improved on average between baseline and endline, but the gain is not as marked as for Kiswahili skills. The significant gain in average scale scores for maths between baseline and midline of 0.4 logits (0.2 SD) has been maintained to endline, but there was no additional significant gain in the last two years.

Figure 6: Distribution of pupil learning in maths by performance band in treatment areas, baseline, midline, endline (%) (trends in programme areas)

Comparing the distribution of pupils by performance band over time, in Figure 6, there is a clear shift towards the top performance bands, away from the bottom bands, with the middle band similar in size to the baseline share. The most strongly significant changes over time occurred at the top and bottom of the distribution. The share of pupils with below emerging Standard 1 level skills (bottom band) fell by just under five percentage points, to reach 9% by endline. Mirroring this change, the share of pupils able to demonstrate Standard 2 level skills (top band) rose by five percentage points, to reach 9%, close to the logframe target for 2018 of 10%.

The positive change over time in maths performance is starker when considering the absolute number of pupils achieving at each level. In the 17 districts represented by the evaluation survey sample, estimates of the number of pupils in government primary schools achieving at Standard 2 level in maths increased from 6,046 pupils at baseline to 18,753 pupils at endline more than a three-fold gain. Chapter 5 in Volume II contains the details of this analysis.

Nonetheless echoing the point made at midline, with more than 60% of Standard 3 pupils estimated to be achieving at Standard 1 level or lower, it is clear that the majority of pupils will need further support to catch up with curriculum expectations in maths, and to mitigate against the risk of falling further behind.

The disbursement-linked indicator of early grade maths achievement in the Government’s EPforR program is the average percentage of correct answers on level 2 addition and subtraction questions obtained by pupils at the end of Standard 2.40 Standard 3 pupils in programme schools (four months into the school year) scored 25% on average at baseline on these types of questions. This increased to an average score of 30% at midline, but then did not change significantly in the two years between midline and endline (see Volume II, Chapter 5). Although the gain in numeracy skills is positive overall, it is far short of the Government’s benchmark of 80% achievement in this indicator of early grade maths skills.41

3.3.2 What is the impact of EQUIP-T on early grade pupil learning?

After four years of EQUIP-T programme interventions, the impact modelling finds strong evidence that the programme has had a positive impact on pupil learning in Kiswahili and in maths. The programme has helped pupils with both weaker and stronger skills in both subjects, and, as a result, has also had a positive impact on average scale scores. The details are in Box 6

Box 6: EQUIP-T impact on early grade pupil learning

40 Level 2 addition and subtraction questions are intended to test skills in the Standard 2 curriculum. 41 MOEST and PO-RALG 2018b, p.153.