Impact Evaluation of Education Quality Improvement Programme in Tanzania: Endline Quantitative Technical Report,

The impact evaluation will also assess cost-effectiveness of the EQUIP-T programme and the fiscal affordability of rolling out EQUIP-T to regions and districts beyond the initial programme areas in a separate fiscal study (see OPM 2014a).

A.5 Evaluation questions

The original TOR specified key questions related to the OECD-DAC evaluation themes of relevance, effectiveness, impact and sustainability, to be answered by the impact evaluation. These questions are shown in Table 20, together with what the impact evaluation will measure, given the EQUIP-T programme design and changes to the TOR agreed with DFID.

Table 20 What the impact evaluation will measure

Original terms of reference evaluation questions

“Have the programme interventions targeted the most necessary, most economical and appropriate combination of interventions for improvements in the quality of education?”

Measured by the impact evaluation under agreed TOR

Partly, as the EQUIP-T components had already been determined by the MA during their inception phase and differential roll out of different EQUIP-T components was not deemed possible by DFID. The qualitative component will examine which EQUIP-T components were perceived to raise education quality.

“Has pupil-teacher ‘time on task’ been significantly increased in target schools?” Yes.

“Are better pedagogic practices that promote effective learning, demonstrably in place?” Yes.

“Have the target councils been able to increase learning outcomes for girls / boys, including disadvantaged children, beyond those more generally obtained in comparable areas?”

“Do councils have costed plans in place that are realistic both fiscally and institutionally for the long term maintenance of quality within schools including provision and quality of teachers, operations, inputs and maintenance of school infrastructure?”

Yes.

Partly if possible. The qualitative component will through the district level interviews attempt to collect information on the availability of costed plans, but not their quality, for the EQUIP-T programme councils (districts) selected as qualitative research sites.

“Improved education quality.” Yes.

“Improved teaching of early-grade reading and numeracy resulting in more children able to read with comprehension” and with curriculum appropriate numeracy skills.” Yes.

“Improved teaching of early-grade reading and numeracy resulting in more children able to read with comprehension” and with curriculum appropriate numeracy skills.” Yes.

“More time on task for primary school children, resulting in more children passing their end of primary school examinations” Yes.

“More girls able to make the transition to secondary school”.

No, as the EQUIP-T programme will focus on the early grades and impact of the programme, if any, on this outcome would be highly unlikely to be detectable within the life of the impact evaluation.

Pupil learning results should be disaggregated by gender. Yes.

The impact evaluation should examine impact for disabled children.

No, because the EQUIP-T programme does not contain any component or activities aimed at this particular group171

The impact evaluation should include poverty measures for pupils. Yes.

A.6 Revised purpose of the impact evaluation

Thus the impact evaluation will:

 Generate evidence on impact of EQUIP-T on learning outcomes for pupils in primary education, including any differential impacts for girls and boys;

 Provide evidence on the fiscal affordability of scaling up EQUIP-T beyond the initial EQUIP-T regions and districts (separate fiscal study);

 Assess perceptions of effectiveness of different EQUIP-T components through the qualitative research and explore possibilities to do so through the quantitative component; and

 Communicate evidence generated by the impact evaluation to policymakers and key education stakeholders, including DFID and MOEVT172 to promote accountability and lesson learning.

A.7 Changes to the impact evaluation design since the technical proposal

In addition to the reduction of scope of the TOR outlined above the following changes to the IE design compared to the technical proposal were made based on discussions with DFID during the inception phase and feedback from the first Reference Group meeting for the impact evaluation173

 Scope of impact evaluation expanded for the qualitative component to examine perceived EQUIP-T contributions to changes in relevant outcomes and outputs, to the extent possible within the scope of the IE;

 Quantitative fieldwork to start in March 2014 (first start date was October 2013, second start date was January 2014);

 Use of EGRA/EGMA style pupil learning assessments instead of UWEZO assessment testing 3,000 standard 3 pupils in Kiswahili (EGRA) and mathematics (EGMA);

 Test standard 3 pupils (Kiswahili and mathematics) instead of standard 2 and standard 5 pupils;

 Administer teacher development needs assessment (TDNA) to standards 1-3 (Kiswahili and mathematics) and 4-7 (mathematics) teachers instead of to standard 2 and standard 5 teachers;

 One standard 2 Kiswahili and one standard 2 mathematics lesson will be observed for each sample school instead of one standard 3 lesson and one standard 5 lesson;

 Replace pupil tracer survey to collect data for poverty measure by data collected at school level (from tested pupils’ parents);

171 MA-OPM and DFID-OPM correspondence December 2, 2013.

172 Prior to the new government which took office in November 2015, the Ministry had responsibility for vocational training and was known as the Ministry of Education and Vocational training (MOEVT).

173 The draft inception report was subsequently further revised based on comments received from the SEQAS review on March 7, 2014.

Impact Evaluation of Education Quality Improvement Programme in Tanzania: Endline Quantitative Technical Report, Volume I

 To obtain school sample: in the second stage, match control schools to treatment schools using PSM instead of random selection;

 17 EQUIP-T programme (treatment) districts in the five EQUIP-T regions covered by the impact evaluation will be surveyed instead of 20 districts due to contamination by other education programmes or projects.