Impact Evaluation of Education Quality Improvement Programme in Tanzania: Midline Technical Report, Volume I: Results and Discussion

Table 2: ML quantitative survey sample sizes

Source: IE ML survey. Notes: (1) In four treatment schools and four control schools, there were fewer than 15 eligible pupils, so the targets are fewer than 1,500. (2) The samples includes 21 HTs/acting HTs (treatment) and 14 (control) who teach Stds. 1–3. Out of the 39 teachers in treatment and control schools who did not sit for the interview, one refused, while 38 were unavailable (absent on the day and could not be reached over the phone later). Some 11% of teachers in treatment schools and 6% in control schools were interviewed over the phone. (3) 94 maths (arithmetic) lessons and 137 Kiswahili lessons (either reading or writing) were observed. Some of these subjects were taught consecutively (without a break) in one class period. 172 separate class periods were observed.

1.3.4 ML qualitative instruments and sample

The same nine schools and communities across three treatment districts/regions that were purposively sampled as sites for the qualitative research for the BL were visited again. The original sampling approach was theoretically informed and designed to generate responses from a selected number of individuals and groups that are broadly representative (though not statistically) of groups relevant to EQUIP-T, and which allow some identification of heterogeneous impact (Volume II, Chapter 4, describes the sampling strategy) The ML fieldwork took place in April/May 2016, at the same time as the quantitative survey. This was a change from the timing of the BL qualitative research, which took place between late June and early August.11

As with the BL, the qualitative part of the IE makes use of two research instruments – key informant interviews (KIIs) and focus group discussions (FGDs). All of the KIIs and FGDs utilise structured and unstructured methodologies. Structured methods allow for the efficient testing of pre-specified hypotheses, and unstructured methodologies allow for unanticipated or context specifics to be captured and for new hypothesis to be developed. The sampling of respondents

11 As explained in the IE ML Planning Report (OPM, 2016a, p. 22), at BL the timing was designed mainly to ensure sufficient time for the early results from the quantitative survey to feed into the qualitative design. At ML, the qualitative research was easier to frame in advance, given the rich BL findings, and the concurrent timing helped to strengthen the integrated analysis process.

Impact Evaluation of Education Quality Improvement Programme in Tanzania: Midline Technical Report, Volume I: Results and Discussion and type of instruments that were used is almost the same as the BL, and is set out in Table 3 below.

Table 3: ML IE qualitative research participants, sampling and instruments

10 fathers and 10 mothers (not from the same family) selected at random from a list of parents of children in Standard 1–3, and those involved with PTPs 6

FGD x 2

Change: At BL fathers and mothers were interviewed together, and the HT selected the parents based on guidelines. Children 3 boys and 3 girls selected at random from Standard 3.

Source: OPM team. Note: (1) In the HT’s absence, AHT was interviewed. (2) Or another member of the committee if they were not available. (3) If there were more than eight such teachers, eight of them were selected randomly to participate. (4) Although last time teachers from higher grades were invited if there were not enough teachers in Standards 1–3. (5) Aiming for attendance of four to 10 people. (6) The randomisation was expected to produce a group with some heterogeneity in regard to socioeconomic status and religion, but if researchers felt this was not the case, purposive sampling was allowed, with assistance from the HT.

The qualitative research team was able to carry out all of the KIIs and FGDs intended.12 The size of the focus groups ranged from as few as three teachers up to as many as 10 participants in the mothers/fathers FGDs. This was due to strong turn-out of parents, while some of the schools only had small numbers of Standards 1 to 3 teachers

12 The only exception was the fact that the Technical Lead for Component 3 at EQUIP-T MA HQ was not available.

Table 4: Actual qualitative sample at ML

1.3.5 Use of mixed methods at ML

Members of the quantitative and qualitative research teams worked together at the ML design, data collection and analytical stages of this ML IE study to ensure that the study is able to take full advantage of its mixed methods design. The ML evaluation matrix, which serves as a framework for the ML research, was jointly developed in January/February 2016, following a review of the BL findings and interviews with key education sector stakeholders, including EQUIP-T programme staff (see OPM, 2016a for more details of this process), to establish the research priorities. The tools for both methods were then developed concurrently, with the two teams working closely together to ensure complementarity.

As at the BL, the research areas emphasised in the two sets of ML tools differ because of their different methodological strengths. The ML qualitative research focuses particularly on district planning and management, community participation and accountability, teacher motivation and the relationship between teachers, HTs and WECs. Qualitative methods are particularly well suited to capturing perceptions of changing roles, responsibilities, relationships and accountabilities. By contrast, the quantitative survey is uniquely placed to robustly estimate the impact of the programme on pupil learning and other outcomes. It concentrates on areas of the expanded TOC related to pupil learning, teacher capacity and performance, and SLM, while still covering some quantifiable aspects of community engagement with schools, and ward- and district-level management of schools.

The concurrent timing of the ML quantitative survey and qualitative research (April/May 2016), whereby qualitative research teams visited schools shortly after the quantitative survey team did so, had a number of advantages for strengthening the research process. The qualitative team were able to contact the school in advance of visits using updated information from the quantitative team, in order to arrange for the participants to be available at the school. Observations on research priorities from the early stages of the quantitative fieldwork were also shared with the lead qualitative researchers, which enabled them to modify parts of the planned research. For example, a decision was taken to include non-participatory lesson observation in each of the research sites, to enrich the quantitative evidence.

An early sharing of draft quantitative and qualitative findings during a team workshop held in early August 2016 permitted a rich discussion and pointed to areas for further investigation in both data sets. This type of information sharing and enrichment continued into the report writing phase. Each chapter in Part B of this report, which contains the findings, was co-authored by a member from each of the quantitative and qualitative teams. This ‘buddy’ system works by members of each team sharing and commenting on iterative drafts – sometimes sparking further data analysis to further validate or confute links in the TOC supported by one set of data.

1.3.6 Costing analysis at ML

For the ML costing study, spending by the EQUIP-T MA was analysed, along with monitoring data relating to the five components. In addition, spending reports from Local Government Authorities (LGAs) were used to analyse spending trends since the EQUIP-T fiscal decentralisation model came into place in late 2015. This also allowed a deeper look at the costs of different models of delivering teacher INSET.13

1.4 Changes in education sector context and implications for the IE

1.4.1 Education policy changes since BL

Since the BL research in 2014 there have been at least four national policy changes that affect primary education across the country (see below and Annex D.1 for more details). Recognising the changing education context since the BL is very important for the IE as national trends rather than EQUIP-T could be driving some of the observed changes.

 The introduction of a new curriculum for Standards 1 and 2 pupils that focuses on reading, writing and arithmetic competencies (3Rs), rather than a larger set of subjects, and that promotes a new phonics-based approach to teaching children to read. Schools began implementing this during the 2015 school year.

 Free education (no fees or compulsory additional contributions) for parents and guardians of children from Standards 1 to Form 4, from the start of the 2016 school year.

 School capitation grants paid directly to school bank accounts rather than via LGAs, from December 2015.

 The transfer of the management of primary education from the Prime Minister’s Office to the President’s Office 14

The new Government, which took office in late 2015, introduced the last three policy changes, and also set a very high-profile national agenda for encouraging public servants (including education professionals) to work hard, and to carry out their duties professionally, to improve public services.

1.4.2 Major primary education programme interventions since BL, including the Literacy and Numeracy Support Programme (LANES)

A set of prominent donor-funded programmes, including EQUIP-T, have been working in Tanzania in the last two years to improve the quality of primary education, under the umbrella of the Education Sector Development Programme (ESDP) II. Although interventions differ across these programmes, they share at least one common objective: that of improving early grade pupil learning.

13 Spending data at the MA was not found to be categorised in sufficiently consistent detail to be able to estimate unit costs of specific phases of activity. This limitation was discussed in the ML IE Planning Report (OPM 2016a, p. 32).

14 New instruments of governance were issued by the President’s Office in April 2016. The President’s Office is now managing a 10-year basic education cycle (which includes primary education).