Impact Evaluation of Education Quality Improvement Programme in Tanzania: Endline Quantitative Technical Report, Volume II

Draft versions of the instrument and protocol sections of the manual were shared in softcopy with interviewers as a reference during the training, and used as guidelines by the trainers. The manual was updated on an ongoing basis during the training and pilot phase where updated conventions or additional clarifications were needed. The final version of the manual was shared in softcopy with all fieldworkers at the end of the pilot phase.

D.3 Training and pilot

Enumerator training and a field pilot took place in Dar es Salaam and Dodoma from 26 March to 14 April 2018. A total of 60 trainees participated in the training. The training was delivered by four members of the fieldwork management team, the overall project manager of the impact evaluation, and another member of the core evaluation team.

The main objective of the training was to ensure that team members would be able to master the instruments, understand and correctly implement the fieldwork protocols, comfortably use CAPI, and be able to perform data validation. Supervisors were furthermore trained on their extra responsibilities of data management, fieldwork and financial management, logistical tasks, and the transmission of data files to the data manager.

The training had two components: a classroom-based training component and a field-based component that included a full scale pilot. The performance of enumerators was assessed on an ongoing basis, using written assessments and observation of performance in the field and these scores were recorded. At the end of the training and pilot phase, the final fieldwork team was selected using this information.

A higher number of data collectors than needed for data collection were invited to and attended the training. This allowed for a selection of the best suited candidates at the end of the training and provided a pool of reserve additional trained staff that could be called upon in case of enumerator attrition during data collection.

D.4 Fieldwork organisation

D.4.1 Fieldwork plan

The fieldwork plan was designed to cover all 200 schools within all 12 regions and 25 districts for the duration of not more than six weeks. The plan had to cater for the short fieldwork time window dictated by the end of the school mid-term break and the start of exams at the end of the term; rainy season; allowing the fieldwork management team to supervise teams during the first week of implementation; minimising travel days between districts and during the weekdays; suitable allocation of teams to districts to address cultural and language barriers; and flexibility to deal with unforeseen circumstances.

D.4.2 Fieldwork model

The team composition and fieldwork model at endline were the same as those at midline with the exception of adding one more field team to deal with the shorter timeframe at endline and to ensure that the fieldwork is completed within five to six weeks. At endline there were four treatment teams composed of five enumerators and one supervisor, four control teams of four enumerators and one supervisor each, and one team of five enumerators and one supervisor that visited control and treatment areas. Each team visited and completed one school on one day.

D.5 Fieldwork implementation

The fieldwork started on 16 April and ended on 21 May 2018 with no breaks in-between, except for a couple of days of bank holidays and a few travel days for some of the teams Teams communicated regularly with OPM to report delays and/or any event likely to affect the feasibility of the fieldwork plan.

D.5.1 Replacements

D.5.1.1 Schools

All schools that were interviewed at baseline and midline were revisited and interviewed at endline, and hence no replacement of schools took place. There were only two cases where teams visited a school and were unable to conduct the survey because they had to report to the district office due to security concerns. In those cases, the teams rearranged to come back another day to conduct the survey in those schools.

D.5.1.2 Pupils and teachers

Only 68 pupils (out of 2,999 pupils) were replaced. The reasons for replacement were: 29 were unavailable due to sudden events such as illness, 30 were absent (but had been recorded by the teacher as present and hence were part of the sampling frame), 5 could not speak or see or hear at all and thus they were not asked to sit for an hour long interview, and 4 for some other reason.

No replacement was done for the teacher interviews or lesson observations, as no sampling was required.

D.5.2 Response rates per instrument

Table 2 in Chapter 3 above shows the generally high response rates for each instrument. Here is some further information underlying the response rates for selected instruments:

 If the parent or guardian of the tested pupil or other adult household member could not be reached, as a last resort, the poverty scorecard was administered to the pupil. This happened in 230 out of 2,992 cases (8%). Some of the reasons given by enumerators were that the pupil is boarding and parents live far away, pupil lives too far away to be reached, and parents were not found at home after repeated visits.

 Some 97 of the 889 teacher interviews (11%) were conducted over the phone, as the teacher was absent on the day of the survey.

 In 40 out of the 200 schools, the head teacher was absent on the day of the survey and as a result the assistant head teacher or another teacher was interviewed instead to collect information related to school records. After fieldwork ended, head teachers in 38 of those 40 schools were reached over the phone to complete the missing modules of the head teacher interview.

D.6 Quality control and data checking protocols

At the end of each working day, supervisors collected all interview files from their team members and uploaded them into a shared and organised Dropbox folder that was set up by the data manager. The data manager would receive all files from all nine teams and export them into Stata data files (a statistical programme) and then run daily checks on all files to make sure they are complete and identify potential errors.

Several mechanisms were put in place in order to ensure high quality of the data collected during the survey. These are briefly summarised in turn below:

D.6.1 Selection and supervision of enumerators

As discussed above, each enumerator was supervised at least once by the training team during the training, piloting and first week of data collection. This allowed a well-informed selection of enumerators and their allocation into roles matching individual strengths and weaknesses.

D.6.2 CAPI built-in routing and validations

One important quality control means in CAPI surveys is the use of automatic routing and checking rules built into the CAPI questionnaires that flag simple errors during the interview, that is early enough for them be corrected during the interview. In each CAPI instrument, validations and checks were incorporated in the design in order to significantly reduce errors and inaccuracies during data collection. In addition to having automatic skip patterns built into the design to eliminate errors resulting from wrong skips, the CAPI validations also checked for missing fields, out of range values and simple inconsistencies within instruments.

D.6.3 Secondary consistency checks and cleaning in Stata

The endline survey exploited another key advantage of CAPI surveys, the immediate availability of data, by running a range of secondary consistency checks across all data on a daily basis in Stata. Data received from the field were exported to Stata the following day, and a range of do-files were run to assess consistency and completeness, and make corrections if necessary. The checks comprised the following:

 ID uniqueness and matching across instruments;

 Completeness of observations: target sample size versus actual; and

 Intra and inter-instrument consistency and out of range checks. The data manager ran the checking do-file on a daily basis on the latest cleaned data. This would return a list of potential issues in the long format which the data manager would then investigate and undertake the necessary cleaning actions. Whenever any issue was flagged, effort to obtain an explanation was undertaken either by reviewing enumerator comments or phoning teams

On a daily basis, the data manager collated, shared and discussed all flagged errors with the supervisors in the field, who in turn discussed them with their team members. Throughout the fieldwork, occurrences of errors were monitored in order to keep an eye on the performance of data collectors and constantly provide them with feedback to improve.

In addition to the checking and cleaning process, all enumerator comments as well as other specify variables were translated from Kiswahili to English. All translated entries were further reviewed by the data analysis team in order to 1) ensure that they are understandable and properly translated into English and 2) none of the other specify answers for multiple response questions are in fact synonymous to one of the existing response items. The revision resulted in a long list of other specify items that were then recoded into one the available response items.

D.6.4 Monitoring fieldwork progress and performance indicators

In addition to the above checks that were specific to each instrument, the survey team built a dashboard that allowed for daily monitoring of the general progress of the fieldwork and specific indicators revealing the performance of teams and enumerators over time. For example, indicators included number of control/treatment schools completed, number of instruments completed within each school, average interviewing time of each instrument, time of the day when the pupil tests were conducted, number of pupils interviewed for the scorecard instead of their parents, number of teacher