Appendix C: Survey methodology

The Service Delivery Indicators (SDI) initiative aims to provide national and subnational evidence on the quality of primary education and basic health service delivery from the perspective of an average citizen accessing these services. To do so, SDI survey instruments are structured meticulously to collect information on three aspects of service delivery: (1) provider effort, (2) provider knowledge, and (3) facility inputs. Provider effort is measured by collecting data on teachers’ and health workers’ absenteeism rates, health care providers’ caseloads, and teachers’ time spent in teaching activities. Provider knowledge is measured by collecting data on teachers’ knowledge of the curriculum and quality of their pedagogy as well as health workers’ diagnostic and treatment accuracy, adherence to clinical guidelines, and management of maternal and neonatal complications. Facility inputs are measured by observing each facility’s equipment and infrastructure availability, each health facility’s medicine availability, and each school’s student-to-textbook and student-to-teacher ratios.

Survey teams visit schools and health facilities to collect data through a combination of observation and interview techniques.1 SDI survey instruments are nimble and use cutting-edge data collection methods, allowing for relatively

rapid fieldwork and data analysis. In addition, SDI projects involve rigorous quality control with defined standards. Through a participatory approach during planning and design, World Bank staff, working alongside local technical teams and other stakeholders, ensure that survey modules reflect national standards while maintaining cross-country comparability. During the implementation of data collection activities, enumerators receive comprehensive training and follow established protocols, and trained supervisors randomly check fieldwork. SDI analysis follows internationally accepted definitions of indicators and uses reproducible statistical programming.

The overall duration of an SDI project depends on the sample size and country context. As a rough estimate, an SDI survey can take between 12 and 18 months, depending on procurement, planning, and seasonality. Its project cycle encompasses four stages: (1) survey planning and design, (2) survey implementation, (3) analysis and capacity building, and (4) dissemination and mobilization.

During the planning and design stage, the World Bank’s SDI core team (hosted at the Human Development Practice Group’s Chief Economist Office), task teams, and government authorities engage in initial conversations to understand the SDI survey’s goals and the processes involved, identify the country’s priorities and capacity needs, and align the SDI survey with the country’s sector strategy. These initial conversations involve virtual and in-country consultations and, potentially, field visits to facilities.

After the SDI survey’s overall goals are defined and agreed upon, the SDI core team, task teams, and a local technical working group collaborate to contextualize and adapt the survey instruments2 and define sampling procedures. The local technical group includes ministry and regional officials, development partners, and colleagues from research and academic institutions. Survey protocols and instruments are submitted to the corresponding authorized ethics committee or institutional review board for approval. Once survey instruments are finalized and approved, a computer-assisted personal interviewing designer is hired to program the instruments onto tablets. Additionally, a field coordinator is assigned to supervise and coordinate the overall data collection fieldwork, and a survey firm is hired to conduct data collection activities.

During this planning and design stage, roles and responsibilities among the different stakeholders involved in the SDI project are also defined and agreed upon. In general, the SDI core team’s main responsibility is to ensure that a high-quality survey can take place, guaranteeing indicators that are nationally representative and comparable across time and countries. The core team’s assigned functions often include providing (1) standardized materials and suggestions for survey adaptation, (2) technical guidance on survey design and sampling procedures, (3) protocol for and assistance in quality control, and (4) guidance on training, planning, and procurement (when needed).

Task teams are responsible for the overall coordination of the project, ensuring that operations run smoothly during the different stages of the work. Task teams lead the dialogue with government authorities and identify the local technical working group. In addition, task teams are responsible for contracting and managing the survey firm, taking the lead in analysis and report writing, and creating a dissemination strategy for the SDI results.

Governments provide a unique perspective about the country’s contextual challenges, opportunities, and needs. They also promote an appropriate environment that ensures compliance with SDI survey quality standards and buy-in from national and subnational stakeholders.

Finally, the fieldwork team (field coordinator, survey firm, and information technology [IT] technicians) agree on a contract to comply fully with survey protocols and guidelines when conducting data collection activities. During the survey implementation stage, the survey firm’s supervisors and enumerators are trained for approximately three weeks with a combination of workshop and field-based sessions. In parallel, instruments are piloted and finalized for approximately two weeks. Once data collection activities begin, they take approximately eight weeks to be completed, depending on the sample size and logistical considerations.

The survey firm conducts these data collection activities by deploying several survey teams to the field. Survey teams are usually composed of a field team supervisor, an IT technician, and two to three enumerators. The field coordinator tracks and oversees work conducted by the survey firm and its survey teams and communicates overall progress to task teams daily. The field coordinator is responsible for overall data quality control during the implementation stage.

The field team supervisor is the senior member of each field team. He or she is responsible for the well-being and safety of team members, completion of the assigned workload, and maintenance of data quality for that team. Each field team supervisor receives his or her assignments from and reports to the field coordinator.

The responsibilities of the enumerators include completing all required interviews, observations, and assessments or tests. Enumerators are also expected to check completed survey forms to ensure that all questions were asked and responses were properly recorded. This check should be done at the end of the first visit to allow for later completion of any questions that were inadvertently skipped or for which responses were incorrectly recorded. Enumerators upload their data as frequently as possible to allow for real-time checking of results. Finally, IT technicians are employed to provide technical support to the survey teams, helping with troubleshooting tablet and software malfunctions during the fieldwork.3

Before the fieldwork starts, introduction letters are sent to all facilities in the sample to ensure the timely flow of information and preparations for smooth and efficient data collection. Apart from requesting permission and collaboration from the facility to allow the field team to conduct the survey, these letters also detail the scope of the survey and preparations that facilities must have in place before the arrival of the field team. The letters also specify the number of visits that field teams will make to the sampled facilities.4 Once survey teams arrive at the facilities, field team supervisors, enumerators, and IT technicians carry special identification documents. They bring copies of introduction letters and official letters sent to facilities by their corresponding governing body (for example, ministry) to introduce the survey.

Data collection activities consist of two visits to each sampled facility: an announced visit following by an unannounced visit (an example on how survey visits are usually structured is shown in table C.1).

During the announced visit, enumerators collect data on the following: • The facility’s inputs and procedures • The full roster of teachers or health workers and their qualifications • Professional knowledge of up to 10 randomly selected teachers (measured using fourth-grade tests and pedagogical assessments) or health workers (measured using patient case simulations on symptoms of high-burden diseases) • Knowledge of up to 10 randomly selected pupils in math, language, and nonverbal reasoning (in schools) • Classroom observation for 45–60 minutes of one randomly selected lesson (only for SDI education surveys) • Perceptions and satisfaction of visiting clients (only in health facilities and for some countries)

During the unannounced visit, which occurs at least five days after the announced visit, enumerators collect data on the following: • Attendance check of presampled teachers or health facility workers • Unstaffed classrooms (only for schools)5

Detailed coordination and specialized skills are required to implement SDI surveys successfully. Therefore, survey firms comply with strict selection and training processes when they hire enumerators and assemble survey teams. The “patient case simulations” (or “clinical vignettes”) are a good example of the high level of coordination and specialization required by teams to implement SDI surveys. Patient case simulations are answered by health care providers (who provide care for patients) to measure their knowledge and competence in diagnosing and treating key illnesses and complications. To conduct this section of the survey successfully, at least two enumerators (often with previous medical knowledge and expertise) are assigned to a survey team and receive intense training on a detailed set of instructions that must be followed during fieldwork