Appendix D: Methodological groundwork for the SDI teacher and student assessments

The Service Delivery Indicators (SDI) teacher and student assessments are based on a study commissioned by the SDI program and completed in 2010 by Andrew Cunningham, David Johnson, and Rachel Dowling (Cunningham, Johnson, and Dowling 2012). Their study reviewed, analyzed, and summarized the national third- and fourth-grade curricula of 12 countries in order to identify common themes (for example, problem solving, long division): Botswana, Ethiopia, The Gambia, Kenya, Madagascar, Namibia, Nigeria, Rwanda, the Seychelles, South Africa, Tanzania, and Uganda. The study focused on Africa,

because this is the region in which the SDI program was conceived. Study results informed the content and question format used for the teacher and student assessments in SDI education surveys.

The SDI teacher assessment was designed with the objective of examining whether teachers have the basic reading, writing, and arithmetic skills that lower-primary students need in order to progress in their education. A score of 80 percent correct is interpreted as the minimum knowledge required for a teacher to be effective, which becomes an SDI indicator representative at the national level. The tests also examine the extent to which teachers demonstrate mastery of skills that are above the level they are teaching (but still at the level of primary school content).

The SDI teacher assessment includes two sections, each lasting 35 minutes. The teacher knowledge section resembles grading a math and literacy exam. It asks teachers to correct a letter with grammatical errors, fill in missing words in sentences, solve arithmetic problems, read the time on a clock, and read and understand simple graphs. All of the content is based on material drawn from the national curricula of the 12 reference countries. Similarly, the pedagogical section resembles tasks that primary teachers face daily: preparing to teach a lesson, assessing differences in children’s abilities as learners, and correctly evaluating students’ learning achievements and progress. Because of the nature of the content, teacher assessments are paper based and implemented in groups at an arranged time when teachers do not have to worry about other responsibilities (for example, during lunch break or after classes). Unfortunately, only teachers who are present in the school can be assessed, so the results represent an upper boundary if it is assumed that teacher absence and knowledge are negatively correlated.

The SDI student assessment includes three sections: mathematics, language, and nonverbal reasoning. For the language section, the language of instruction in a given country is always tested. However, in some countries, the vernacular or other common languages are also tested in a subsample of students. Exams are always translated by the local implementation team and validated by government counterparts. The language questions include items on correctly identifying a letter or word, matching a word with an image, reading a sentence and a paragraph out loud, and answering questions based on a simple story. The nonverbal reasoning section includes four questions on completing patterns following Raven’s Progressive Matrices. The mathematics section is the most extensive and covers items on identifying and ordering numbers, adding, subtracting, and multiplying up to three digits, dividing up to two digits, handling fractions, completing a word problem needing arithmetic, and completing a sequence of numbers. Because of the multiple languages tested and the small number of nonverbal items, the math section is the most comparable across countries.

Unlike the teacher assessment, the SDI student assessment was initially designed only to serve as a correlate for other SDI indicators, not to result in an SDI indicator nor to serve as a large-scale international assessment. The content of the student assessment was also derived from Cunningham, Johnson, and Dowling (2012) and includes topics covered in the third- and fourth-grade national curricula of the 12 countries. Each student has 25 minutes to complete the test, including 10 minutes for the language section, 10 minutes for the mathematics section, and 5 minutes for the nonverbal reasoning section. If the student has not completed a section after the allocated time, the test proctor makes the student move on to the next section. The student assessment is implemented orally and is conducted one-on-one to prevent guessing and missing information from students who are unable to read. For the written component, the proctor informs the student that he or she can answer questions in any order.

In contrast with the teacher assessment, the time of the year in which the SDI student assessment is implemented might affect the results, particularly because the items are drawn from the third- and fourth-grade curricula and the assessment is taken by students somewhere in the middle of the fourth-grade school year. Although this is important for interpreting observed differences, SDI students’ assessments are not the only international assessments that suffer from this caveat, and informative cross-country comparisons are still useful.

Given its solid grounding in the national curricula of multiple countries, the breadth of topics covered, and the careful implementation guidelines followed, the SDI student assessment results prove to be robust to several psychometric properties. SDI tends to be implemented in countries where information about the education sector is scant and no other international assessments have been implemented. The SDI team is currently working with the Harmonized Learning Outcomes team to have SDI assessments results included in this database, in instances where the results are useful and no better information is available. This effort will help to improve the precision and increase the availability of other World Bank Group initiatives such as the Human Capital Index.

Reference

Cunningham, A., D. Johnson, and R. Dowling. 2012. “Teaching Standards and Curriculum Review.” Report 1-31072012, World Bank Group, Washington, DC.

Have teachers mastered the subject matter they are teaching? Can doctors accurately diagnose and treat critical health conditions? Are schools and health facilities sufficiently stocked with needed equipment and supplies? Are they sufficiently supported and staffed to optimize learning and health care outcomes? For the past decade, the World Bank’s Service Delivery Indicators (SDI) surveys have collected nationally representative data in countries across Africa to answer these questions. The surveys aim to measure the quality of services where they meet citizens: in schools and health facilities. The Quality of Health and Education Systems Across Africa: Evidence from a Decade of Service Delivery Indicators Surveys identifies areas of achievement and constraint in service delivery, shedding light on how service delivery may foster or stunt human capital accumulation. SDI surveys show that schools and health clinics across Africa are still falling short in some critical areas. The delivery of primary care services is very heterogenous between and within countries. Many health facilities lack the basic necessities to provide proper care, such as essential medicines, basic diagnostic equipment, and adequate water and sanitation. Moreover, health care providers’ ability to diagnose and treat common health conditions correctly is low and distributed unevenly. Health personnel’s absence from health facilities remains a concern across the surveyed countries. Learning is low, and, not unlike health care, levels of student learning vary significantly across countries: less than half of grade 4 students can recite a simple sentence or perform basic mathematical operations. This deficient learning is correlated with teachers’ low levels of content knowledge and sub-par pedagogy skills. Some schools are also missing crucial inputs, such as blackboards or private and gendered toilets, and struggle with high pupil-teacher ratios. Despite these challenges, success stories in both sectors illustrate the quality of service delivery that could be achieved and showcase the dedication of teachers and medical staff across Africa. By studying data from thousands of facilities, considering the local context, and drawing insights from the literature, this book offers important insights for how countries can strengthen health and education systems and build back better in the wake of the massive disruptions brought about by the COVID-19 pandemic.

ISBN 978-1-4648-1675-8