The Reading Club: Developing a between-class tracking intervention that works at scale by NewGlobe

The Reading Club: Developing a between-class tracking intervention that works at scale

Date: December 2021 Team: Academics, NewGlobe Acknowledgments: Special thanks to:

Leaders in Learning. https://newglobe.education/

The Reading Club: Developing a between-class tracking intervention that works at scale

Contents 1. 2. 3. 4.

Abstract Outline/ Draft Results Discussion

The Reading Club: Developing a between-class tracking intervention that works at scale

Page 04 Page 05 Page 09 Page 09

1. Abstract Properly leveled instruction can play a crucial role in addressing the skills gap experienced by millions of pupils around the world who lack the foundational literacy and numeracy skills required for national standards-based content. One commonly recommended approach, tracking, sorts pupils into classrooms based on skill level instead of age for some or all instruction. Despite the agreement on the potential of this approach, school systems face significant challenges in translating this theory into functional programs, especially at scale; when implemented in school, TaRL programs tend to be replaced with business as usual and classrooms are not re-organized by learning level (Banerjee et al. 2016). Successfully operationalized programs require extensive training, logistical coordination, and most importantly, buy-in from teachers, pupils, and parents.

Because Bridge designs its regular lesson content to meet the needs of the median pupil in each classroom, tracking is a viable option to provide more levelled instruction for its pupils. Bridge takes advantage of its extensive school network to research and test different programs for tracking pupils based on skill level. During a recent small-sample pilot in its Kenyan academies, Bridge investigated the logistical possibilities for tracking and engaged with school leaders, teachers, pupils and parents to identify the keys to community buy-in. This pilot informed the design of a relatively straightforward tracking program for literacy skills that involves transferring pupils between grades for their regular literacy revision classes. Based on pilot observations and community feedback, this program incorporates the following features to optimize operational simplicity and community buy-in:

The Reading Club: Developing a between-class tracking intervention that works at scale

1. Branding as a “Reading Club” opportunity for pupils to work with pupils from a different grade in order to reduce pupil perception of picking out high and low performers 2. Sorting pupils into groups based on guidance from a baseline literacy assessment score cutoff but ultimately deferring to teacher discretion 3. Only transferring pupils between adjacent grade pairs 4. Using existing literacy revision lessons to reduce the additional teacher training needs To rigorously evaluate the impact of this program, Bridge is partnering with researchers from Harvard University to conduct an ongoing randomized controlled trial including 50 treatment Bridge academies in Kenya (Grade 1 and Grade 2) and a second RCT to be conducted with Bridge-operated LEAP academies in Liberia (Grade 1 with Grade 2 and Grade 3 with Grade 4) over the course of 2018-19 Liberia school year. In addition to guiding Bridge’s own decision making, Bridge hopes that this evaluation process will provide other practitioners with useful takeaways for designing and implementing their own tracking programs. This paper describes Bridge’s piloting and testing process for the Reading Club program in Kenya and Liberia and shares preliminary findings from the empirical study, highlighting both the successful features of the program and those features that we recommend improving or avoiding.

2. Outline/Draft: Introduction 1. Recent literature suggests that grouping pupils into homogeneous classrooms based on ability levels (called “tracking”) can benefit pupils a. Can benefit pupils at both ends of the spectrum – both pupils sorted into the less advanced and more advanced group b. Requires effective testing aligned with instructional content to properly sort pupils c. Requires adequate staffing and space and effective logistics and monitoring to ensure that pupils are consistently divided into their appropriate classroom to receive this instruction 2. This paper discusses and analyses the efforts of international education group Bridge International to implement a tracking program to improve literacy learning gains for its pupils in sub-Saharan Africa and Asia a. Bridge program focuses on ease and efficiency of implementation b. Bridge program focuses on early grade literacy c. Ongoing implementation in Kenya and Liberia d. In 2018, Bridge conducted a short term experiment in Kenya involving 50 treatment 3. The case of Bridge International adds to the existing literature for the following reasons: a. Quasi-experimental analysis is potentially limited by selection bias due to possible inherent differences between the types of schools that have tracking programs and schools that don’t (e.g. Collins and Gan 2013). This paper circumvents this issue by taking advantage of Bridge’s broad network with standardized organization, management The Reading Club: Developing a between-class tracking intervention that works at scale

approach, and curriculum design, combined with the ability to randomly assign academies for treatment. b. Different regional and social context – most studies are based on data from the US (e.g. Kulik and Kulik 1992) c. Experimental evidence from sub-Saharan Africa typically has small sample size (e.g. Adodo and Adbayewa 2011) d. Bridge’s experimental evidence from Kenya can engage with some of the most relevant and significant recent results from a large, experimental study in Kenya – Duflo, Dupas, Kremer 2010 4. Summary of findings (TBD – need to get access to the 50v50 data from Carol and confirm with her what findings we can share from the existing analysis and to what extent we can use the data for additional analyses)

Introducing the Bridge Case: Reading Club 1. Bridge International: a. Founded in 2009 b. Educated over 500,000 children through more than 1,000 schools across Africa and India c. Initially operated community-based private schools. Now, majority of pupils attend government schools receiving technical support from Bridge d. Provides teacher training, teacher technology, lesson guides aligned to the national syllabus, and data-driven professional development e. Background on Bridge in Kenya (parent-pay model, 300 academies, Baby Class – Standard 8) f. Background on Bridge in Liberia (partnership with government, 25 schools in Year 1, 43 additional schools in Year 2, Beginner – Grade 6) The Reading Club: Developing a between-class tracking intervention that works at scale

2. Program Development: a. Initial Vision: i. Sorted and grouped G1 and G2 pupils into G1 and G2 classrooms depending on performance on G1 and G2 English and Literacy assessments ii. Every afternoon, instead of their regular class level’s literacy revision lesson, pupils receive the literacy revision lesson that they are sorted into: iii. Lower group (Blue Group) receives regular G1 literacy lesson iv. Upper group (Purple Group) receives regular G2 literacy lesson b. One-Academy Trial i. Initial English and Literacy Exam cutoff scores determined by after testing of G1 and G2 English and Literacy exams to a sample of 60 pupils ii. Determine logistics and schedule of dividing pupils into two groups iii. Receive initial feedback from school leaders, teachers and pupils c. Five-Academy Pilot i. Plan implemented in four additional academies with the following pieces developed from trial: 1. School leader and teacher communication plan 2. Teacher training plan 3. Sorting plan 4. Schedule and Logistics ii. Feedback from teachers and pupils leads to branding of the program as Reading Club to reduce similarities to class promotions or demotions – framed to pupils as an opportunity to read with their friends from other classes d. 50 Academy Pilot in Kenya i. Training and observations to ensure implementation ii. Early feedback from observations suggests that cutoffs are too low – most pupils sorted into Purple Group, but teachers worry that some pupils in Purple Group are not able to learn 1. Cutoffs adjusted 2. Decision to ultimately defer to teacher discretion iii. Baseline and Endline data collected in June and October 2018 to test impact of pilot. 1. 50 treatment schools vs 50 control schools 2. Ongoing observations to track implementation of Reading Club in treatment schools

Situating the Bridge Case in TaRL Research and Theory: 1. Literature Review a. Slavin (1987) b. Kulik and Kulik (1992) c. Adodo and Adbayewa (2011) d. Collins and Gan (2013) e. Banerjee et al. (2016) f. Duflo, Dupas, Kremer (2010) 2. Reference the stylized model of the impact of TaRL presented in Duflo, Dupas, Kremer 2010, which breaks down student outcomes as a function of the following: a. Student Baseline level b. Teacher effort (may be affected by tracking group assignment – teachers in more advanced group may feel more appreciated and put in more effort) The Reading Club: Developing a between-class tracking intervention that works at scale

Difference between the teacher’s instructional level and the student’s level - the greater this distance, the lower the gains of teacher effort. At a certain distance, there are no returns on teacher effort, as the instruction simply has no meaning for that student. d. Peer effects – assumes that pupils learn from other pupils, so exposure to higher achieving pupils positively impacts student learning 3. How does the Bridge Reading Club program specifically interact with each aspect of the model? a. Peer effect should increase Purple Group learning and reduce Blue Group learning b. Adjustments in the teacher’s main instruction level is constrained by teacher guide – Blue Group teacher teaches the Grade 1 lesson and Purple Group teacher teaches the Grade 2 lesson. c. Because lesson guides are designed for median G1 and median G2 pupil, we might expect that pupils in the middle of the spectrum will do worse if assigned to Blue Group because of a lower Peer effect and no difference in the distance between their level and the instruction level d. Teacher effort may be higher in Purple Group than Blue Group – possibly backed up by teacher feedback from Blue Group teachers that they were placed in the more challenging position, having to deal with lower performers

4. Outcomes to investigate based on theory: a. Student Learning (endline – baseline) i. Overall Impact ii. Heterogeneous Impacts 1. By initial reading level 2. By gender 3. By group placement (for students starting around median) The Reading Club: Developing a between-class tracking intervention that works at scale

b. Teacher Effort – did observable teacher metrics differ between groups? i. Teacher Attendance ii. Lesson Completion

Results: 1. Analysis Methodology – still need details from Carol for existing analysis. If possible, data can also be used to dig into additional points of analysis that may not already have been done (heterogeneity, impact of group assignment on median student, any impact on teacher effort) 2. Present results for the outcomes discussed above: a. Student Learning (endline – baseline) i. Overall Impact ii. Heterogeneous Impacts 1. By initial reading level 2. By gender 3. By group placement (for students starting around median) b. Teacher Effort – did observable teacher metrics differ between groups? i. Teacher Attendance ii. Lesson Completion

Discussion: 1. What do results say about the theory of TaRL? 2. Study Limitations a. Possible implementation issues b. Short time period of 4 months (Dufo, Dupas, Kremer 2010 found ~0.14 effect size in 18 months) 3. Moving Forward a. Lessons Learned in Kenya b. Small scale implementation ongoing in Liberia

The Reading Club: Developing a between-class tracking intervention that works at scale

Leaders in Learning https://newglobe.education/

The Reading Club: Developing a between-class tracking intervention that works at scale