PRM Carteret Report John Hopkins JHU

Efficacy Study of Phonics, Reading, and Me in Carteret Public Schools

Michael A. Cook, PhD

Jane Eisinger, MS

Steven M. Ross, PhD

June 2023

EXECUTIVE SUMMARY

Efficacy Study of Phonics, Reading, and Me in Carteret Public Schools

In November 2022, The Center for Research and Reform in Education (CRRE) at Johns Hopkins University contracted with Learning Without Tears (LWT) to conduct a mixed-methods evaluation study of the impact of LWT’s Phonics, Reading, and Me (PRM) in Nathan Hale Elementary school in the Carteret (NJ) Public Schools district. This quasi-experimental design (QED) study compared the early literacy achievement of Grades K-2 students, as measured by each district’s early literacy progress monitoring assessments, of students assigned to use PRM in relation to a control group that continued with business-as-usual instruction.

Learning Without Tears has developed educational products designed to build handwriting skills and fluency that solidify the foundation to build strong literacy success. The focus of the present research is LWT’s Phonics, Reading, and Me program. PRM is a K-3 supplemental print and digital program that takes students from learning letters to fluent reading through meaningful phonics instruction. PRM sequentially and systematically helps students acquire the range of skills they need to become proficient readers. This supplemental program prioritizes phonics skills that can be difficult to learn but impact students significantly along the path to reading. It is designed to support the gaps in core literacy programs with foundational skills learned through PRM rich, connected text. PRM has partnered with SoapBox Lab’s engine to gauge student’s oral reading skills in each lesson. Students read connected text in books and through Student Digital Learning to practice decoding and recognizing common words while gaining content knowledge. The key learning outcome is independent reading proficiency. Student digital learning experiences are adapted based on their performance in formative assessment checkpoints. When students need more support, they receive it for a particular skill. All students start with an oral reading screener to assess skill proficiency. The results of this screener will place them in an adaptive learning path. In summary, PRM:

• Focuses on the most targeted and high-utility phonics skills and word study

• Provides practice in phonics and high-frequency words through decodable texts

• Supports and complements core literacy curricula

• Aligns to the science of reading

The present QED was designed to address the following research questions:

1. How does student growth on early literacy measures compare to that of comparison students?

2. Do program outcomes vary based on demographics (i.e., ethnicity, ELL or SPED status)?

StudySample

The study sample consisted of 205 Grades K-2 students across 12 classrooms in one elementary school. Demographics showed that the student sample was majority Hispanic and low-income (FARMS) students. Interestingly, the comparison group contained considerably larger percentages of ELL students and considerably smaller percentages of special education students than did the treatment group.

ProgramImpactonStudentAchievement

PRM had a positive, though not statistically significant, impact on early literacy achievement, as measured by the i-Ready Reading assessment, in spring 2023, controlling for winter 2023 i-Ready scores and demographic variables. Students who participated in PRM averaged slightly more than 6-points larger winter-to-spring gains (effect size = .11) than did comparison students who did not participate in PRM. Subgroup analyses showed that PRM had significant impacts on early literacy achievement for ELL students.

Conclusions

The key results and conclusions of this evaluation are as follows:

• PRM students scored more than 6-points higher on the spring 2023 iReady Reading assessment than did comparison students, controlling for winter 2023 i-Ready scores and demographic variables. This impact did not reach statistical significance, although the effect size was 0.11, indicating a small to medium practical impact.

• Subgroup analyses showed that PRM had significant positive impacts for ELL students, with PRM ELL students outgaining ELL comparison students by 23 points.

• PRM impacts for Grade 2 students, as well as Hispanic and FARMS students, were positive and approached statistical significance (p< 10). These advantages ranged between 8-11 points.

Efficacy Study of Phonics, Reading, and Me in Carteret Public Schools

In November 2022, The Center for Research and Reform in Education (CRRE) at Johns Hopkins University contracted with Learning Without Tears (LWT) to conduct a mixed-methods evaluation study of the impact of LWT’s Phonics, Reading, and Me (PRM) in Nathan Hale Elementary school in the Carteret (NJ) Public Schools district. This quasi-experimental design (QED) study compared the early literacy achievement of Grades K-2 students, as measured by each district’s early literacy progress monitoring assessments, of students assigned to use PRM in relation to a control group that continued with business-as-usual instruction.

Learning Without Tears has developed educational products designed to build handwriting skills and fluency that solidify the foundation to build strong literacy success. The focus of the present research is LWT’s Phonics, Reading, and Me program. PRM is a K-3 supplemental print and digital program that takes students from learning letters to fluent reading through meaningful phonics instruction. PRM sequentially and systematically helps students acquire the range of skills they need to become proficient readers. This supplemental program prioritizes phonics skills that can be difficult to learn but impact students significantly along the path to reading. It is designed to support the gaps in core literacy programs with foundational skills learned through PRM rich, connected text. PRM has partnered with SoapBox Lab’s engine to gauge student’s oral reading skills in each lesson. Students read connected text in books and through Student Digital Learning to practice decoding and recognizing common words while gaining content knowledge. The key learning outcome is independent reading proficiency. Student digital learning experiences are adapted based on their performance in formative assessment checkpoints. When students need more support, they receive it for a particular skill. All students start with an oral reading screener to assess skill proficiency. The results of this screener will place them in an adaptive learning path. In summary, PRM:

• Focuses on the most targeted and high-utility phonics skills and word study

• Provides practice in phonics and high-frequency words through decodable texts

• Supports and complements core literacy curricula

• Aligns to the science of reading

The present QED was designed to address the following research questions:

1. How does student growth on early literacy measures compare to that of comparison students?

2. Do program outcomes vary based on demographics (i.e., ethnicity, ELL or SPED status)?

Method

ResearchDesign

In this QED study, early literacy scores for Grades K-2 students who used PRM were compared with those from comparison students in Grades K-2 who did not use PRM. Overall scores were analyzed for all students with non-missing pretest, posttest, and demographic data. Even though condition assignment was assigned at the classroom level, all analyses were conducted at the student level, due to the small sample size of both students and classrooms.

Participants

Participants were drawn from Nathan Hale Elementary School in Carteret Public Schools. Carteret Public Schools is categorized by the National Center for Educational Statistics (NCES, https://nces.ed.gov/) as a suburban district, with nearly 3,900 students across three elementary schools, one middle school, and one high school. Hispanic students make up the largest ethnic group in Carteret Public Schools (39%), followed by Asian (23%), White (22%), and Black (12%).

Participants in this evaluation included Grades K-2 students in the 2022-23 school year that were in classrooms selected by the vendor (LWT) to be treatment (PRM) or comparison (business-as-usual) classrooms. Student data from a total of 12 classrooms across Grades K-2 were used in this evaluation. A classroom-level QED was used in this study, meaning that certain classrooms across Grades K-2 in Nathan Hale Elementary used PRM, while other classrooms, across the same grade levels, did not use PRM. Comparison classrooms engaged in business-as-usual practices by teaching early literacy using preexisting methods.

Pretest and posttest data, along with demographic data, was supplied to CRRE by Carteret Public Schools research personnel. The initial data file consisted of 242 students (58 T and 184 C) across 12 classrooms identified by LWT as participating in the evaluation. Attrition involved students who were missing one of either a pretest or posttest score, resulting in a final analytic sample size of 205 (55 T and 150 C) students for the main early literacy analyses.

Table 1 shows the unadjusted mean pretest early literacy scores for treatment students who participated in the PRM program, and comparison students.

Table 1

Baselineequivalence,winter2023i-ReadyReading(n=205)

Baseline equivalence was met for the analytic sample, as evidenced by the standardized mean difference of +0.12, well under the WWC threshold of 0.25 SDs (WWC, 2020). However, demographic characteristics of treatment and comparison students were considerably different, as evidenced by Table 2.

Table 2

Demographiccharacteristics,bycondition

Note. * p< .05.

PRM classrooms had significantly smaller percentages of ELL and female students and significantly larger percentages of special education students than did comparison classrooms. The considerable differences in ELL and special education students specifically indicated that, even though baseline equivalence was met on the prior early literacy achievement measure, adjustments needed to be made to make the two conditions as similar as possible. For this reason, propensity-score weighting (PSW) was conducted on the analytic sample. Treatment students were each given a weight of one, and comparison students were each given a weight of:

Students with weights of greater than 10 were dropped from analyses, as weights of these magnitudes are indicative of individual students who would have an outsized influence on analytic results.

The result of these PSW procedures was that comparison students who were more similar to treatment students (in terms of prior achievement and demographic

covariates) were weighted more heavily in the analyses, and comparison students who were less similar to treatment students were weighted less. This approach resulted in the creation of weighted comparison groups that were as similar as possible to the observed groups of treatment students. After these weights were applied to comparison students, baseline equivalence was achieved for winter 2023 i-Ready Reading scores, as shown in Table 3. These weights will be incorporated in the main impact analyses in this evaluation.

Table 3

BaselineequivalenceafterPSW,winter2023i-ReadyReading(n=205)

Measures

Data sources for the current study include pretest and posttest i-Ready Reading scores, as well as limited demographic data that include student grade, ethnicity, gender, and ELL and SPED status indicator variables.

I-Ready Diagnostic Assessment. Overall i-Ready Diagnostic Reading assessment scores were obtained for all Grades K-2 students in study classrooms in the 2022-23 school year. We obtained fall, winter, and spring i-Ready scores, but focused mainly on winter scores as a prior achievement variable and spring scores as the outcome variables, as PRM implementation did not begin until February 2023. i-Ready diagnostic assessment scores range from 0-800 and are vertically scaled and nationally normed across grades, meaning that scores can be directly compared to each other, regardless of a student’s current grade level. We initially requested only i-Ready Reading scores, but Carteret also transferred i-Ready Mathematics scores to CRRE. We analyzed i-Ready Mathematics scores in supplemental analyses, which can be found in Appendix A.

Student demographics. The dataset that Carteret Public Schools compiled for this evaluation included a set of student-level demographic variables. Indicator variables for student gender and ethnicity were present, along with indicators for economic disadvantage, special education, and English-language learner. All of these variables were recoded into dichotomous variables for analytic purposes.

AnalyticalApproach

Multiple linear regression (MLR) was used to compare i-Ready Reading achievement gains for treatment and comparison students. MLR was used instead of hierarchical linear modeling (HLM), even though condition assignment occurred at the classroom level, due to the small number of classrooms and teachers involved in this evaluation. Overall i-Ready Reading scores were used as the main outcome variable in these analyses. These models controlled for demographic variables including grade level and teacher, as well as student-level demographic variables including gender and ethnicity, along with indicator variables for ELL, SPED, and economically disadvantaged status. Descriptive statistics such as means and standard deviations were also computed for i-Ready scores, where appropriate.

Results

We begin by descriptively analyzing i-Ready Diagnostic scores from the winter and spring of the 2022-23 school year, by grade. Next, we present the results of MLRs that examine the impact of the PRM on winter to spring i-Ready score gains. Finally, we conduct subgroup analyses to examine potential differential impacts of PRM for student sub-populations of interest.

ScreenerofHandwritingProficiencyResults

Table 4 shows the unadjusted mean total pretest and posttest i-Ready scores for all students included in the current analytic sample. Scores are disaggregated by condition (treatment or comparison) and grade level.

Table 4

Unadjustedaveragei-ReadyReadingscores,bygrade

Note.SDs are in parentheses.

Patterns of gains favored treatment students in Grades 1 and 2, as treatment students averaged nearly 30-point gains, as compared to only 24-point gains for comparison students in Grade 1 and 19-point gains for comparison students in Grade 2. Comparison students outgained treatment students in kindergarten, with these students averaging 26-point gains, as compared to 20-point gains for treatment students.

Regression analyses. Next, we examine the results of multiple linear regression (MLR) analyses that examine the impact of PRM on i-Ready Reading overall score gains for all grades combined. These models controlled for student demographics and grade levels. Relevant model statistics and coefficients for the MLR examining the impact of PRM on i-Ready Reading overall score gains are displayed in Table 5.

Table 5

ImpactofPRMoni-ReadyReadingscores

205

Note.***p< .001.

Students in PRM classrooms averaged nearly 7-point larger i-Ready reading gains from winter to spring than did students in comparison classrooms. While the direction of this impact is positive, it did not reach the threshold for statistical significance (p= .108). This impact resulted in a standardized effect size of just over 0.10, indicating that PRM students outgained comparison students by an average of approximately onetenth of a standard deviation, which is a small to moderate practical impact.

Subgroup analyses. Next, we performed a set of subgroup analyses to examine potential differential impacts of PRM across different student populations. Specifically, we examined whether the impacts of PRM differed across grade levels, as well as for ELL, special education students, FARMS students, and Hispanic students.

First, we conducted subgroup analyses examining potential differential effects of PRM across different grade levels. Then, we conducted a separate subgroup analysis examining potential differential effects of PRM for each student subgroup of interest. Table 6 displays impact estimates of PRM across each grade level, and for each subgroup.

Table 6

ImpactofPRMonspring2023i-ReadyReadingscoresbysubgroup

(n= 59)

2nd (n= 84)

Subgroups

17)

(n= 115)

(n= 169)

Notes.1. ^p<.10; ** p< .01. 2. The treatment effect for each subgroup was calculated by adding the overall treatment effect and the treatment interaction terms for the subgroup. The pvalues reported in this table show whether PRM had a positive effect for the subgroup relative to similar comparison students.

A significant positive impact of PRM was observed for ELL students, with PRM ELL students outscoring ELL comparison students by 23 points. Marginally significant impacts of PRM (p< .10) were evidence for Grade 2 students, as well as for Hispanic and FARMS students. Specifically, Grade 2 PRM students outgained Grade 2 comparison students by 11 points, while Hispanic PRM students outgained Hispanic comparison students by 10 points, and FARMS PRM students outgained FARMS comparison students by 8-points. These results provide suggestive evidence supporting the efficacy of PRM in closing early literacy achievement gaps for at-risk student subgroups.

Summary & Discussion

The purpose of this study was to evaluate the efficacy of the Phonics, Reading, and Me early literacy program. The main focus was comparing early literacy gains for Grades K-2 students who participated in PRM to that of otherwise similar Grades K-2 students who did not participate in the program but were taught early literacy using other strategies and resources. This report presented quantitative early literacy achievement findings.

Results showed that students who participated in PRM outgained comparison students who did not use the program from winter 2023 to spring 2023, although this advantage did not reach statistical significance. PRM students outscored comparison students by over 6 points, on average, after controlling for pretest performance from winter 2023 and available demographic variables. Subgroup analyses showed that ELL PRM students significantly outgained ELL comparison students, while advantages for Grade 2 students, as well as Hispanic and FARMS students, approached statistical significance.

Some important limitations of this evaluation should be noted. Students and classrooms that were selected by the vendor for study inclusion were contained in one elementary school; thus, generalization of results may not apply to other school contexts or populations. No information was available regarding PRM usage or implementation in Carteret Public Schools; thus, no inferences can be made regarding program best practices or associations between program usage levels and achievement trends. The results of this evaluation overall provide suggestive positive evidence

Johns Hopkins University, 2023

regarding the efficacy of the PRM program for improving students’ early literacy achievement.

Conclusions

The key results and conclusions of this evaluation are as follows:

• PRM students scored more than 6 points higher on the spring 2023 iReady Reading assessment than did comparison students, controlling for winter 2023 i-Ready scores and demographic variables. This impact did not reach statistical significance, although the effect size was 0.11, indicating a small to medium practical impact.

• Subgroup analyses showed that PRM had significant positive impacts for ELL students, with PRM ELL students outgaining ELL comparison students by 23 points.

• PRM impacts for Grade 2 students, as well as Hispanic and FARMS students, were positive and approached statistical significance (p< 10). These advantages ranged between 8-11 points.

Table A1

Appendix A: i-Ready Math Analyses

Unadjustedaveragei-ReadyMathscores,bygrade

Note.SDs are in parentheses

Table A2

ImpactofPRMoni-ReadyMathscores

Note.***p< .001.

Table A3

ImpactofPRMonspring2023i-ReadyMathscoresbysubgroup

Note.^ p< .10; * p< .05.

Turn static files into dynamic content formats.

Create a flipbook