Phonics, Reading, and Me : Internal Evaluation
ESSA Level III Study
Prepared for:
Learning Without Tears
Prepared by LearnPlatform by Instructure:
Elizabeth Allen Green, Ph.D., Researcher
Alexandra Lee, Ph.D., Researcher
Molly Henschel, Ph.D., Associate Director of Research
March 1, 2024
EXECUTIVE SUMMARY
Learning Without Tears contracted with LearnPlatform by Instructure (LearnPlatform), a thirdparty edtech research company, to examine whether the rate of students’ literacy skill growth varies depending on their starting level (i.e., below, on, or above) while using Phonics, Reading, and Me LearnPlatform designed the study to satisfy Level III requirements (Promising Evidence) according to the Every Student Succeeds Act (ESSA).
Study Sample and Measures
The study included 402 students in kindergarten through fifth grade from multiple schools nationwide. It used data from Fall 2023 to provide insights into Phonics, Reading, and Me implementation and its relationship with student outcomes in literacy Specifically, the Learning Without Tears team provided LearnPlatform with usage data on Phonics, Reading, and Me as well as outcome data using an internal assessment
Main Research Findings
Researchers used descriptive statistics to describe participant characteristics and examine the extent of students’ use of Phonics, Reading, and Me during the Fall 2023. To examine whether students’ use of Phonics, Reading, and Me led to significant growth in skill proficiency, researchers used multi-group latent growth curve analysis with covariates (i.e., lesson difficulty and total scaffolds)
Key Findings
Students who started lessons at lower proficiency received significantly more scaffolds compared to those who started at higher proficiency levels.
On average, students’ literacy skill proficiency significantly grew over the course of the Phonics, Reading, and Me lesson sequence
On average, students who started with below-level literacy skills increased at a faster rate compared to other students over the course of the Phonics, Reading, and Me lesson sequence.
Note: These findings are statistically significant at the p < 0.05 level.
Conclusions
This study satisfies ESSA evidence requirements for Level III (Promising Evidence) given the positive, statistically significant findings.
FIGURES
Introduction
Phonics, Reading, and Me, by Learning Without Tears, is an adaptive, data-driven, supplemental learning solution for kindergarten through third grade students 1 It is an explicit and systematic instructional framework grounded in the Science of Reading research 2 It aims at helping students develop the foundational literacy skills needed to become fluent readers
As part of their ongoing efforts to demonstrate the effectiveness of Phonics, Reading, and Me, Learning Without Tears contracted with LearnPlatform by Instructure (LearnPlatform), a thirdparty edtech research company, to examine whether the rate of students’ literacy skill proficiency growth varies depending on their starting level (i.e., below, on, or above) while using Phonics, Reading, and Me. LearnPlatform designed the study to satisfy Level III requirements (Promising Evidence) according to the Every Student Succeeds Act (ESSA).
The current study had the following research questions:
Program Implementation
1. To what extent did students use Phonics, Reading, and Me3 during the Fall 2023?
a. On average, how many Phonics, Reading, and Me lessons did students complete? Which focal skills were covered during the lessons?
b. On average, how many Phonics, Reading, and Me scaffolds did students complete per lesson?
Effectiveness
2. While using Phonics, Reading, and Me, does students’ literacy skill proficiency significantly grow over the course of the lesson sequence?
a. Does the rate of students’ literacy skill growth vary depending on their starting level (i.e., below and on)?
1 Phonics, Reading, and Me is geared toward K through third grade, however this study examined K through fifth grade.
2 The Science of Reading is a research field dedicated to understanding how people learn to read and the most effective instructional approaches. The five key components are phonological awareness, phonics, fluency, vocabulary, and comprehension. This research field advocates for teaching strategies that are grounded in research and tailored to address the diverse backgrounds and needs of students (National Research. Panel, 2000)
3 Phonics, Reading, and Me includes both a print and digital format. This study focuses on the digital format.
Methods
This section of the report briefly describes the study’s setting, participants, measures, and analytical methods.
Setting
This study examined users of Phonics, Reading, and Me from multiple schools nationwide during the Fall 2023
Participants
The treatment-only sample included 402 total students across kindergarten (K) through fifth grade (90 kindergarten students, 108 first grade students, 9 second grade students, 96 third grade students, 91 fourth grade students, and 8 fifth grade students)
Measures
Researchers used Fall 2023 student-level usage (i.e., average number of lessons and focal skill covered, number of scaffolds per lesson) to inform the extent to which students used Phonics, Reading, and Me and whether its use relates to literacy skill proficiency growth depending on their starting level (below, on, or above). The Phonics, Reading, and Me internal reading assessment was used to measure students’ key foundational phonics and word study skills Students were assessed four times during a lesson. The first assessment (pretest), and each subsequent one, placed them below, on, or above-level for the lesson sequence (four total). The fourth internal reading assessment (posttest) measured students’ literacy skill proficiency at the end of the lesson When a student was placed below-level, they received a scaffolded activity that on-level and above-level students did not receive. Below-level students received scaffolded support4 for grade level texts.
Data Analysis
Researchers used a variety of quantitative analytic approaches Descriptive statistics were used to describe participant characteristics and examine the extent students used Phonics, Reading, and Me during Fall 2023
Multi-group latent growth curve analysis with covariates was used to examine whether students’ literacy skill proficiency significantly improved during a Phonics, Reading, and Me lesson sequence As part of this analysis, researchers tested a series of models to see whether a growth model better fits the data than a no-growth model. After selecting the best model for the data, researchers conducted covariate analyses to assess whether the rate of change in students’ literacy skills was associated with the lesson’s focal skill and total scaffolds used. Researchers translated the standardized slope coefficient for the latent growth curve models into percentile point estimates to translate the findings for practice.
4 Scaffolded supports are specific and personalized and include strategic rereads, audio supports, and sub-skill practice.
Program Implementation
This section presents descriptive findings related to Phonics, Reading, and Me implementation. Specifically, researchers analyzed usage data from the program to determine the extent to which students used Phonics, Reading, and Me during the Fall 2023 Appendix A provides additional information regarding implementation.
To what extent did students use Phonics, Reading, and Me during the Fall 2023?
On average, how many Phonics, Reading, and Me lessons did students complete? Which focal skills were covered during the lessons?
Over the duration of the study, students completed an average of three Phonics, Reading, and Me lessons (range: 1 to 9 total number of lessons) Table 1 provides the average number of Phonics, Reading, and Me lessons students completed across all grades, including grade level breakdowns for Fall 2023
Over the course of the study, students also covered a variety of focal skills across 14 different lessons Table 2 provides an overview of the focal skills covered during Fall 2023.
Focal Skill
Short Vowel a
Short Vowel i
Short Vowel o
Short Vowel Review: a, i, o
Short Vowel u
Short Vowel e
Initial Consonant Blends with l
Initial Consonant Blends with r
Long a Spelled a_e
Long i Spelled i_e
Compound Words with Complex Vowels
Multisyllable Words with Open and Closed Syllables
Multisyllable Words with Blends and Digraphs
Multisyllable Words That Divide Between Vowels
On average, how many Phonics, Reading, and Me scaffolds did students complete per lesson?
Over the duration of the study, students who were placed below-level after the first assessment (pretest) for each lesson, completed an average of two Phonics, Reading, and Me scaffolds over the lesson sequence. Figure 1 provides the average number of Phonics, Reading, and Me scaffolds completed per lesson for students who started below-level at the beginning of each lesson The latent growth analysis that examined whether students’ literacy skill proficiency improved during a Phonics, Reading, and Me lesson sequence, indicated that students who started lessons at lower proficiency received significantly more scaffolds compared to those who started at higher proficiency (p < .001) 5
5
Words
Words
Long a Spelled a_e Multisyllable
Words That Divide Between Vowels
Vowels
Short Vowel e
Short Vowel u
Short Vowel Review: a, i, o
Short Vowel o
Short Vowel i
Short Vowel a
Effectiveness Findings
The following section details the effectiveness findings examining whether students' literacy skills significantly improved during a Phonics, Reading, and Me lesson sequence In each model, researchers controlled for student-level covariates including lesson difficulty and total scaffolds 6 Furthermore, Researchers used a multi-growth approach to determine whether the rate of growth was faster for students starting below and on-level Additional information on these analyses and findings can be found in Appendix B Researchers reported statistically significant findings at the p < .05 level. To determine the magnitude of the relationship, researchers calculated percentile point improvement estimates using the slope coefficients.
While using Phonics, Reading, and Me, does students’ literacy skill proficiency significantly grow over the course of the lesson sequence?
Results from a latent basis growth model indicated that, overall, students’ literacy skill proficiency increased over the course of the lesson sequence (i.e., statistically significant positive slope estimate; see Figure 2) The slope estimate for this model (i.e., skills scores for students overall) is equivalent to 8 percentile point improvement; so, a student at the 50th percentile would be expected to move to the 58th percentile for each Phonics, Reading, and Me lesson sequence they complete.
6 Lesson difficulty was highly correlated with students’ grade level, so grade level was not included as a covariate due to multi-collinearity. A sum of scaffolds used over the course of the lesson sequence was calculated so that total scaffolds could be included as a covariate in the latent growth curve models.
Students had significant growth across the Phonics, Reading, and Me lesson sequence across different baseline skill levels Overall Below-level On-level
Note. Slope estimates by group: overall = .216, p < 001; below-level = .327, p < .001; on-level = .088, p < .001.
Does the rate of students’ literacy skill growth vary depending on their starting level (i.e., below and on)?
To examine if there were differences in the growth trajectories depending on students’ starting skill level, researchers used a multi-group approach and fit two separate growth models: one for students who were below-level and one for students who were on-level prior to completing the Phonics, Reading, and Me lesson sequence. The multi-group models indicated that students in both groups had significant positive growth trajectories (see Figure 2 above) The skills score slope estimate for the below-level students is equivalent to 12 percentile point improvement; so, an average below-level student at the 50th percentile would be expected to move to the 62nd percentile for each Phonics, Reading, and Me lesson sequence they complete. For on-level students, the slope is equivalent to a 3 percentile point improvement
Conclusions and Recommendations
In this study, students covered 14 focal skills and completed an average of three Phonics, Reading, and Me lessons. Students who started below-level after the first assessment (pretest) for each lesson, completed an average of two scaffolds over the lesson sequence Further, students who started the lesson sequence at a lower proficiency received significantly more scaffolds than students who started at higher proficiency. Results showed that, on average, all students who used Phonics, Reading, and Me had significant growth in their literacy skill proficiency across the lesson sequence regardless of starting levels However, students who started with below-level literacy skills increased at a faster rate compared to other students over the course of the lesson sequence.
Based on the positive findings for K through fifth grade students, this study offers results to satisfy ESSA evidence requirements for Level III (Promising Evidence). Specifically, this study met the following criteria for Level III:
Correlative design
Proper design and implementation
Statistical controls through covariates
At least one statistically significant, positive finding
In the future, Learning Without Tears could consider the following:
• Conduct another correlational study with different sites with a larger sample of students participating in Phonics, Reading, and Me
• Conduct a quasi-experimental study in a different context, with sample randomization or matching, to satisfy ESSA Level II (Moderate Evidence) requirements.
• Conduct an experimental study with sample randomization to satisfy ESSA Level I (Strong Evidence) requirements.
References
National Reading Panel (US), National Institute of Child Health, & Human Development (US). (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups. National Institute of Child Health and Human Development, National Institutes of Health. Preacher, K. J. (2018). Latent growth curve models. In The reviewer’s guide to quantitative methods in the social sciences (pp. 178-192). Routledge.
Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1-36. URL http://www.jstatsoft.org/v48/i02/
Appendix A. Additional Information on Phonics, Reading, and Me Implementation
To better understand the student usage for Phonics, Reading, and Me during the Fall 2023, researchers conducted descriptive statistics using usage data provided by Learning Without Tears See Table A1 and Table A2 for the average usage, focal skill, and ranges for all students and grade levels.
Appendix B. Additional Information on Study Findings
The following sections provide additional details regarding latent growth curve analyses examining whether students’ literacy skill proficiency significantly grows over the of Phonics, Reading, and Me
While using Phonics, Reading, and Me, does students’ literacy skill proficiency significantly grow over the course of the lesson sequence?
Researchers tested a series of latent growth curve models to see which model best fits the data (Preacher, 2018; Rosseel, 2012) According to the chi-square and Confirmatory Factor Index (CFI) statistics, the best fitting model was the latent basis model, which freely estimated the rate of change allowing for non-linearity (see Table B1)
Note Lower chi-square estimates indicate better fit whereas higher CFI estimates indicate better fit. Bolding indicates the model selected for analysis and interpretation
After selecting the latent basis model as the best model for analysis and interpretation, researchers examined the intercept, slope, and covariate estimates for three models: (1) an overall model that included all students in the sample; (2) a below-level model that included only the students who were assigned as being below-level prior to their first lesson; and, (3) an on-level model that included only the students who were on-level prior to their first lesson (see Table B2).
The models used students’ completion of the full lesson sequence (i.e., four lessons) as the usage variable, the skills scores at the four assessment time points (i.e., first assessment, second assessment, third assessment, and fourth assessment) as the outcome variable, and included two covariates (i.e., digital lesson content area and numbers of scaffolds). All three models had statistically significant positive slope estimates, which indicated that students had significant growth in their skill proficiency overall, below-level, and on-level.
Latent basis model (overall)
Latent basis model (below-level)
Latent basis model (on-level)
Note Intercept is the model estimated initial skill level, slope is the growth estimate across the four time points. Every model included two covariates: (1) lesson level which is the difficulty of the lesson skill covered (i.e., short vowels (CVC), short vowels with blends, long vowels with silent e, multisyllabic words); and (2) scaffold sum which is the total number of scaffolds completed over the course of the lesson sequence.