SCHOOL Readiness 2018-2019 Program Evaluation Executive Summary Prepared by: CNM • AWare Research Solutions
Camp Fire First Texas + 817.831.2111 + CampFireFW.org
EXECUTIVE SUMMARY School readiness is a multi-faceted concept that refers to the early skills, knowledge, behaviors, and developmental milestones that prepare children to enter and succeed in school.1,2 Empirical studies suggest that children enrolled in high-quality early education and development programs tend to be more prepared with the skills and knowledge needed to enter and succeed in school.3, 4 The Camp Fire School Readiness Program (CFSRP) was established in 2005 and is designed to improve children’s school readiness by improving the quality of child development centers in target neighborhoods within the Fort Worth Independent School District (FWISD). Using rigorous, comprehensive, research-based programs, the CFSRP provides professional development, coaching and mentoring for directors and teachers, as well as support for family engagement activities. Although comprehensive, the program’s primary focus is on promoting children’s social-emotional and early language/literacy development.
Evaluation Objectives The evaluation includes four objectives that address both the process and the outcomes of implementing the CFSRP in the targeted child development centers. • To assess the extent to which the CFSRP is implemented with fidelity. This ‘process’ portion of the evaluation was added in the 2018-2019 school year to help Camp Fire staff identify program elements that might need continued and/or increased focus and support. • To assess the program’s effectiveness in improving children’s developmental skills, particularly early literacy and social-emotional skills. • To assess the program’s effectiveness in improving center and classroom quality. • To assess the impact of the program on children’s school readiness and academic success in FWISD schools
1,177 Children
Key Findings: CFSRP Demographics During the 2018-2019 program year, the Camp Fire School Readiness Program served 1,177 children, with 168 Center teachers, in 87 classrooms, across 19 Centers. Twenty-four percent of children served were infants (0-18 months), 36% were toddlers (19-36 months), 23% preschool (3-year-olds), 17% prekindergarten (4- and 5-year-olds). Over half (56%) of the children served were identified as African American.
74% were retained for the full program year
24 %
36 %
40 %
Infants
Toddlers
Preschool & Pre-K
1
Karoly, L.A., Kilburn, M.R., Cannon, J. (2005). Children at risk: Consequences for school readiness and beyond (Research Brief). Santa Monica, CA: RAND Corporation. Retrieved from http://www.rand.org/pubs/research_briefs/RB9144.html.
2
Texas Early Learning Council. (2011). Defining school readiness: National trends in school readiness definitions. Retrieved from http:// earlylearningtexas.org/media/10138/trends%20in%20school%20readiness%20final%2011-1.pdf.
3
Pears, K.C., Fisher, P.A., Kim, H.K., Bruce, J., Healey, C.V., & Yoerger, K. (2013). Immediate effects of a school readiness intervention for children in foster care. Early Education and Development, 24(6), 771-791.
4
Barnett, W.S., & Yarosz, D.J. (2007). Preschool policy brief: Who goes to preschool and why does it matter? Retrieved from http://nieer.org/ policy-issue/policy-brief-who-goes-to-preschool-and-why-does-it-matter-updated.
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
2
Key Findings: Teacher Retention The turnover rates of early childhood teachers at child development centers are four times higher than the rates observed among elementary school teachers.5 The annual retention rate for center-based child care professionals is estimated to be between 60% and 70%.6, 7 Of the 168 teachers employed during the 2018-19 school year, 132 worked from BOY to EOY for a 79% retention rate (see Figure 1). Figure 1: Teacher Retention (Percentage of Teachers Employed for the Full Year) 100% 86% 79%
79%
80%
71%
70%
64%
60%
60% 54%
40%
20%
0%
2013-2014
2014-2015
2015-2016
2016-2017
2017-2018
2018-2019
Key Findings: CFSRP Program Implementation • Professional Development enhances teacher and director performance, but participation showed room for improvement. Teachers’ use of Professional Development was below expected levels, but the majority of those who did participate engaged at levels that made them eligible for a full stipend. Directors reported that the professional development opportunities provided by the CFSRP improved their leadership and management practices (Figure 2). Figure 2: Director Ratings of Professional Development and Stipend, 2018-2019
Director Experiences with Professional Development and Stipend 5.3
Extent stipend motivated directors to participate in PD Extent directors think stipend motivated teachers to participate in PD
9.2
8.0
Center's level of participation in the CFSRP
8.6
PD helped improve quality of management skills
8.3
PD helped improve quality of leadership 0
1
2
3
4
5
6
7
8
9
10
5
Whitebook, M. & Sakai, L. (2003). Turnover begets turnover: An examination of job and occupational instability among child care center staff. Early Childhood Research Quarterly, 18, 273-293.
6
Whitebook, M. & Sakai, L. (2003). Turnover begets turnover: An examination of job and occupational instability among child care center staff. Early Childhood Research Quarterly, 18, 273-293.
7
Baumgartner, J.J., Carson, R.L., Apavaloaie, L., & Tsouloupas, C. (2009). Uncovering common stressful factors and coping strategies among child care providers. Child and Youth Care Forum, 38, 239-251.
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
3
• Mentorship was a valuable support for teachers and directors and CFFT is working to ensure that mentor focus remains on target. Directors rated their mentoring experiences with the CFSRP positively, most specifically helping with challenges they may face with staff and helping in areas in which they want support. When providing support to teachers, mentors spent the majority of Center visit time conducting observations and providing support with instructional planning (Table 1). Mentors also spent time helping teachers with assessments; while important, the amount of time spent on this activity was more than expected. Anecdotal evidence suggests that mentors were spending time teaching teachers how to administer assessments and then mentors were spending time entering data from assessments and managing the data collection process. The CFSRP will investigate this finding further to determine if more mentor development is needed. Table 1: One-on-One Mentoring Activities
Type of Visit Activity
Number of Visit Activities
Observation Instructional Planning Help with Child Assessments Reflective Follow Up Modeling Side-by-Side Coaching Problem Solving Generate Options Total
128 94 64 54 27 23 18 408
Percent of Visit Activities 31.37% 23.04% 15.69% 13.24% 6.62% 5.64% 4.41% 100.00%
Key Findings: Child Outcomes • The results of the outcome evaluation demonstrated that the CFSRP is having positive results for both infant and toddler groups, but showed some areas for growth with preschool-aged children. • The majority of infants either improved or continued to perform on target in each domain (i.e., Communication, Fine Motor, Gross Motor, Personal-Social, and Problem Solving) (Figure 3). • Toddlers performed equally well, with all end-of-year goals met for development and a large majority of toddlers improving or continuing to perform on target in each area (Table 2 and Figure 4). • Preschool children (ages 4 and 5 years old) did not meet goals for development at the end of the year; however, the percentage of 4- and 5-year-olds making acceptable progress at the beginning- and end-ofyear increased across all domains (Table 3 and Figure 5). • All children, infant through preschool, showed an improvement in social-emotional well-being with an increase in the percentage of children in the “strengths” category, which means the child displayed social and emotional strengths. Infants and toddlers also showed a decrease in the percentage of children in the “needs” category. Children in the “needs” category are in need of instruction to further develop social and emotional skills. Preschool children had a small increase in the “needs” category (Figure 6).
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
4
Table 2: CFSRP Infant and Toddler Developmental Outcomes: Targets Versus Actual % of Children Developmentally On-Target at End-of-Year Infants (N=89) Toddlers (N=180)
ASQ Developmental Domain
Target
EOY Actual
Target
EOY Actual
Problem Solving Skills
70%
78%
80%
83%
Communication Skills
65%
63%
80%
84%
Gross Motor Skills
75%
76%
85%
89%
Fine Motor Skills
70%
72%
70%
76%
Personal-Social Skills
70%
75%
80%
88%
Figure 3: Age-Appropriate Infant Development, Change from BOY to EOY, 2018-2019 (N=89) Improved/Maintained Age Level Declined
100%
Maintained Below Age Level
90%
Percent of Children
70%
80%
79%
80%
78%
76%
67%
60% 50% 40% 30%
21%
20% 10% 0%
19%
17%
11% 4% Communication
18% 6%
1%
Fine Motor
15% 8%
Gross Motor
Personal-Social
Problem Solving
Figure 4: Age-Appropriate Toddler Development, Change from BOY to EOY, 2018-2019 (N=180) Improved/Maintained Age Level Declined
Maintained Below Age Level
100% 90%
90%
89%
86%
86%
81%
80%
Percent of Children
70% 60% 50% 40% 30% 20% 10% 0%
16% 6%
8%
Communication
10% 3% Fine Motor
9%
1%
2%
Gross Motor
Personal-Social
10% 4% Problem Solving
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
5
Table 3: CFSRP Preschool Developmental Outcomes: Targets Versus Actual % of Children Developmentally On-Target at End-of-Year
CPALLS+ Developmental Domain
Preschool 4- and 5-Year-Olds (N=56) Target
EOY Actual
Listening
90%
84%
Rhyming I
80%
70%
Rhyming II
60%
48%
Math
95%
94%
Figure 5: CPALLS+, Percentage of 4- and 5-Year-Old Children Improving or Making Acceptable Progress from BOY to EOY, 2018-2019 (N=56) 90%
79%
Percent of Children
80% 66%
70% 60%
48%
50%
43%
40% 30% 20%
32% 18% 9%
10% 0%
21%
15% 7%
4%
Listening Improved
23%
2%
Math
16% 7%
Rhyming
No Change: Making Acceptable Progress
9%
Rhyming 2
No Change: Needs More Assistance
Declined
Figure 6: Social Emotional Development at BOY and EOY, 2018-2019 100% 90% 80%
10.0% 26.7%
17.6%
21.3%
10.1%
11.3%
70% 56.7%
60% 50% 40%
39.8% 47.2%
71.4%
63.1%
63.3%
30% 20% 10% 0%
33.3%
42.6%
31.5%
18.5%
10.0% BOY
EOY
Infants (N=30)
BOY
EOY
Toddlers (N=108) Strength
Typical
BOY
25.6% EOY
Preschool (N=168)
Need
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
6
• The CFSRP continues to contribute positively to children’s kindergarten readiness. As in prior years, prekindergarten and kindergarten children entered FWISD with significantly higher ratings than a demographically matched comparison group on language and literacy skills assessed by the Texas Kindergarten Entry Assessment (TX-KEA) (Figures 7 and 8). Figure 7: Fall 2019 Comparisons of Pre-Kindergarten Literacy Ratings (CIRCLE Assessment)
Figure 8: Fall 2019 Comparisons of Kindergarten Language and Literacy Ratings (TX-KEA)
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
7
• The CFSRP continues to contribute to children’s academic success through the first grade. In each of the past seven years. FWISD kindergarten and first grade children who had attended one of the CFSRP centers had higher reading and math scores on standardized tests than the demographically similar comparison groups Figures 9 through 12). Figure 9: Comparisons of Kindergarten Reading Achievement (CFSRP and Comparison Groups; ITBS 2016-2019) 8
• The differences fade as the children progress to the second and third grade. Second grade CFSRP children have had similar math and reading scores as the comparison group on standardized tests four years and third grade CFSRP and comparison group students have also similar reading and math scores on the for the past three years on the State of Texas Assessment of Academic Readiness (STAAR).
Key Findings: Center Outcomes • Classroom-level care giving, support, and environment varied by age group and Center level. When measuring classroom-level care giving, support, and environment, infant and toddler classrooms showed growth from beginning- to end-of-year. Preschool classrooms struggled in these areas, showing decreases in classroom scores from beginning- to end-of-year (Figure 13). As to be expected, center level differences existed in some cases, with Level 2 Centers performing at lower levels and Level 4 Centers performing at higher levels. • Leadership and management quality at CFSRP Centers improved from beginning- to end-of-year. A majority of CFSRP Centers met the performance goals for Staff Development, Program Evaluation, and Family Support and Involvement domains (Table 4). There was room for growth when it came to Staff Orientation, with a minority of Centers meeting performance goals, and Family Partnerships, with scores coming in just below the “Good Quality” threshold (Figure 10). Again, as expected, Level 4 Centers had the strongest performance and Level 2 had the lowest; but Level 2 Centers also showed growth from beginning- to end-of-year.
8
CFSRP kindergarten children included in these analyses would have attended a CFSRP center either one year earlier as 4-year olds or two years earlier as 3-year olds. (CFSRP N=150 in 2016, 107 in 2017, 108 in 2018,105 in 2019; COMP N=552 in 2016, 739 in 2017, 553 in 2018, 523 in 2019).
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
8
Table 4: PAS Performance – Score Increase and Quality Threshold PAS Domain
Staff Orientation Staff Development Program Evaluation Family Support and Involvement
Target Increase 0.5 0.5 1.0 0.5
Overall Actual Increase 0.75 2.25 1.09 1.12
Score of 5 or Above No Yes No Yes
Center-Level Performance At Least 1-Point Score of 5 or Increase Above 3 out of 7 (43%) 2 out of 7 (29%) 6 out of 7 (86%) 6 out of 7 (86%) 8 out of 11 (73%) 6 out of 11 (55%) 7 out of 9 (78%) 8 out of 9 (89%)
NOTE: Green shading indicates that the CFSRP met that goal. Two Centers did not have both a BOY and EOY score for comparison for Staff Orientation and Staff Development.
Figure 10: Center Leadership and Management Practices at BOY and EOY, 2018-2019 7.00 6.50
Average Score
6.00
5.56
5.00 4.00
4.27
4.25 3.50
3.00
4.44
3.18
2.75 2.00 1.00
Staff Orientation
Staff Development
Human Resources Development (N=8)
BOY
Program Evaluation
Family Support and Involvement
Program Planning and Evaluation (N=11)
Family Partnerships (N=9)
EOY
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
9
RECOMMENDATIONS The following recommendations are based on the findings of the 2018-2019 evaluation. Professional Development: • Simplify the professional development program design so that it is easier for Center teachers and directors to understand and follow requirements. • Continue to investigate the influence of a stipend of teacher and director professional development participation. • Increase the requirements of professional development participation – move all teachers (both FT and PT) through PD. • Add accountability measures to ensure teacher and director participation. • Add additional supports such as transportation and childcare assistance to increase participation. Mentoring: • Formalize the mentoring requirements and implement accountability measures (i.e., further investigating the amount of mentoring and type of mentoring activities completed and what guidelines the CFSRP should adopt to reflect best practices in the area). • Provide additional training and support for teacher-mentors. For example, providing targeted mentoring for teachers in multi-age classrooms. Child Outcomes: • Target preschool outcomes in professional development sessions. In addition, mentors should pay particular attention to preschool classrooms while out doing Center visits. Take caution that the increased focus on preschool outcomes does not detract from any necessary focus on infant and toddler outcomes. • Further investigate Center Level differences in child outcomes and apply professional development and mentoring support where needed to maintain positive results over time. • Continue use of social emotional tools and professional development that inform both teacher and family interactions with children (i.e., DECA and related support activities). Center Outcomes: • Continue to focus on classroom environment and management in infant, toddler, and preschool classrooms. Specifically, mentors can be focusing on this area during their Center visits. The CFSRP should determine whether specific professional development sessions focused on each domain would be beneficial. • Provide professional development and oversight to Center directors related to staff orientation, program evaluation, and family communication. Determine if specific resources may be beneficial. In addition, mentors should target these areas when working directly with directors at their Centers. • Further investigate Center Level differences in Center outcomes and apply professional develop and mentoring support where needed to maintain positive results over time. • Further investigate the impact of Universal Pre-K and multi-age classrooms and create appropriate PD and mentoring responses to that information Evaluation Procedures: • Continue to increase and improve the tracking of process evaluation outputs. • This includes increasing data entry compliance with teacher and Center identification numbers and tracking teacher full-time or part-time status. • Continue to focus on administering the CFSRP-identified assessments at the set time of year, regardless of impact on match status. • Further investigate Center Level differences in child and Center outcomes and apply professional development and mentoring support where needed to maintain positive results over time. Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
10
SCHOOL Readiness CAMP FIRE FIRST TEXAS
2018-2019
Participant Demographics
1%
Asian
14%
Hispanic
Caucasian
21%
1,177 Children 74% were retained for the full program year
8% Multiracial
African American
56%
24 %
36 %
40 %
Infants
Toddlers
Preschool & Pre-K
49 % Male
201
51 %
Directors & Teachers
25
Level 2 Basic
Female
60% Have only a high school diploma or GED
15% Have less than
25%
1 year experience
Have 11 or more years experience
26%
45
Some College
Level 3 Excelling
Experience in the field
12%
17
79%
Program teacher retention rate
Have an Associate’s, Bachelor’s or Master’s degree
36%
Have 7-10 years experience
12%
Level 4 Sustaining
12%
Have 1-3 years experience
Have 4-6 years experience
87
Classrooms
have 80% of3+ teachers years in the CFSRP
* 2% Not Reported
Camp Fire School Readiness Program | 2018-2019 Program Evaluation Executive Summary
11