Camp Fire School Readiness & Early Education Apprenticeship Program Evaluation Report 2020-2021

•

•

•

•

•

•

•

•

•

•

49 CONCLUSION 51 LIST OF APPENDICES

52 APPENDIX A: CFSRP Program Components

53 APPENDIX B: Evaluation Methods

54 APPENDIX C: Child Development and Center Quality: Assessment Tools and Data Analysis

56 APPENDIX D: Data Analysis and Assessment Tools for FWISD Data

59 APPENDIX E: CFSRP and Comparison Group Demographics (FWISD, Fall 2022)

61 APPENDIX F:

Year-to-Year Comparisons of Prekindergarten Readiness Results (Fall 2017 – Fall 2022)

62 APPENDIX G:

Year-to-Year Comparisons of Kindergarten Readiness Results (Fall 2018 – Fall 2022)

•

•

•

•

63 APPENDIX H:

Year-to-Year Comparisons of Kindergarten Fall MAP Reading Fluency Results (Fall 2020 – Fall 2022)

64 APPENDIX I:

Year-to-Year Comparisons of Kindergarten Fall MAP Reading Growth Results (Fall 2020 - Fall 2022)

65 APPENDIX J:

Year-to-Year Comparisons of 3rd Grade STAAR Results (Spring 2017 - Spring 2022)

66 APPENDIX K: Early Education Apprenticeship Tracks 67 PROGRAM DEMOGRAPHICS

List of Figures

12 Figure 1: CFSRP Theory of Change

19 Figure 2: Teacher Retention Rates from BOY to EOY, 2013-2022

19 Figure 3: Student Retention Rates from BOY to EOY, 2013-2022

23 Figure 4: Percentage of Infants

Demonstrating Age-Appropriate Skills by Assessment Area (ASQ ®3), 2021-2022 (N=51)

23 Figure 5: Percentage of Toddlers

Demonstrating Age-Appropriate Skills by Assessment Area (ASQ ®3), 2021-2022 (N=104)

24 Figure 6: Percentage of 4- and 5-YearOld Children Improving or Making Acceptable Progress from BOY to EOY (CPALLS+), 2021-2022 (N=12)

24 Figure 7: Percentage of 4- and 5-YearOld Children Improving or Making Acceptable Progress from BOY to EOY (CPALLS+), 2021-2022 (N=12)

25 Figure 8: Social Emotional Development at BOY and EOY (DECA), 2021-2022

27 Figure 9: CFSRP Classroom

Environment and Management at BOY and EOY (CLASS™), 2021-2022

28 Figure 10: Percentage of CFSRP Preschool Classrooms at or Above the Preschool Quality Threshold (CLASS™), 2021-2022

28 Figure 11: Center Management Scores at BOY and EOY (PAS), 2021-2022

29 Figure 12: Comparisons of Prekindergarten Early Literacy Skills (Fall 2022)

31 Figure 13: Comparison of Kindergarten Early Literacy Skills (TX-KEA, Fall 2022)

31 Figure 14: Comparison of Kindergarten Social Competence Skills (TX-KEA, Fall 2022)

32 Figure 15: Comparison of Kindergarten Language Skills (MAP ® Fluency™, Fall 2022)

32 Figure 16: Comparison of Kindergarten Decoding Skills (MAP ® Fluency™, Fall 2022)

32 Figure 17: Comparison of Kindergarten Overall Achievement (MAP ® Reading Growth™, Fall 2022)

32 Figure 18: Comparison of Kindergarten Foundational Skills (MAP ® Reading Growth™, Fall 2022)

List of Tables

13 Table 1: Description of CFSRP Professional Development Levels

14 Table 2: CFSRP Participating Centers, 2021-2022

15 Table 3: Number of Centers, Classrooms, Staff, and Children served by the Camp Fire School Readiness Program, 2009-2022

16 Table 4: CFSRP Student Characteristics, 2020-2021 (N=772)

17 Table 5: CFSRP Staff Characteristics, 2021-2022 (N=150)

17 Table 6: CFSRP Mentor Characteristics, 2020-2021 (N=6)

20 Table 7: Mentor Activities, 2021-2022

22 Table 8: CFSRP Infant and Toddler Development Outcome Goals Versus Actual (ASQ ® -3), 2021-2022

24 Table 9: CFSRP Prekindergarten Development Outcome Goals versus Actual (CPALLS+), 2021-2022

32 Figure 19: Comparison of Kindergarten Comprehension Skills (MAP ® Reading Growth™, Fall 2022)

34 Figure 20: Percent of Students At or Above Average Reading Achievement Levels MAP ® Reading Growth™ (Spring 2022)

35 Figure 21: Percent of Students Meeting Reading Growth™ Targets (MAP ® Reading Growth™, Spring 2022)

35 Figure 22: Percent of Students At or Above Average Reading Growth™ (MAP ® Reading Growth™, Spring 2022)

39 Figure 23: EEAP Theory of Change

44 Figure 24: Percentage of Apprentices Showing Increased Classroom Skill (BPOT), 2021-2022

44 Figure 25: EEAP Classroom Environment and Management at BOY and EOY (CLASS™), 2021-2022

45 Figure 26: Percentage of EEAP Preschool Classrooms at or Above the Preschool Quality Threshold (CLASS™), 2021-2022

27 Table 10: CLASS™ Domain Descriptions

40 Table 11: EEAP New Apprentice Characteristics (N=29), 2021-2022

41 Table 12: EEAP Host Sites, 2021-2022

42 Table 13: Apprentice Program Progression, by Cohort

43 Table 14: Apprentice Reasons for Leaving EEAP before Completion, 2021-2022

46 Table 15: Apprentice Career Outcomes, by Cohort

Executive Summary

Background

Camp Fire First Texas has been committed to serving young children, their families, and the North Texas Community for over 100 years. Two prominent avenues for strengthening early childhood services and supports that Camp Fire First Texas has pioneered are the Camp Fire School Readiness Program (CFSRP) and Camp Fire Early Education Apprenticeship Program (EEAP). Through these initiatives, children and educators access highquality programming and support. CFSRP aims to strengthen child school readiness (i.e., literacy, developmental, and social-emotional growth) through educator professional development and individualized mentoring. EEAP provides a career and educational pathway for early childhood educators by offering financial incentives, professional development, individualized incentives, and connection to higher educational institutions.

Report Structure

This report includes evaluation of both CFSRP and EEAP. Each program is treated separately, allowing readers to examine the results of each program separately, or to consider their results jointly. Evaluation of each program addresses specific questions for the 2021–2022 school year:

SCHOOL READINESS PROGRAM

To what extent did the CFSRP…

1. Implement professional development, stipend allocation, and mentorship activities as intended?

2. Improve the percentage of children demonstrating ageappropriate developmental, early literacy, and social-emotional skills during the 2021-2022 program year?

3. Impact children’s growth in developmental, early literacy, and social-emotional skills during the 2021-2022 program year?

4. Enhance the quality of teaching, classroom management, and centers’ family engagement practices during the 2021-2022 program year?

5. Impact CFSRP children’s school readiness and academic success as they enter prekindergarten and kindergarten and progress to the third grade?

EARLY EDUCATION APPRENTICESHIP PROGRAM

1. What differences in participant retention and success are evident?

2. How did EEAP components collectively support educators’ knowledge and skills?

3. To what extent are EEAP participants attaining (and maintaining) portable child care credentials and wage increases?

EEAP Results

PARTICIPANT RETENTION AND SUCCESS

The first and pilot EEAP cohort ran from 2020-2022, enrolling 23 and graduating 12 apprentices. A second cohort began in 2021 and graduated 12 apprentices of 29 in the first year. Across both cohorts, nearly 75% of apprentices either graduated within 12 months or successfully completed the first year and began the second. Across the first and completed cohort, 52% of apprentices completed the program; Camp Fire expects higher completion rates among future, non-pilot

KNOWLEDGE AND SKILL DEVELOPMENT

Apprentices were expected to improve the teaching, child interaction, and classroom management skills associated with high-quality early childhood classrooms. In the 2021-2022 school year, 22 of the 24 classrooms with apprentices showed improvement in teaching best practices over the course of the school year, and classrooms generally showed strong growth in classroom best practices. Apprentice classrooms met the quality threshold for classroom organization, with additional growth desired for emotional and instructional support. Apprentices perceived the program as having a major effect on their professional (ability as a teacher, confidence in their professional skill) and personal (ability as a family member or parent, confidence in their personal abilities) lives and as impacting their ability to seek professional opportunities. All graduates have indicated the intention to pursue additional steps to build their educational and career attainment.

CREDENTIAL AND WAGE INCREASES

Since its beginning in 2020, EEAP has aided 17 early childhood educators in obtaining a Child Development Associate (CDA) credential and graduated 24 apprentices. Apprentices who completed the program by August 2022 experienced notable wage growth of up to an average of $2.90/hour, with average wages of up to $15.42. As EEAP matures, the program increasingly connects apprentices with higher education institution; 10 of the 12 graduates from the 2021-2022 cohort have been referred to either Tarrant County College or Tarleton State University for further education, supported by scholarship funds.

Overall, both the CFSRP and EEAP have achieved notable steps in strengthening the early childhood system and supports available in North Texas, in spite of persistent challenges. Continued investment in efforts to support young children and the educators who help them thrive can build on the accomplishments and innovations of Camp Fire

Glossary of Terms

ASQ®-3 Ages and Stages Questionnaires®, Third Edition

A developmental screening tool for use with children ages one month to 5 years old. The ASQ®-3 highlights a child’s developmental strengths and areas of concern. Camp Fire uses ASQ®-3 to see where individual children need additional support, assess development over time, and identify developmental domains to emphasize in the classroom.

BOY Beginning of Year

Generally refers to measurements taken at the beginning of a school year (September through November).

BPOT Best Practices Observation Tool

A research-based checklist for classroom observers to measure the presence or absence of teaching practices that align with the CFSRP professional development curriculum. Checklists are tailored to classroom type (infant, toddler, Pre-K3 and Pre-K4).

CFSRP Camp Fire School Readiness Program

Camp Fire First Texas’ research-based initiative to strengthen school readiness through improving language and socialemotional skills among children age 0-5 and enhancing the quality of teaching and childcare centers.

CIRCLE Circle Progress Monitoring Tool

A criterion-referenced assessment used to identify prekindergarten students who are on track, need monitoring, or need support for development of early literacy and social emotional skills.

CLASS™ Classroom Assessment Scoring System

A quality improvement tool focused on teacher-student interactions that measures the teaching quality within a classroom. Camp Fire uses CLASS™ to gauge classroom quality and provide individualized feedback to improve educator practices.

CPALLS+

An assessment of prekindergarten learning (literacy, math, science, social studies), social development, and critical Head Start skills. Camp Fire uses CPALLS+ to assess older children’s language and math skill development over time.

DECA Devereux Early Childhood Assessment

A strengths-based social-emotional screening and assessment tool for infants and toddlers (ages one month to 36 months) and preschoolers (ages three to five years). Camp Fire uses DECA to assess children’s social-emotional wellbeing and level of protective factors.

EEAP

Camp Fire Early Education Apprenticeship Program

Camp Fire First Texas’ workforce development initiative to build a career pathway for early childhood educators by combining paid, on-the-job learning with coaching, professional development, and professional certification.

EOY End of year

Generally refers to measurements taken at the end of a school year (April through June).

FWISD Fort Worth Independent School District

The K-12 public school system that serves most students after they age out of the CFSRP.

MAP® Reading Fluency™

An online screening and progress monitoring tool that assesses basic reading skills with an emphasis on oral fluency (e.g., listening comprehension, words per minute, accuracy, decoding). It is part of a standardized, norm-referenced series of assessments that can be used to measure students’ performance against the performance of a national sample. Students receive a MAP® Reading Fluency™ rating of below, approaching, meeting, or exceeding grade level expectations.

MAP® Growth™

An online screening and progress monitoring tool that assesses early literacy skills, reading comprehension, and use of vocabulary. It is part of a standardized, norm-referenced series of assessments that can be used to measure students’ performance against the performance of a national sample. Students receive a score that places them at one of five levels (low, low average, average, high average, or high), based on the national sample.

MOY Middle of year

Generally refers to measurements taken in the middle of a school year (January or February).

PAS Program Administration Scale

A leadership and program management assessment tool designed for center-based early childhood education programs. Camp Fire uses the PAS to assess center quality and support continuous improvement.

TRS Texas Rising Star

The Texas Rising Star program is a quality rating and improvement system (QRIS) for child care programs participating in the Texas Workforce Commission’s (TWC) Child Care Services program. Source: texasrisingstar.org/about-trs

TX-KEA Texas Kindergarten Entry Assessment

An assessment of entering Texas kindergarten students’ language, literacy, STEM, social emotional, executive function, and academic motor skill. Kindergarten teachers in Texas administer the TX-KEA to their students at the beginning, middle, and end of the school year.

Introduction

Camp Fire First Texas’ commitment to youth development begins right from the start, with early childhood. By strengthening the systems, programs, and professionals that support infants and young children, Camp Fire invests in strong starts for children across Fort Worth and beyond.

Since 2005, the Camp Fire School Readiness Program (CFSRP) has helped children be ready to start school through improving quality of care at neighborhood childcare centers. CFSRP supports early education programs that feed into the Fort Worth Independent School District with professional development and individual coaching/mentoring focused on teaching practices that improve language and social-emotional skills among children age 0-5.

In 2020, Camp Fire began its Early Education Apprenticeship Program (EEAP), a 1-2 year program combining paid on-the-job learning, coaching, professional development, and professional certification to strengthen career and educational pathways for early childhood professionals. EEAP is an innovative approach to a critical need: stabilizing the workforce through improved wages and decreased turnover. The first U.S. Department of Labor-certified early childhood apprenticeship program in the state of Texas, EEAP is supported by the Texas Workforce Commission, Workforce Solutions for Tarrant County, Tarrant County College, Tarleton State University, and T.E.A.C.H. Early Childhood Texas.

Each year, Camp Fire prepares an evaluation report to share with partners and the community. The evaluation provides opportunities to reflect on challenges and successes in the preceding year, make adjustments to more effectively support children’s education, and have transparent, productive conversations across the community. The report separately addresses the activities and achievements of CFSRP and EEAP, enabling readers to consider each program independently as well as jointly.

CFSRP

CAMP FIRE SCHOOL READINESS PROGRAM

The CFSRP evaluation summarizes literacy, developmental, and social-emotional growth among children attending childcare centers supported by CFSRP in 2021-2022, center and classroom quality measures, and academic outcomes through third grade. It addresses five central evaluation questions. To what extent did the CFSRP:

1. Implement professional development, stipend allocation, and mentorship activities as intended?

2. Improve the percentage of children demonstrating ageappropriate developmental, early literacy, and socialemotional skills during the 2021-2022 program year?

3. Impact children’s growth in developmental, early literacy, and social-emotional skills during the 2021-2022 program year?

4. Enhance the quality of teaching, classroom management, and centers’ family engagement practices during the 2021-2022 program year?

5. Impact CFSRP children’s school readiness and academic success as they enter prekindergarten and kindergarten and progress to the third grade?

EEAP

EARLY EDUCATION APPRENTICESHIP PROGRAM

The EEAP evaluation summarizes participation and outcomes for the first full cohort to complete apprenticeships since EEAP was instituted in 2020. Central evaluation questions for EEAP include:

1. What differences in participant retention and success are evident?

2. How did EEAP components collectively support educators’ knowledge and skills?

3. To what extent are EEAP participants attaining (and maintaining) portable child care credentials and wage increases?

Camp Fire School Readiness Program

Healthy development in early childhood supports lifelong health, academic achievement, economic stability, and community strength. High-quality early childhood care and education are foundational for social, cognitive, and developmental growth that lead to kindergarten readiness.1 Nowhere else are the benefits of high-quality early care as pronounced as among children from low-income households, who are at increased risk of entering kindergarten unprepared,2 with lasting disadvantages,3 but with strong potential to close gaps when supported in the first years of life.4 The CFSRP addresses this need by partnering with childcare centers in targeted areas of Fort Worth to support educators with the knowledge and skills to provide quality care to neighborhood children. This solution meets families and neighborhoods where they are, enhancing quality care without displacing current providers or developing new childcare centers.

The CFSRP measures its success in terms of child learning and development, classroom quality, and center quality at the beginning and end of each school year, as shown in the CFSRP theory of change5 (Figure 1). The components of the program are outlined in detail in Appendix A. To understand how and why these outcomes came about, the evaluation includes program implementation indicators: teacher and student retention, teacher participation in professional development, and mentoring activities.

Camp Fire School Readiness Program

Center/Teacher Outcomes

(FIRST-ORDER)

Child Outcomes

(SECOND-ORDER)

Teachers improve teaching practices

Improved language and socialemotional skills in CFSRP children

Directors improve leadership and management practices

CFSRP children are prepared to enter school

Improved center and classroom quality

CFSRP children demonstrate academic success in school

1 Snow, K. L. (2006). Measuring school readiness: conceptual and practical considerations. Early Educ. Dev. 17, 7–41. doi: 10.1207/s15566935eed1701_2

2 Reardon, S. F. & Portilla, X. A. (2016). Recent trends in income, racial, and ethic school readiness gaps at kindergarten entry. AERA Open 2(3), 1-18. doi: 10.1177/2332858416657343

3 Garcia, E. & Weiss, E. (2017). Education inequalities at the school starting gate. Economic Policy Institute. epi.org/132500

4 Magnuson, K.A., Meyers, M.K., Ruhm, C.J., & Waldfogel, J. (2004). Inequality in preschool education and school readiness. American Educational Research Journal. 41(1):115-157. doi:10.3102/00028312041001115

5 A theory of change provides an illustration of a program’s impact pathway—the logical causal change that is expected to occur as a result of program activities.

Figure 1: CFSRP Theory of Change

Program Characteristics

PROGRAM TIERS

The CFSRP is designed to operate over multiple years, bringing center partners in to first learn the ropes (Level 2), then commit to intensive professional development and mentoring (Level 3), and then receive ongoing professional development support to maintain quality over time (Level 4). By 2021-2022, CFSRP matured to the point of having a majority of centers and classrooms at the sustainability level (Table 1).

LEVEL 2 1 (4)

RELATIONSHIP BUILDING BETWEEN CFSRP AND THE CENTER

Does not include professional development.

BASIC

Center participates for one year

INTENSE

LEVEL 3 3 (17)

Center participates for three years

SUSTAINABILITY

LEVEL 4 8 (29)

Center participation begins after the third, intensity-level year and continues as long as the center remains in the program

1: Description of CFSRP Professional Development Levels

CENTER CHARACTERISTICS

In the 2021-2022 school year, the CFSRP served 12 child development centers, reaching 150 staff members and 726 children in targeted areas of Fort Worth. A majority of participating centers were engaged in CFSRP Level 4 (56%). Of the 12 child development centers, one participated in Level 2 programming, three in Level 3 intensive support, and eight in Level 4 sustainability work. Nine centers (75%) also participated in Texas Rising Star (TRS) during the 2021-2022 school year; each of those centers rated either three or four stars (Table 2).

Over time, the number and type of child care providers supported through the CFSRP has shifted. At its inception in 2009, CFSRP served home-based and center-based providers. Starting in 2016, the CFSRP exclusively served center-based providers, helping them to enhance the quality of the education they provide, strengthen classroom management and teacherchild interactions, and advance their Texas quality rating and improvement rating through TRS. The number of classrooms served in a year provides a strong indicator of CFSRP investment of time and money. The classrooms participating in the CFSRP in 2021-2022 were primarily single age group classrooms (92%). Following a trend from the 2020-2021 school year, the number of classrooms participating in CFSRP in 2021-2022 decreased somewhat, as four centers left the program, decreasing the number of classrooms served. The decline in classrooms served is attributable to center closures due to staff strain and persistent low enrollment associated with COVID-19, as well as to shifting partner priorities.

Table 2: CFSRP Participating Centers, 2021-2022

Table 3: Number of Centers, Classrooms, Staff, and Children served by the Camp Fire School Readiness Program, 2009-2022

STUDENT CHARACTERISTICS

Center recruitment to CFSRP is designed to engage early education programs in targeted areas served by Fort Worth ISD. The CFSRP served a diverse set of 726 young children in the 2021–2022 school year (Table 4). Students are predominantly African American (53%), Caucasian (24%), or Hispanic/Latino (14%), in keeping with prior years. Ages served are well balanced from infancy (25%) to prekindergarten (14%), with toddlers representing the greatest share (33%).

Race/Ethnicity

STAFF CHARACTERISTICS

The CFSRP engages a variety of early childhood professionals—including center directors, lead, assistant, and float classroom teachers, and others—in professional development and support to enhance quality of care and education. In approximate alignment with student demographics, 55% of staff supported by the CFSRP in 2021–2022 are African American, 19% are Caucasian, and 20% are Hispanic/Latino (Table 5). The vast majority of staff (94%) are female.

In general, early childhood staff in childcare centers have less formal training and education than those affiliated with school districts. Staff supported by the CFSRP follow this trend: most have less than an associate’s degree: 59% have a high school diploma or GED as their highest educational attainment, and 17% have some college but less than an associate’s degree. Nineteen percent have some type of childcare credential, generally a CDA

Years of experience in the field of childcare varied across staff, with approximately one third of staff having three years of experience or less (38%) and one third having ten years or more (32%). This division is consistent with prior years.

Table 4: CFSRP Student Characteristics, 2020-2021 (N=772)

MENTOR CHARACTERISTICS

Camp Fire assigns experienced mentors to support teachers in the program. These mentors provide teachers with guidance on best practices in the classroom, how to handle challenging situations, and how to support child development and provide high quality care in age-appropriate ways. Camp Fire had six mentors in 2020-2021, for a 33:1 teacher to mentor ratio. All mentors had at least a bachelor’s degree, and half had a master’s degree. Four were African American, and two were Caucasian. Two had more than 10 years of childcare experience, and four had between seven and 10 years of experience (see Table 6).

Table 5: CFSRP Staff Characteristics, 2021-2022 (N=150)

Table 6: CFSRP Mentor Characteristics, 2020-2021 (N=6)

Program Implementation6

Process evaluation describes the extent to which the CFSRP program implemented program activities as intended and how those activities resulted in the expected program outcomes. Findings from a process evaluation enable Camp Fire to determine which aspects of the program are working as expected, and which processes require additional support and clarification in order to be most effective.

TEACHER RETENTION BEGINNING OF YEAR (BOY) TO END OF YEAR (EOY)

Low teacher retention is a persistent challenge to providing high-quality early childhood care and education. Turnover adds program expenses, increases staff burdens, and can harm child development.7 Children benefit from having a secure attachment to caregivers;8 thus, teacher turnover within a school year has been negatively associated with children’s language, literacy, and socioemotional development.9,10,11 Further, teacher turnover is highest among the types of centers CFSRP serves: centers that are not school district-sponsored or otherwise publicly funded.12

Within U.S. early childhood programs, annual teacher turnover has historically been around 25-30%.13 Recently the strain on the early childhood education workforce has increased due to pandemic-related employment declines and wage competition from other industries. In Texas, 86% of childcare providers reported center staffing shortages in summer 2021.14

In the context of state and national teacher retention trends, CFSRP teachers showed above-average employment stability. Of the 150 CFSRP staff employed during the 2021–2022 school year, 118 remained at their childcare center from beginning to end of year, for a 79% retention rate (see Figure 2).15 Teacher retention rates for the program are typical for the CFSRP in recent years and exceed those expected industry-wide in regular years as well as during the pandemic.

6 A description of the methods used for this report are provided in Appendix B.

7 Buffett Early Childhood Institute at the University of Nebraska. (2018). Early childhood teacher turnover in Nebraska. https://buffettinstitute.nebraska.edu/-/ media/beci/docs/early-childhood-teacher-turnover-in-nebraska-new.pdf

8 Folbre, N. (2012). For Love or Money New York: Russell Sage Foundation.

9 Markowitz, A. J. (2019). Within-year teacher turnover in Head Start and children’s school readiness. EdPolicyWorks, University of Virginia. https://curry.virginia. edu/sites/default/files/uploads/epw/70_Teacher_Turnover_in_Head_Start.pdf

10 Hamre, B., Hatfield, B., Pianta, R., & Jamil, F. (2014). Evidence for general and domain-specific elements of teacher-child interactions: Associations with preschool children’s development. Child Development, 85(3), 1257-1274.

11 Hale-Jinks, C., Knopf, H., & Kemple, K. (2006). Tackling teacher turnover in childcare: Understanding causes and consequences, identifying solutions. Childhood Education, 82(4), 219–226.

12 Grunewald, R., Nunn, R., Palmer, V. (2022). Examining teacher turnover in early care and education. Federal Reserve Bank of Minneapolis. https://www. minneapolisfed.org/article/2022/examining-teacher-turnover-in-early-care-and-education

13 Institute of Medicine and National Research Council. (2012). The Early Childhood Care and Education Workforce: Challenges and Opportunities: A Workshop Report The National Academies Press.

14 National Association for the Education of Young Children. (2021). State Survey Data: Child Care at a Time of Progress and Peril https://www.naeyc.org/sites/ default/files/wysiwyg/user-74/naeyc_survey_statedatawithquotes_sep2021.pdf

15 Retention rates during the 2019-2020 school year were artificially high due to centers keeping teachers on staff, in hiatus, during the coronavirus pandemic. Not all teachers on the staff lists were not actually present in childcare centers. This unique situation led to an artificially high retention rate on paper, in spite of having fewer children to serve in spring 2020.

STUDENT RETENTION (BOY TO EOY)

Continuity—in terms of having a consistent learning environment as well as maintaining healthy relationships with a consistent adult—is a critical element of early childhood learning and development.16 17 Additionally, because Camp Fire measures developmental change from beginning of year to end of year, stable class enrollment is imperative to understanding student and program outcomes. During the 2019-2020 school year—i.e., the beginning of the COVID-19 pandemic—centers participating in the CFSRP saw particularly high levels of student retention (see Figure 3). From fall 2020 to spring 2021, Camp Fire centers retained 555 of 772 students for 72% retention, a typical rate for recent years.

16 Sabol, T.J. & Pianta, R.C. (2012). Recent trends in research on teacher-child relationships. Attachment & Human Development 14(3). doi: 10.1080/14616734.2012.672262

17 Curby, T.W., Grimm, K.J., & Pianta, R.C. (2010). Stability and change in early childhood classroom interactions during the first two hours of a day. Early Childhood Research Quarterly 25(3): 373-384. doi: 10.1016/j.ecresq.2010.02.004

Figure 2: Teacher Retention Rates from BOY to EOY, 2013-2022

Figure 3: Student Retention Rates from BOY to EOY, 2013-2022

TEACHER AND DIRECTOR PROFESSIONAL DEVELOPMENT PARTICIPATION

Teachers have the opportunity to increase their knowledge and skills related to classroom management and child development through the Foundational Professional Development and the Early Education Institute (EEI) courses the CFSRP provides. Directors have the opportunity to increase their knowledge of leadership practices, business management, and child development through the CFSRP Director’s Institute (DI).

After making programmatic adjustments in response to the COVID-19 pandemic in spring 2020, Camp Fire shifted from in-person professional development to virtual and hybrid modalities during the 2020-2021 school year. All 100 full-time teachers at Level 3 (“Intense”) centers were expected to participate in EEI and foundational professional development, per the terms of Camp Fire’s agreements with each center. Of those 29 full-time teachers at Level 3 centers participated in EEI in 2020-2021, for a 29% participation rate. There were 30 Directors and Assistant Directors eligible to participate in the Directors Institute (DI); of those 18 attended, for a 60% participation rate.

TEACHER AND DIRECTOR STIPENDS

Historically, Camp Fire has awarded stipends to eligible teachers and directors based on attendance in professional development and demonstrated competency on assessments. In the 2020-2021 school year, Camp Fire shifted most stipends to participants in Camp Fire’s new Early Education Apprenticeship Program. Within CFSRP, directors of Level 4 centers and mentor teachers were eligible to receive stipends at the end of the school year, based on completion of agreed upon leadership and mentoring duties. A total of 13 CFSRP teachers and directors received stipends in 2020-2021.

ONE-ON-ONE MENTOR ACTIVITY

CFSRP mentors typically provide on-site, individualized coaching to teachers and directors. Mentors help integrate assessment results into teacher action plans, improve the use of best practices in teaching and classroom management, and coach teachers and directors on how to provide quality education.

One-on-one mentoring activities in the 2020-2021 school year were adjusted to accommodate safety protocols limiting visitors to center facilities. As a result, Camp Fire mentoring shifted from side-by-side work in classrooms to use of Bluetooth and tablets, enabling mentors to see the classroom and provide guidance to teachers without physically entering the space. With the use of these tools Camp Fire mentors were able to mimic side-by-side work and provide real-time coaching in classrooms while maintaining COVID-19 safety precautions at each center. Table 7 summarizes the CFSRP’s key mentoring activities and the number of visits in which mentors focused on each activity. During the pandemic, family engagements and support for the physical classroom environment were both necessarily limited. Mentors were especially supportive of teacher reflective follow-up on situations encountered in the classroom and classroom observation in the 2020-2021 school year.

*Activity only relevant to director mentoring visits.

Child Outcomes18

The CFSRP measures its success in helping children attain ageappropriate developmental, early literacy, and social-emotional skills through consistent use of validated assessments. For younger children, key outcomes are developmental: gross and fine motor skills, as well as communication, problem solving, and personal-social skills. For prekindergarten students, key outcomes are academic and developmental: literacy, numeracy, and social-emotional skills. For all students, social-emotional outcomes are an additional outcome and dimension of childhood development that supports school readiness. Descriptions of the assessments used to measure child outcomes are available in Appendix C.

Each year, to determine the program’s level of success in each of these domains, CFSRP staff set targets for the percent of children developmentally on-target at the end of the year. This evaluation assesses the extent to which these targets are met, as well as the extent to which individual children maintain and/or improve their developmental skills.

INFANTS AND TODDLERS: CHILD DEVELOPMENT

Center teachers administered the Ages and Stages Questionnaire (ASQ®-3) to infants and toddlers ages 0-3 to screen for potential developmental concerns. Developmental concerns are identified when a child scores below the cut-off in at least one of the ASQ®-3’s five developmental domains (i.e., problem solving, communication, fine motor, gross motor, and personal-social skills). When developmental concerns are identified, center staff use the screening results to implement individualized instruction in their classrooms and refer families to external support services if the concerns are more severe.

Infants and toddlers were developmentally on target across every developmental outcome domain except for personal-social skills (Table 8). In the aggregate, infants in CFSRP classrooms demonstrated developmental progress in all domains from beginning to end of year (Figure 4); toddlers demonstrated developmental progress in communication, fine motor, and personal-social skills (Figure 5) When considering individual change from beginning (BOY) to end of year (EOY), the majority of infants and toddlers either improved or continued to perform at or above their “real” age in each domain.

18 A description of CFSRP assessments is provided in Appendix C.

*Shaded end of year figures met or exceeded the year’s target.

Table 8: CFSRP Infant and Toddler Development Outcome Goals Versus Actual (ASQ®-3), 2021-2022

Figure 4: Percentage of Infants Demonstrating Age-Appropriate Skills by Assessment Area (ASQ®-3), 2021-2022 (N=51)

Figure 5: Percentage of Toddlers Demonstrating Age-Appropriate Skills by Assessment Area (ASQ®-3), 2021-2022 (N=104)

PREKINDERGARTEN CHILDREN: MATH AND LITERACY

Center teachers administered the CPALLS+ assessment to children ages three-and-a-half through five to assess their listening skills, ability to determine if two words rhyme (rhyming 1), ability to independently present a word that rhymes with a given word (rhyming 2), and early math skills. Rhyming 2 is one of the most challenging skills to master, and many children may not master rhyming 2 skills until they turn five or six years old. For the purposes of the evaluation, the results for four- and five-year-old children (i.e., prekindergarten) are considered.

Prekindergarten children collectively met targets for listening and math, but not for rhyming 1 or 2 (Table 9). However, the percentage of 4- and 5-year-old prekindergarten children meeting the target increased substantially from beginning to end of year across all domains, at least 25% in all listening, rhyming 1, and rhyming 2 (Figure 6).

Prekindergarten students generally fared well on academic outcomes when considering individual change from beginning to end of year, as well. Students showed particularly strong progress in math and listening, where 100% and 92% of students, respectively, improved or maintained acceptable progress by the end of the year (Figure 7).

NO CHANGE: MAKING ACCEPTABLE PROGRESS

NO CHANGE: NEEDS MORE ASSISTANCE

NO CHANGE: MAKING ACCEPTABLE PROGRESS IMPROVED

NO CHANGE: NEEDS MORE ASSISTANCE

DECLINED

Table 9: CFSRP Prekindergarten Development Outcome Goals versus Actual (CPALLS+), 2021-2022

Figure 6: Percentage of 4- and 5-Year-Old Children Improving or Making Acceptable Progress from BOY to EOY (CPALLS+), 2021-2022 (N=12)

Figure 7: Percentage of 4- and 5-YearOld Children Improving or Making Acceptable

from BOY to EOY (CPALLS+),

ALL AGES: SOCIAL-EMOTIONAL DEVELOPMENT

Center teachers administered the Devereux Early Childhood Assessment (DECA) to infant, toddler, and prekindergarten students at CFSRP-supported centers participating at Level 3 and 4.19 The DECA identifies whether children’s social-emotional skills need intervention (need), are within typical range for their age (typical), or exceed the typical range (strength).

The percentage of infants and toddlers in the “strengths” category increased from beginning to end of year, while the percentage of preschool children in the strengths category held steady. The number of infants showing need slightly increased, in contrast to declining need among toddlers and preschool children (Figure 7). From beginning to end of year, 52% of infants, 57% of toddlers, and 55% of prekindergarten students showed improvement in their DECA score.

19 Only 10 of 13 CFSRP-supported Level 3-4 centers were able to successfully administer the DECA in 2020-2021, due to staffing issues and pandemic strains that limited the extent and quality of child assessments. 2020-2021 is the first year in which not all CFSRP centers participated in the DECA assessment.

Figure 8: Social Emotional Development at BOY and EOY (DECA), 2021-2022

Classroom and Center Outcomes

CLASSROOM ENVIRONMENT QUALITY

The Classroom Assessment Scoring System (CLASS™) is a widelyused measure of the quality of teacher-child interactions in centerbased early childhood programs. Certified CLASS™ observers conducted the assessment at the beginning and end of the school year.20 Different domains are captured based on the developmental stage of the children in the classroom. In infant classrooms, CLASS™ measures teacher ability to respond to and interact with infants during play, routine care and other activities (responsive caregiving). In toddler classrooms, it measures teachers’ ability to promote intentional, prosocial interactions that encourage children’s capacity for self-regulation and social skills (emotional-behavioral support) and teachers’ ability to promote emerging, expressive language skills in children (engaged support for learning). In preschool classrooms, it measures teacher ability to foster positive relationships and respond to children’s emotions or interests (emotional support); set clear behavioral guidelines and maintain a classroom that supports children’s interactions with teachers and peers through the effective management of children’s time, behavior, and attention (classroom organization); and help children learn to solve problems, develop more complex language skills, and use feedback to deepen children’s skills and knowledge (instructional support). Empirical studies have shown positive outcomes for children in classrooms with high CLASS™ assessment ratings.21,22 Studies have also provided evidence of a threshold effect indicating a minimal level at which classroom quality in preschool classrooms is met to achieve positive student outcomes.23,24 A description of the teacher-student interaction domains CLASS™ captures and associated quality thresholds is available in Table 10.

In 2020–2021, the CFSRP CLASS™ assessment changed from using trained observers from Southern Methodist University (SMU) to trained observers at Camp Fire. Staff and mentors noted that Camp Fire staff tend to score their centers slightly lower than SMU observers, but that the new approach resulted in stronger director satisfaction with the process and improvements to the flow of implementation for all involved. A total of 9 of the 12 centers participating in the CFSRP were able to complete CLASS™ assessments at both beginning and end of year. Barriers to completion on schedule included teacher turnover, which delayed training and administration timelines.25

CLASS™ provides ratings that place the quality of teacher-child interactions in a high (5.6 or above), mid (3.0 to 5.5) or low range. Camp Fire targets at scores in the mid-quality range or higher in all domains. It expects an overall increase from beginning to end of year in each CLASS™ domain for infant, toddler, and preschool classrooms. From beginning to end of year 2021–2022, CLASS™ scores increased slightly for infant and toddler classrooms in the mid (infant – responsive caregiving and toddler – emotional and behavioral support) and low (toddler – engaged support for learning) range. CLASS™ scores decreased somewhat in pre-K classrooms across all domains. Quality thresholds are defined for preschool classrooms, and Camp Fire aims to see classrooms meet quality thresholds, as well. At the end of the year, average preschool CLASS™ scores met the quality threshold for classroom organization but fell short in emotional support and instructional support (Figure 9).

20 Bringing assessment in house was a new development in 2020-2021. In previous years, Camp Fire contracted with the Center on Research and Evaluation at Southern Methodist University to conduct all classroom assessments. Prior to the most recent year, Camp Fire staff were certified as CLASS™ observers.

21 Carr, R., Mokrova, I., Vernon-Feagans, Burchinal, M. (2019). Cumulative classroom quality during prekindergarten and kindergarten and children’s language, literacy, and mathematics skills. Early Childhood Research Quarterly, 47, 218-228.

22 Vitiello, V. E., Bassok, D., Hamre, B. K., Player, D., & Williford, A. P. (2018). Measuring the quality of teacher–child interactions at scale: Comparing researchbased and state observation approaches. Early Childhood Research Quarterly, 44(3), 161-169.

23 Burchinal, M., Vandergrift, N., Pianta, R., & Mashburn, A. (2010). Threshold analysis of association between childcare quality and child outcomes for low-income children in prekindergarten programs. Early Childhood Research Quarterly, 25, 166-176.

24 For Pre-K CLASS™, the quality threshold is set at 5 for the Emotional Support and Classroom Organization domains, and at 3.25 for the Instructional Support domain. For infants and toddlers, scores of 3 to 5 indicate a mix of effective teacher-child interactions, while scores of 6 to 7 indicate consistently effective teacher-child interactions.

25 The centers that were not able to complete CLASS™ assessments in 2021-2022 include Amazing Creations, Kiddyland Childcare, and Mother Goose.

Table 10: CLASS™ Domain Descriptions

Figure 9: CFSRP Classroom Environment and Management at BOY and EOY (CLASS™), 2021-2022

The percentage of preschool classrooms at or above the quality threshold for each domain gives an idea of classroom strengths and areas for improvement. The percentage of classrooms meeting the quality threshold decreased from beginning to end of year for classroom organization and instructional support and held steady for emotional support. However, the proportion of classrooms meeting the threshold for emotional support remained consistently low throughout the year (Figure 10). There is room for improvement in each domain, particularly emotional support and instructional support. A potential explanation for lower-than-expected outcomes among preschool classrooms is elevated teacher and class changes among preschool rooms due to staffing and enrollment demands throughout the school year, and associated educator stressors. Camp Fire will watch for trends in preschool classroom stability in 2022–2023.

CHILDCARE CENTER QUALITY

The Program Administration Scale (PAS) is a research-based instrument that captures childcare center leadership and management quality. Each CFSRP-supported center was assessed by the director and director’s mentor to identify areas of strength and improvement. CFSRP staff identified four focal areas: staff orientation, staff development, program evaluation, and family support and involvement. Collectively, CFSRP-supported centers exhibited notable growth in all four areas, with the greatest improvements in staff development (Figure 11). This trend matches results for center quality improvements in 2020–2021.

Figure 10: Percentage of CFSRP Preschool Classrooms at or Above the Preschool Quality Threshold (CLASS™), 2021-2022

Figure 11: Center Management Scores at BOY and EOY (PAS), 2021-2022

Classroom Organization Emotional Support Instructional Support Staff Development Staff Orientation

Program Evaluation Family Support and Involvement

CFSRP Contributions to School Readiness

The CFSRP also measures its success in terms of children’s school readiness upon entering FWISD prekindergarten or kindergarten and continued academic progress through third grade. Using results from FWISD assessments already in place, the evaluation examines student performance comparing cohorts of CFSRP children currently in grades Pre-K through three to demographically similar groups of children who did not attend a CFSRP center. The comparison groups were randomly selected and matched for school, grade level, socioeconomic status and gender. For prekindergarten and kindergarten students, the evaluation compares results of assessments administered at the beginning of the year. For each subsequent grade level, the results are from assessments administered at the end of each year. Where a sufficient number of students in the Camp Fire group is available (≥40), statistical analyses were used to determine if any observed differences are statistically significant. Additional information about each assessment is included in the sections below and in Appendix D. Appendix D also describes the matching process. Appendix E shows the demographic characteristics of the CFSRP students and their comparison groups.

PREKINDERGARTEN LITERACY SKILLS (FALL 2022)

FWISD administers the Circle Progress Monitoring Tool (CIRCLE)26 to prekindergarten students at the beginning of each year to identify specific early literacy skills that the children have (or have not) developed. Teachers use the CIRCLE results to plan instruction targeting individual student needs.

Figure 12 shows comparisons of the CIRCLE results for the children who attended a CFSRP center in the year prior to entering prekindergarten and their demographically similar comparison group. When interpreting these results, it is important to consider the small number of children in the CFSRP group. Discussion with FWISD staff highlighted a lingering COVID-19 impact as a potential reason for the small group number. COVID-19 and the RSV virus were both prevalent in the community throughout fall of 2022, and parents may have simply been reluctant to send their children to school.

Descriptive analyses revealed higher percentages of CFSRP students on target for Letter Naming and Rhyming skills and similar ratings for Vocabulary, Phonological Awareness, and Syllabication. These findings are similar to prior years (see Appendix F), also showing higher ratings for the CFSRP group in the areas of Letter Naming and Rhyming.

CIRCLE is a criterion-referenced assessment that identifies students who are on-track, need monitoring, or need support for their early literacy and social emotional skills. CLI Engage (2017). CIRCLE Progress Monitoring System. https://cliengage.org/public/tools/assessment/circle-progress-monitoring/

Figure 12: Comparisons of Prekindergarten Early Literacy Skills (Fall 2022)

Letter Naming Vocabulary Phon. Awareness Syllabication Rhyming I Listening*

*Smaller group sizes for this item (CFSRP n=20; Comp. n=64)

KINDERGARTEN READINESS (FALL 2022)

Kindergarten Assessments

FWISD administers the Texas Kindergarten Entry Assessment (TX-KEA)27 and the MAP® Reading Fluency™28 and Growth™29 assessments to kindergarten students at the beginning of each year. As with the prekindergarten assessments, these assessments are screening tools that help teachers plan instruction that targets individual student needs.

The MAP® Reading Fluency™ and Growth™ assessments each measure slightly different aspects of children’s reading ability. MAP® Reading Fluency™ assesses the extent to which students have mastered specific grade level reading skills. MAP® Reading Growth™ assesses children’s ability to use their reading skills for learning.

In prior years, the kindergarten comparison group included students who may have attended FWISD prekindergarten. For the current evaluation, FWISD staff identified the students who attended FWISD prekindergarten in the prior year, and these students were excluded from the comparison group. With this new process:

• the CFSRP group included kindergarten students who attended a CFSRP center as prekindergarteners in 2021-2022 or as 3-year-olds30, in 2020-2021, and

• the comparison group included students who did not attend FWISD prekindergarten in the prior year or a CFSRP center at any time.

Descriptive analyses revealed that CFSRP children had higher ratings on all literacy measures relative to students who had never attended a CFSRP center or FWISD prekindergarten (see Figure 13). These findings are similar to those of prior years, including lower percentages of CFSRP students on track with blending skills (see Appendix G).

Descriptive analyses revealed no differences between the groups for executive function. However, the comparison group received higher ratings for social emotional skills (see Figure 14). Appendix G shows group comparisons from prior years.

27 TX-KEA is a screening tool designed to assess whether kindergarten students are on-track with their literacy and social emotional skills. TX-KEA meets the Texas Educations Agency requirement for school districts to administer a Kindergarten assessment for all Kindergarten students. https://www.texaskea.org/.

28 MAP® Reading Fluency™ is an online screening and progress monitoring tool that assesses basic reading skills with an emphasis on oral fluency (e.g., listening comprehension, words per minute, accuracy, decoding). Students receive a rating of below, approaching, meeting, or exceeding grade level expectations. https://www.nwea.org/map-reading-fluency/

29 The Kindergarten MAP® Reading Growth™ is a standardized, norm referenced assessment that also focuses on early literacy including reading comprehension and use of vocabulary. Students are rated as low, low average, average, high average, or high, based on the performance of a national sample. https://www. nwea.org/map-growth/

30 Some CFSRP students entered prekindergarten in FWISD once they turned 4 years old while others remained at a CFSRP center for their prekindergarten year.

OVERALL Vocabulary Listen. Comp. OVERALL Letter Names Letter Sounds Blending

EXECUTIVE FUNCTION SOCIAL EMOTIONAL

31 Although TX-KEA includes additional literacy measures as well as math measures, the evaluation focuses on overall language and literacy scores and on 5 specific skills identified by CFSRP staff as most important for kindergarten readiness.

Figure 13: Comparison of Kindergarten Early Literacy31 Skills (TX-KEA, Fall 2022)

Figure 14: Comparison of Kindergarten Social Competence Skills (TX-KEA, Fall 2022)

MAP® Reading Fluency™ and Growth™ Results (Fall 2022)

Figures 15 and 16 show the percentages of children in each group rated below, approaching, meets, or exceeds grade level expectations for MAP® Fluency™ Language and Decoding measures. CFSRP students had higher percentages of students that met or exceeded grade level expectations than the comparison group for three of the four measures (Listening Comprehension, Picture Vocabulary and Phonics - highlighted in green). The ratings were similar for both groups on the phonological awareness measures. These results are similar to those from the Fall 2020 and 2021 analyses, when CFSRP students had higher ratings on all measures (see Appendix H).

MAP® Reading Growth™Results

MAP® Reading Growth™ rates students as low, low-average, average, highaverage or high compared to a national sample. Figures 17, 18 and 19 show the results for Overall Reading Achievement and for the measures of Foundational Skills and Comprehension Skills. Appendix I shows the results for the Fall 2021 and Fall 2020 kindergarten MAP Reading Growth™ assessments.

Overall, the CFSRP group had higher percentages of students rated at or above average than the comparison group of students who never attended a CFSRP program or FWISD prekindergarten. The CFSRP group also had higher percentages of children at the highest level (20% vs. 6%, Figure 17).

On both MAP Reading Growth measures of Foundational Skills and on both measures of Comprehension, the CFSRP group had higher percentages of students at or above average than the comparison group of students who never attended a CFSRP program or FWISD prekindergarten (Figures 18 and 19).

Figure 15: Comparison of Kindergarten Language Skills (MAP® Fluency™, Fall 2022)

Figure 16: Comparison of Kindergarten Decoding Skills (MAP® Fluency™, Fall 2022)

Figure 19: Comparison of Kindergarten Comprehension Skills (MAP® Reading Growth™, Fall 2022)

Figure 18: Comparison of Kindergarten Foundational Skills (MAP® Reading Growth™, Fall 2022)

Figure 17: Comparison of Kindergarten Overall Achievement (MAP® Reading Growth™, Fall 2022)

CONTINUED ACADEMIC SUCCESS

Using a cross-sectional design, the evaluation has followed earlier cohorts of CFSRP children at the end of each academic year (kindergarten through 3rd grade) to assess the extent to which they experience continued academic success. Due to COVID-19 interruptions, the evaluations in the past two years did not include these follow up analyses. The current evaluation resumes this process.

In prior years, the assessments included the Iowa Test of Basic Skills (ITBS) for students at the end of their kindergarten, first and second grade years and the State of Texas Assessment of Academic Readiness (STAAR) at the end of their third grade year. As an alternative, the current evaluation uses the MAP® Reading Growth™ Assessment to measure student achievement at the end of each year and to measure student growth over time32. For kindergarten, first and second grade students, the MAP® assessments have replaced the ITBS. For third grade students, the MAP® Reading Growth™ assessment presents an alternative to STAAR for several reasons, primarily because it provides consistent measures from one grade level to the next as well as standardized measures and measures of academic growth.33

To be consistent with earlier evaluations, and because of the highstakes nature of STAAR, the evaluation also included analyses of the Spring 2022 STAAR data for third grade students. However, there is

limited evidence that suggests that the quality of prekindergarten programs can be assessed through third grade criterion referenced assessments;34 therefore, these results are provided in Appendix J rather than discussed in the report.

As with the analyses of the prekindergarten and kindergarten readiness assessments, analyses of the MAP® Reading Growth™ assessments compare results of the CFSRP students at each grade level with a demographically similar group of students who did not attend one of the CFSRP centers. The kindergarten, first, second, and third grade groups are different cross-sectional samples. They are not longitudinal samples that follow the same students from year to year. For each group, the results identify:

• the percentage of students at or above average grade level achievement at the end of the 2021-2022 school year, based on the national normed sample (see Figure 20),

• the percentage of students who met their growth targets at the end of the 2021-2022 school year (see Figure 21), and

• the percentage of students with average or above average growth by the end of the 2021-2022 school year, compared to the national sample (see Figure 22).

32 CFSRP also reviewed results of the MAP® Fluency™ and MAP Math Growth™ Assessments for internal use.

33 Differences include the following: (1) STAAR is a criterion referenced assessment measuring whether a student meets a designated grade level standard at a given time. (2) As a standardized assessment, MAP® Reading Growth™ provides measures of end-of-year student achievement relative to other similar students from a norm-referenced nationwide sample. (3) MAP® Reading Growth also includes measures at the end of the year that indicate the extent to which students meet projected growth targets. The targets set for each student are based on their baseline beginning of year scores and the observed growth for similar students in the norm-referenced sample. (4) MAP® Reading Growth™ assesses the amount of students’ growth from beginning to end of year, relative to the amount of growth observed in the norm-referenced sample.

34 Heckman, J. (2015). “Pre-K researchers can’t get past the third grade.” The Hechinger Report. October 15, 2015. https://hechingerreport.org/pre-k-researcherscant-get-past-the-third-grade/

End of Year Reading Achievement

As shown in Figure 20, results of the end-of-year achievement level analyses for kindergarten, first, and second grade are mixed, with higher ratings for the CFSRP students in some areas and higher ratings for the comparison group students in other areas (highlighted in yellow).

However, at third grade, CFSRP students outperform comparison group students on all measures. The differences are statistically significant for measures of Vocabulary and Analyzing Text, as well as Overall Reading Achievement (highlighted in green). The evaluation will continue to examine the MAP® Reading Growth™ results to identify potential trends in these group differences over time.

For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

Figure 20: Percent of Students At or Above Average Reading Achievement Levels MAP® Reading Growth™ (Spring 2022)

Reading Growth

When considering the extent to which students met their projected growth targets at the end of the 2021–2022 school year, and the amount of growth from the beginning to the end of the year, the results are also mixed. In kindergarten and first grade, higher percentages of students in the comparison group met their growth targets, while in second and third grade, higher percentages of CFSRP students met their growth targets (Figure 21). Similarly, a higher percentage of first grade students in the comparison group showed average or above average reading skill growth than the CFSRP group, and a higher percentage of third grade students in the CFSRP group showed average or above average reading skill growth than the comparison group (Figure 22).35

CFRSP: AT OR ABOVE AVERAGE

CFRSP: AT OR ABOVE

COMP: AT OR ABOVE AVERAGE READING GROWTH

COMP: AT OR ABOVE

35 Comparisons are relative to the MAP® Reading Growth’s norm-referenced sample.

Figure 21: Percent of Students Meeting Reading Growth™ Targets (MAP® Reading Growth™, Spring 2022)

Spr.‘22 1st Spr.‘22

Figure 22: Percent of Students At or Above Average Reading Growth™ (MAP® Reading Growth™, Spring 2022)

Kinder Spr.‘22 1st Spr.‘22 2nd Spr.‘22 3rd Spr.‘22

READING GROWTH

Conclusions and Discussion

PROCESS EVALUATION

Teacher and student retention from beginning to end of year remained high (79% and 74% respectively), and exceeded national averages. Teacher and director professional development and stipends for participating in professional development remained high as well (97% participation in professional development and 88% stipend attainment).

CHILD OUTCOMES

Overall, child outcomes were positive: children showed appropriate developmental and academic trajectories, with improvements in mathematics and literacy from beginning to end of year and steady or increasing social-emotional skills.

Infants and toddlers showed strong development and met targets in all ASQ®-3 domains except for personal-social skills. Lower performance in personal-social skills was also a struggle in the 2020-2021 school year, but infant communication development has since improved.

Among prekindergarten students, academic results, as measured by CPALLS+, were moderately positive. Targets were met for mathematics and listening, but not for rhyming. In all domains, the percentage of children improving or making acceptable progress increased over the course of the year or, in the case of mathematics, remained steady at 100% making acceptable progress.

Finally, results from the DECA assessment showed that social emotional skills increased among infants (from 33% to 62% being in a place of strength) and toddlers (from 32% to 42%) and held constant among preschoolers from BOY to EOY.

CLASSROOM AND CENTER OUTCOMES

Classroom quality was variable during the 2021-2022 school year. Overall, CLASS™ scores increased among infant and toddler classrooms but declined somewhat among preschool classrooms. Further, only one of three preschool quality thresholds was met. An overwhelming majority (80%) of preschool classrooms met the

quality threshold for classroom organization. At the same time, achieving quality in emotional support proved challenging (17%), and the percentage of classrooms achieving quality instructional support decreased by half, to 40%, from beginning to end of year.

Center quality showed consistent improvement. Collectively, CFSRPsupported centers exhibited growth in all four areas assessed— staff orientation, staff development, program evaluation, and family support and involvement. The strongest areas at the end of the year were staff development.

SCHOOL READINESS OUTCOMES

Prekindergarten and Kindergarten Readiness

Although the number of children in FWISD prekindergarten and kindergarten with prior attendance at a CFSRP center is small, Camp Fire staff can use the results to better understand how their program contributes to school readiness and to identify strategies for improving the program.

Results of the descriptive analyses suggest that CFSRP continues to contribute to prekindergarten and kindergarten early literacy skills. FWISD prekindergarten students who attended a CFSRP center had higher ratings than the comparison group in the areas of letter naming and rhyming (emerging literacy skills for this age group), suggesting that the CFSRP approach with these skills is successful.

FWISD kindergarten students with prior attendance at a CFSRP center had higher literacy ratings than the comparison group of students with no prior CFSRP or FWISD prekindergarten attendance on TX-KEA (kindergarten readiness assessment) literacy measures, as well as on MAP® Reading Fluency™ and Growth™ measures. These results highlight the importance of early education to help children be ready to learn in kindergarten.

For CFSRP program improvement efforts, it is important to note that the ratings for children’s blending skills continue to be low. Though this is an emerging skill with expected lower ratings for this age group, CFSRP can consider implementing additional developmentally appropriate teaching strategies for this literacy skill.

CONTINUED ACADEMIC SUCCESS

As noted above, the results of the Spring 2022 MAP® Reading Growth™ measures are mixed, with group differences that are difficult to interpret at this time. At the end of kindergarten, first and second grade, CFSRP children had higher ratings in some areas and the comparison group students had higher ratings in other areas. Given that this is the first year that the evaluation included MAP® Reading Growth™ measures for these grade levels, the evaluation should continue to examine the results of these assessments to identify potential trends in the group differences. Given the availability of

resources (primarily time and effort to match data), the evaluation could also consider a longitudinal approach to follow the same groups of students from kindergarten to third grade.

In consideration of all results, it is important for Camp Fire staff to continue to critically examine child outcomes both while the children are at a Camp Fire center and when they enter school. With these efforts they can continue to increase the quality of their early childhood programs and contribute to healthy development for young children.

2021-2022 CFSRP Evaluation Recommendations

The following recommendations are based on the findings of the 2021-2022 evaluation.

CHILD OUTCOMES:

• Continue targeting personal-social skill development in infant and toddler classrooms through teacher training, modeling, and classroom practices

• Consider additional developmentally appropriate classroom teaching strategies for vocabulary and phonemic awareness skills

• Camp Fire and other community organizations should continue their efforts to support early childhood programs and to encourage families to enroll their children in a quality public or private three- and/or four-year-old program

• Identify additional teaching strategies to help children control their emotions and adapt to the classroom environment

CENTER OUTCOMES:

• Strengthen classroom quality supports, particularly for preschool classrooms

• Identify opportunities to maintain or enhance classroom quality from beginning to end of year

EVALUATION PROCEDURES:

• Work with child care centers to ensure complete and accurate records for educators and students

• Explore the opportunity and potential value of longitudinal analysis of academic outcomes

Camp Fire Early ApprenticeshipEducationProgram

A key component of high-quality early childhood education that has received increasing attention in recent years is a stable, wellqualified workforce of caregivers and educators. Research has made clear that teacher-child relationships are an important element of early childhood learning and development36 and that centers with lower wages have higher teacher turnover rates.37 Further, support in early childhood has potential to reduce disparities and support the economic and academic success of children and families from lower-income and minority populations,38 making wage shortcomings a threat to educational equity and achieving universal kindergarten readiness.

The child care industry has long been challenged by wages and provider turnover; the issue has been compounded in recent years by pandemic-related employment declines and relative wage increases in non-childcare industries that compete for workers.39 Nationwide, median hourly wages for childcare workers (excluding preschool teachers) are $13.22,40 placing childcare workers in the second percentage of all occupations.41 Wage levels in early education are lower still in Texas; in 2019, the median hourly wage among Texas teachers in early learning programs was $10.15, compared to $32.41 among kindergarten teachers. Despite the importance of the early childhood workforce, structural forces limit educators’ ability to attain strong wages and supportive career pathways.

Camp Fire’s Early Education Apprenticeship Program (EEAP)— the first early childhood apprenticeship program in Texas to be registered with the U.S. Department of Labor—works to address these issues. Through EEAP, apprentices have paid positions at host site child development centers, where Camp Fire negotiates wage increases and provides a program stipend in addition to

coaching from experienced educators, professional development through Camp Fire’s Early Education Institute and Foundational Professional Development programs, and the opportunity to obtain a Child Development Associate (CDA) credential before graduation. Through the program, apprentices have the opportunity expand their education and earning potential beyond a CDA by earning up to 33 hours of college credit and being directly connected to local college and university partners for advanced education.

EEAP enrolls apprentices in cohorts each fall and is designed to support progress at an individual pace, with completion at either 12 months (for those with an existing CDA or related credential) or 24 months (for those with limited prior work experience and/or no current CDA; see Appendix K). The theory of change42 below outlines the causal model for the program (Figure 23), with the expectation that through strengthening educators within host sites, educators and centers are both strengthened, ultimately to the benefit of child development.

At the end of the 2021-2022 school year, Camp Fire had graduated its first full cohort of apprentices, and a second cohort is one year into the program. The first program year, launched mid-COVID pandemic, focused on providing access and testing the apprenticeship model within Tarrant County. The second program year expanded hybrid work to allow for participation beyond Tarrant County, to engage Spanish-speaking educators, and to be inclusive of many educator backgrounds while addressing child care deserts and workforce weaknesses. This evaluation report summarizes participant characteristics, program progress, and workforce outcomes for apprentices from both cohorts in 2021-2022.

36 Markowitz, A. J. (2019). Within-year teacher turnover in Head Start and children’s school readiness. University of Virginia EdPolicyWorks Working Paper. https:// curry.virginia.edu/sites/default/files/uploads/epw/70_Teacher_Turnover_in_Head_Start.pdf Hamre, B., Hatfield, B., Pianta, R., and Jamil, F. (2013). Evidence for general and domain-specific elements of teacher-child interactions: Associations with preschool children’s development. Society for Research in Child Development 85(3). doi: 10.1111/cdev.12184.

37 Grunewald, R., Nunn, R., and Palmer, V. (2022). Examining teacher turnover in early care and education. Federal Reserve Bank of Minneapolis. https://www. minneapolisfed.org/article/2022/examining-teacher-turnover-in-early-care-and-education

38 Iruka, I. U., Oliva-Olson, C., & Garcia, E., (2021) Research to practice brief: Delivering on the promise through equitable polices. SRI International. https:// childcareta.acf.hhs.gov/sites/default/files/public/pdgb5ta_equitablepractices_rtp_acc.pdf

39 Ibid.

40 Bureau of Labor Statistics. (2022). Table 39-9011 Childcare workers, Occupational employment and wages, May 2021. https://stats.bls.gov/oes/current/ oes399011.htm

41 Whitebook, M., Phillips, D., and Howes, C. (2014). Worthy Work, Still Unlivable Wages: The Early Childhood Workforce 25 Years after the National Child Care Staffing Study. University of California, Berkeley: Center for the Study of Child Care Employment. https://cscce.berkeley.edu/wp-content/uploads/2014/ ReportFINAL.pdf

42 A theory of change provides an illustration of a program’s impact pathway—the logical causal change that is expected to occur as a result of program activities.

Camp Fire Early Education Apprenticeship Program

Program Components

FIRST ORDER

Apprentice outcomes (teacher or director)

SECOND ORDER Center outcomes

THIRD ORDER

Child outcomes

PD Classes

Improved teaching knowledge and classroom experience

Increased staff wages and qualifications (years of experience, credentials)

Improved child-teacher interactions

On-site Individualized Coaching

Improved teaching practices

Increased teacher retention and staff stability

Handoff to Higher Education Partners for Further Career Progression

Educational and career progression, increased wage capacity

Increased classroom and center quality

Improved child learning and development Children enter pre-K and kindergarten ready

Camp

Figure 23: EEAP Theory of Change

Participant Characteristics

APPRENTICES

An initial cohort of 23 apprentices enrolled beginning in September 2020, followed by 29 apprentices in the second year of the program (Table 11). Both cohorts were overwhelmingly female and came from varied age, racial/ethnic, and educational backgrounds upon program entry. The second year cohort, entering in 2021, had higher hourly wages at entry ($11.49 compared to $10.18) and a higher proportion entered with a CDA (41% compared to 17%). All 23 of the initial cohort of apprentices were expected to complete the apprenticeship by September 2022, 24 months after beginning the program. Among the initial pilot cohort, a total of 12 apprentices graduated (52%). Four apprentices—those who entered the program with a CDA or Associate in Arts (AA)—completed their apprenticeship in 12 months, graduating in September 2021, and eight additional apprentices graduated in August 2022. Among the second cohort, 12 apprentices graduated in August 2022, with 13 more on track to complete the program by summer 2023.

*Race/ethnicity sums to more than 100% due to selection of multiple characteristics.

Table 11: EEAP New Apprentice Characteristics (N=29), 2021-2022

HOST SITES

A total of 15 early childhood centers served as host sites for apprentices during the 20212022 school year (Table 12). These host sites support apprentice professional development, credential attainment, and annual wage increases through providing time, space, and financial support for educator career opportunity. Each site had between one and seven participating apprentices in their classrooms.

Program Participation

APPRENTICE RECRUITMENT

In EEAP’s first and second years (2020-2022), Camp Fire

First Texas recruited apprentices in collaboration with partners, particularly centers participating in the School Readiness Program. EEAP introduced potential apprentices to the opportunity through virtual interest sessions and onsite recruitment with child care center partners. It prioritized admission for teachers who have been in the field at least one year and who work at centers with a Texas Rising Star rating of 3 or 4.

All apprentices agreed to full program participation, with the support of their center director. They agreed to attend program professional development courses, complete associated coursework, meet with an assigned mentor on a regular basis, and maintain employment at their current center (i.e., host site) for a minimum of twelve months after graduation from the program. For this commitment, apprentices would receive up to $2,000 in stipends, assistance with wage increase negotiations, a non-expiring CDA credential, professional development, mentoring support, and assistance with continuing higher education.

APPRENTICE PROGRESS

EEAP is a two-year program that admits new apprentices every year. At the end of the 2021-2022 school year, the pilot cohort completed its apprenticeship and a second cohort was mid-way through the process. Of the 23 apprentices enrolled in EEAP’s pilot year, a total of 12 (52%) completed all requirements and graduated on time. This completion rate is lower than Camp Fire anticipates for the program going forward. A lower initial completion rate is attributable to the program’s mid-pandemic launch and the learning and refinement that occurs during a pilot program.43 Among the 2021 cohort, 21 of the initially enrolled 29 apprentices (72%) have successfully completed the first year and are on track to graduate within the specified 24 month period, including; 12 apprentices who entered with a CDA and successfully completed the apprenticeship within one year (41%, Table 13).

Within both cohorts, a portion of apprentices left the apprenticeship before completing it and earning an associated credential and financial benefit. Within the pilot cohort, reasons for leaving the program included family or personal health concerns and employment changes that precluded continued participation. Within the 2021-2022 cohort, reasons for leaving the program were more likely to be linked to program noncompletion (Table 14).

43 EEAP’s participation and retention was a significant accomplishment in 2020-2021, given pandemic disruptions to the child care industry, workers’ lives, supply chains limiting delivery of technological tools, and limitations inherent to launching a new program under such circumstances. In order to remain in EEAP, apprentices had to be committed to both the requirements of program participation and continued employment at their host site. Given high rates (25-30% annually) of annual teacher turnover among early childhood programs, combined with elevated worker exits from the field during the pandemic, some attrition was expected. Challenges related to health (e.g., family needs or center COVID response) and employment circumstances (e.g., changing centers or leaving the childcare industry) prevented some apprentices from completing their apprenticeship.

Table 13: Apprentice Program Progression, by Cohort

*Early Educators Institute

2020-2021 Cohort

Table 14: Apprentice Reasons for Leaving EEAP before Completion, 2021-2022

Classroom Environment Quality

Trained observers assessed each apprentice in the classroom at the beginning and end of the school year. Classrooms with an EEAP apprentice generally showed strong growth in classroom practices, as measured by the BPOT and CLASS™ tools.44 Twenty-two of the 24 classrooms with apprentices showed improvement in teaching best practices over the course of the school year (Figure 24).

CLASS™ assessments involving apprentice preschool classrooms showed strong quality at both beginning and end of year in most domains. As with the Camp Fire School Readiness Program, Camp Fire expected to see increased CLASS™ scores at end of year within classrooms hosting EEAP apprentices, with scores of 3.0 or higher in toddler and preschool classrooms; this goal was met. Camp Fire also expected to see CLASS™ quality thresholds met in preschool classrooms but achieved the goal only for classroom organization (Figure 25).

Meeting quality thresholds within preschool classrooms that included an apprentice proved to be challenging in 2021-2022. Two-thirds of classrooms met the quality threshold for classroom organization, and less than half met the quality threshold for emotional support (22%) and instructional support (33%) at year’s end (Figure 26). One limitation to comparing beginning to end of year outcomes arises with ongoing staffing adjustments, as CLASS™ is administered by classroom, and non-apprentice teachers in the classroom may change over time and can impact the classroom score.

44 For more information about BPOT, see Appendix C. For more information about CLASS™, see https://teachstone.com/class/.

Figure 24: Percentage of Apprentices Showing Increased Classroom Skill (BPOT), 2021-2022

Figure 25: EEAP Classroom Environment and Management at BOY and EOY (CLASS™), 2021-2022

26: Percentage of EEAP Preschool

Apprentice Outcomes

EEAP is designed to have a lasting, positive effect on the careers of educators. In addition to near-term goals of apprentice program participation and completion and demonstration of classroom best practices, success for EEAP means continued wage growth and educational/credentialing attainment after graduation. The specifics of career growth will vary based on each apprentice’s background and personal careers goals, but in all cases, continued growth is a central program goal.

CREDENTIALING GROWTH

From September 2021 through August 2022, the initial cohort of apprentices resulted in twelve EEAP graduates. Eight of the apprentices earned a CDA through the program, in addition to four apprentices who entered already having the credential. Within the first year of the 2021 cohort, 12 apprentices graduated. An additional nine apprentices earned a CDA through the program in their first year, in addition to 12 who entered with the credential (Table 15).

WAGE GROWTH

Apprentices who completed the program by August 2022 experienced a notable wage increase, with a larger increase within the 2020 cohort ($2.90) but higher overall wages at exit within the 2021 cohort ($15.42).45

GRADUATES’ POST-PROGRAM PATHWAYS

Five of the eight apprentices from the initial cohort who graduated are in the process of pursuing additional education through enrollment at an institution of higher education or currently pursuing enrollment. Ten of the 12 apprentices who have graduated so far from the 20212022 cohort have been referred to a local educational partner (Tarrant County College or Tarleton State University).

APPRENTICE PERSPECTIVES

Apprentices who graduated in August 2022 were invited to reflect on their experience and provide feedback. All 15 graduates who participated indicated that the program had large or major impact on their lives in terms of their ability as a teacher, ability as a family member or parent, confidence in their professional skill, and confidence in their personal abilities. Eighty percent indicated they had large or major growth in their ability to seek new opportunities thanks to the program. Further, all graduates indicated that they had taken or planned steps to register for college classes, achieve career growth, and achieve increased wages.

45 To ensure a comparable basis of comparison, wage changes were considered for apprentices who listed hourly compensation and had information on wages at both program entry and exit.

Table 15: Apprentice Career Outcomes, by Cohort

Discussion

In 2021-2022, the Camp Fire Early Childhood Apprenticeship Program concluded its initial two-year pilot and guided a new cohort of apprentices on pursuing increased education and professional development. At a time when educator stability within a role has been particularly tenuous, EEAP brought stability and opportunity to apprentice and host site participants alike. Initial apprentice outcome results are promising, with graduates showing wage growth, commitment to ongoing educational pursuits, and the stability associated with earning a nonexpiring professional credential. A key area for increasing focus and improvement is building and measuring the quality of teacher-child interactions in the classroom, which showed lower performance than anticipated overall and from beginning to end of year.

MOST SIGNIFICANT CHANGE ATTAINED AS AN EEAP APPRENTICE:

“Being more interactive with family and work.”

“Everything that I have learned in this problem it has helped me along my classroom and has helped me bring different tools to families and also staff members.”

“This class was a stepping stone to furthering my education.”

“Patience, responsibility.”

“Achieving my goal of the CDA.”

“The willingness to strive for higher education”

2021-2022 EEAP Evaluation Recommendations

The following recommendations are based on the findings of the 2021-2022 evaluation.

APPRENTICE SELECTION

• Clearly communicate to potential apprentices recruited through current CFSRP partners the differences between CFSRP and EEAP

POST-PROGRAM SUPPORT

• After connecting graduates with institutes of higher education, utilize data sharing agreements to understand graduates’ postprogram academic outcomes

• Follow up with graduates to track post-program outcomes including career and educational progression, wage increases, and duration with their center and the field

EVALUATION PROCEDURES

• Consistently measure and report on all apprentices’ classroom quality at beginning and end of year, with the capacity for rapid response and improvement before end of year or end of program

• Standardize wage reporting on a consistent, hourly basis

• Collect standardized reasons for leaving the program in apprentice exit interviews

Conclusion

Camp Fire First Texas’ early childhood programs address fundamental family and community needs while pursuing high-quality educational experiences. The Camp Fire School Readiness Program (CFSRP) prepares children for school by supporting teacher and classroom quality through professional development and mentoring. The Camp Fire Early Education Apprenticeship Program (EEAP) strengthens the early education workforce in Tarrant County and surrounding areas through rigorous training and support. Each program has continued to evolve with community circumstances in order to best support the well-being of children, families, communities, and school districts. Overall, Camp Fire’s programs to support educators and students in 2021-2022 showed strong results, innovative solutions, and a commitment to supporting children and educators in North Texas.

List of Appendices

APPENDIX A

CFSRP Program Components

APPENDIX B

Evaluation Methods

APPENDIX C

Child Development and Center Quality: Assessment Tools and Data Analysis

APPENDIX D

Data Analysis and Assessment Tools for FWISD Data

APPENDIX E

CFSRP and Comparison Group Demographics (FWISD, Fall 2021)

APPENDIX F

Year-to-Year Comparisons of Prekindergarten Readiness Results

APPENDIX G

Year-to-Year Comparisons of Kindergarten Readiness Results

APPENDIX H

Year-to-Year Comparisons of Kindergarten Fall MAP® Reading Fluency™ Results (Fall 2020 - Fall 2022)

APPENDIX I

Year-to-Year Comparisons of Kindergarten Fall MAP® Reading Growth™ Results (Fall 2020 - Fall 2022)

APPENDIX J

Year-to-Year Comparisons of 3rd Grade STAAR Results (Spring 2017 - Spring 2022)

APPENDIX K

Early Education Apprenticeship Tracks

APPENDIX A: CFSRP PROGRAM COMPONENTS

LEVELS OF SUPPORT

The CFSRP supports 50 classrooms in 13 licensed, non-residential child development centers that provide care services and early education. The CFSRP engages participating child development centers at four levels of professional development intensity. A child development center’s movement from a lower intensity level (Level 1: Initial Relationship Building) to a higher intensity level (Level 4: Sustainability) is determined by factors such as length of participation in the program, class participation requirements, and center performance and capacity. The table below shows the four professional development levels, three of which include professional development support.

PROFESSIONAL DEVELOPMENT (PD) AND STIPENDS

Professional development is offered to teachers through Foundational PD and the Early Education Institute (EEI). Both are designed to increase knowledge and skills in techniques that promote child development and classroom management. In addition, the EEI specifically addresses five components of reading science cited as critical for effective early literacy instruction.46 Directors attend many teacher sessions and participate in the Director’s Institute (DI). The DI is designed to increase knowledge of child development and center business management and leadership practices.

Historically, in an effort to promote teacher retention and engagement, full-time teachers who have completed Foundational PD and directors have been eligible for a stipend based on attendance and demonstrated competency in Camp Fire PD. The incentive pay is distributed at the conclusion of the EEI at the end of the program year. Teacher and director fulfillment of the requirements is reviewed prior to payment distribution. In 2020-2021, stipends for PD participation were targeted toward EEAP apprentices, directors of CFSRP Level 4 centers, and CFSRP mentor teachers were eligible for stipends of up to $2,000.

MENTORING

On-site individualized coaching is provided by CFSRP mentors47 who hold Bachelor’s or Master’s degrees in child development or early childhood education; have three to five years of experience coaching, mentoring and/or working in early childhood settings; and hold certifications in child development assessments. CFSRP mentors provide coaching visits to Centers based on Center level, with Level 2 and 3 Centers receiving more visits than Level 4 Centers. The practice-based coaching sessions with teachers consist of creating Teacher Action Plans with SMART (Specific-Measurable-AttainableRealistic-Timely) goals based on needs identified from assessments (e.g., using teaching best practices, improving classroom management) and supporting the attainment of the identified goals. The coaching sessions with directors also consist of setting SMART goals based on needs identified from assessments related to center business management and leadership practices and supporting the attainment of the identified goals.

FAMILY ENGAGEMENT

The CFSRP family engagement component is an ongoing collaboration between directors and center staff, which consists of a focus on reciprocal communication between families and center staff, as well as family support and involvement-- a range of activities that allows a childcare center to be responsive to family needs, including Parent Cafés48 and Playgroups.

46 National Council on Teacher Quality (2020). Program Performance in Early Reading Instruction retrieved from https://www.nctq.org/publications/home

47 The CFSRP established a Mentor Professional Pathway framework which categorizes mentors into four levels based on their existing level of training and experience. For example, Level 1 (Beginning) mentors may need support with close guidance, Level 2 (Developing) mentors may need support with increasing independence, Level 3 (Proficient) mentors may need limited support and can independently enhance the knowledge and skills of others in the profession, and Level 4 (Exemplary) mentors can develop program policies and practices and enhance the knowledge and skills of others in the profession. The CFSRP Director uses the Mentor Professional Pathway framework to monitor mentor needs and promote professional development opportunities. There were 5 mentors during the 2018-2019 school year: one at Level 1, one at Level 2, and 3 at Level 3.

48 Parent Cafés are a type of family meeting/support group that CFSRP has supported partner centers in offering to their parents. Parent Cafés are carefully designed, structured discussions that use the principles of adult learning and family support to help participants explore their strengths, learn about Protective Factors, and create strategies to help strengthen their families. CFSRP has also encouraged the use of play groups as a family support. Play groups provide opportunities for parents and their children to interact together in a planned ‘play activity’ that aligns learning opportunities between school and home. The play groups promote social-emotional development, support parent/child relationships, and encourage parents to interact with other parents in the group. CFSRP also provides information and presentations about community resources to center directors who can then use this information to refer families to supportive services regarding family issues, which is another form of family engagement. The CFSRP uses the Family Engagement Measure from the Program Administration Scale (PAS) to set programmatic goals in this area.

LEVEL 1 0 (0)

APPENDIX A, CONTINUED: CFSRP PROGRAM COMPONENTS

RELATIONSHIP BUILDING BETWEEN CFSRP AND THE CENTER

Does not include professional development

BASIC

LEVEL 2 1 (4)

Center participates for one year

INTENSE

LEVEL 3 3 (17)

Center participates for three years

SUSTAINABILITY

LEVEL 4 8 (29)

APPENDIX B: EVALUATION METHODS49

Center participation begins after the third, intensity-level year and continues as long as the center remains in the program

The CFSRP evaluation consists of both a process and outcome evaluation. The process evaluation component provides a clearer picture of how the CFSRP was being implemented in practice and determines to what extent the program was operating as designed in the theory of change. Findings from process evaluation can inform program improvement and help explain why a program achieved or failed to achieve the intended outcomes. The CFSRP process evaluation included a specific focus on professional development participation, stipend allocation, and mentor activities.

The outcome evaluation included a focus on child outcomes (developmental, academic, and social-emotional) and center outcomes (classroom environment and management, Center leadership and management). The CFSRP outcome evaluation also includes a comparative analysis of outcomes for CFSRP children and demographically similar groups of non-CFSRP children enrolled in Fort Worth Independent School District (FWISD). This part of the evaluation uses assessments already in place in to compare the performance of the CFSRP children and the comparison groups when they enter prekindergarten and at the end of their kindergarten through third grade years.

The EEAP evaluation is a process evaluation—one that is preliminary and descriptive. It examines characteristics, motivations, and barriers/supports for success of apprentices in the initial cohort, as well as a summary of apprentice progress to completion. Its focus is on identifying participant characteristics and needs in order to more completely formulate and scale the program for future cohorts.

49 Assessment tools and data analysis procedures are described in Appendix C and D.

APPENDIX C: CHILD DEVELOPMENT AND CENTER QUALITY: ASSESSMENT TOOLS AND DATA ANALYSIS

The CFSRP contracts with CNM to provide program evaluation consulting services and CNMpact outcomes services. CNM created secure, webbased online data entry spreadsheets for each CFSRP child development center. Directors at CFSRP-supported child development centers entered student and teacher enrollment information, classroom information, and assessment data. The Classroom Assessment Scoring System (CLASS™) was used by trained CLASS™ observers and submitted the final data to the CNM evaluation team to analyze. Other assessment data were provided by the CFSRP Director. Prior to data analysis, CFSRP staff reviewed and cleaned final data. The table below presents each assessment and its associated assessment areas.

Child Outcomes

AGES AND STAGES QUESTIONNAIRE, VERSION 3 (ASQ®-3)

A standardized, screening tool designed to identify infants and young children who are and are not displaying typical age-appropriate development. CFSRP recommends that children ages three years and five months or younger receive the ASQ®-3 assessment.

CIRCLE PHONOLOGICAL AWARENESS, LANGUAGE AND LITERACY SCREENER PLUS MATH (CPALLS+)

A standardized, criterion-referenced assessment designed to measure children’s literacy and language skills. CPALLS+ recommends that children ages three years and six months or older receive the CPALLS+ assessment.

DEVEREUX EARLY CHILDHOOD ASSESSMENT (DECA)

A strengths-based, standardized assessment and planning system that supports educators in promoting children’s social and emotional development, thus promoting resilience.

Cognitive and Physical Development

Language and Literacy Development

The evaluation team calculated the percentage of children meeting the cut-off for developmental skills in five domains at the beginning and end of the year. The results were disaggregated by age group.

The percentage of children demonstrating improvement in developmental skills from the beginning to the end of the year was also calculated.

Language and Literacy Development

Cognitive Development (Math)

The evaluation team calculated the percentage of children meeting the cut-off for language and literacy skills at the beginning and end of the year. Separate analyses were conducted for three-year-old children (MOY-EOY comparisons) and four-and five-year-old children (BOYEOY). The results were disaggregated by age group.

The percentage of four- and five-year old children demonstrating improvement in developmental skills from the beginning to the end of the year was also calculated.

SocialEmotional Development

The evaluation team calculated the percentage of children who scored in the Typical or Strength category in social-emotional/resilience at the beginning and the end of the year. The percentage of children demonstrating improvement in their scores was also calculated.

APPENDIX C, CONTINUED: CHILD DEVELOPMENT AND CENTER QUALITY: ASSESSMENT TOOLS AND DATA

Classroom/Center Outcomes

CLASSROOM ASSESSMENT SCORING SYSTEM (CLASS™)

A standardized, observation-based assessment designed to assess classroom management and quality on a 7-point scale. The Infant CLASS™ measures the quality of responsive caregiving in infant classrooms. The Toddler CLASS™ measures the quality of emotional and behavioral support and engaged support for learning in toddler classrooms. The Pre-K CLASS™ measures the quality of emotional support, classroom organization, and instructional support. For Pre-K CLASS™, the quality threshold is set at 5 for the Emotional Support and Classroom Organization domains, and at 3.25 for the Instructional Support domain.

BEST PRACTICES OBSERVATION TOOL (BPOT)

A research-based observational checklist that measures the presence or absence of research-based teaching practices that align with CFSRP professional development curriculum. This tool is intended for professional development purposes. Teachers in infant classrooms are rated on 105 best-practice teaching strategies, and teachers in toddler classrooms are rated on 110 bestpractice teaching strategies. The BPOT for three-year-old classrooms and four-year-old classrooms includes 110 and 120 best-practice teaching strategies, respectively.

PROGRAM ADMINISTRATION SCALE (PAS)

A 25-item research-based instrument that measures the quality of leadership and management practices of early childhood programs. PAS measures quality on a 7-point scale (1 = inadequate, 3 = minimal, 5 = good, 7 = excellent).

Classroom Management and Quality

The evaluation team included only teachers with matched pre- and post-assessment scores in the analysis. The average CLASS™ pre- and post-assessment scores were compared. In addition, for Pre-K classrooms, analyses examined the percent of teachers who met a researchbased quality threshold for each domain.

Quality in Teaching Practices

Center Leadership and Management Quality

Based on the design of the BPOT assessments, the evaluation team calculated the total observations and created a weighted system that categorized scores as ‘needs support’, ‘emerging’, and ‘consistently meets’. The results were disaggregated by domain.

This assessment is used internally to assist mentors with identifying target areas for teacher development.

The evaluation team used each center’s individual score to calculate an overall average for each of the 10 domains.

APPENDIX D: DATA ANALYSIS AND ASSESSMENT TOOLS FOR FWISD DATA

Camp Fire partners with FWISD to gain access to data from student assessments currently used in the school district. Through this partnership, the CFSRP evaluation team has been able to assess the impact of the program for eight years with analyses of the children’s assessment scores not only as they enter school but also through their kindergarten, 1st, 2nd, and 3rd grade years. Camp Fire contracted with Aware Research Solutions Inc. to provide evaluation and data analysis services.

Each year, using a random selection process50, the evaluation team compares the assessment scores of children who attended a CFSRP-supported center in one of the prior six years and children in demographically similar comparison groups who did not attend one of the centers. This technique allows the evaluation team to create a random, non-biased sample of children who are similar to the sample of CFSRP children and, in turn, make valid comparisons between the two groups. Any statistically significant differences identified in the results provide evidence that the differences between CFSRP children’s scores and the comparison group’s scores can be attributed, in part, to the CFSRP program rather than to random chance.

The groups were matched on the following characteristics:

• School location

• Grade level

• Ethnicity

• Free/Reduced Lunch Status

• Gender

• Limited English Proficiency (LEP) status

• Special Education status

To get the best possible match, the comparison groups are necessarily larger than the CFSRP groups. Students with Special Education and/ or LEP designations included in the analyses only if the CFSRP group included 5 or more with the designation (see Appendix E for the demographic descriptions of the CFSRP and comparison groups). Depending on the grade level for each set of comparison groups, the analyses were conducted with the FWISD assessment data described in the table on the next page, “FWISD Assessments Used in the Evaluation.” Because very few CFSRP children were assessed in Spanish only the English versions of each assessment were included in the analyses. ASSESSMENT

Prekindergarten Readiness51

CIRCLE PROGRESS MONITORING TOOL (CIRCLE)52

CIRCLE is similar to the CPALLS+ assessment used in the CFSRP three and four-year old classrooms. It is a criterion-referenced assessment based on the Texas Education Agency (TEA) prekindergarten guidelines for literacy, math, and social skills. Teachers use CIRCLE at the beginning of the school year to help identify children who meet or do not meet developmental benchmarks so they can plan individualized instruction.

50 For the current evaluation, the random selection process used SAS Statistical Analysis Software, SAS/STAT | SAS to select the comparison group students. 51 This description is taken from the Texas Kindergarten Entry Assessment User Guide at: https://cliengage.org/user-guides/User_Guide_TX-KEA_8.13.2018.pdf 52 CLI Engage (2017). CIRCLE Progress Monitoring System. https://cliengage.org/public/tools/assessment/circle-progress-monitoring/

TEXAS KINDERGARTEN ENTRY ASSESSMENT (TX-KEA)54

TX-KEA is a screening tool designed to assess kindergarten children’s skills in seven areas of school readiness: Language, Early Literacy, Math, Science, Executive Functioning, Social Emotional Skills, and Physical Skills. For each measure, students receive a score indicating if they are on-track, need monitoring or need additional support/ intervention. Teachers use the TX-KEA is to identify children who may need additional support and to plan individualized instruction.

Texas school districts and charter schools are required to administer a Kindergarten assessment for all Kindergarten students. The TX-KEA is on the Commissioner’s approved list of assessment Instruments for meeting this requirement.

The TX-KEA includes four domains: language, literacy, executive functioning, and social emotional competence and emotion management.

APPENDIX D, CONTINUED: DATA ANALYSIS AND ASSESSMENT TOOLS FOR FWISD DATA

Kindergarten Readiness53

LANGUAGE DOMAIN

VOCABULARY is a foundational language skill that supports learning in all content domains. Knowing a student’s vocabulary abilities helps teachers adjust their own vocabulary usage during instruction to levels that are most beneficial for individual children.

LISTENING COMPREHENSION, a foundational language skill that supports learning in all areas, assesses students’ ability to understand verbal information and follow directions. Knowing students’ receptive language abilities allows teachers to adjust the complexity of their own language during instruction.

LITERACY DOMAIN

The LETTER NAMES subtest assesses children’s knowledge of the names associated with various letters of the alphabet. Letter names is one component of letter knowledge which can predict reading achievement

The LETTER SOUNDS subtest assesses children’s knowledge of letter sounds. Letter knowledge at kindergarten entry is a strong predictor of literacy achievement

The BLENDING SOUNDS subtest assesses children’ phonological awareness, or sensitivity to the sound structure of oral language. Phonological awareness is necessary for learning to read and write and is predictive of literacy achievement.

The SPELLING subtest assesses children’s early spelling abilities, which is the ability to use sound-symbol relationships to write words. TX-KEA assesses spelling because it is highly related to later literacy achievement. Attempting to spell words requires children to apply multiple literacy skills simultaneously, such as alphabet knowledge and phonological awareness.

EXECUTIVE FUNCTIONING

The Executive Functioning domain addresses the cognitive skills used by children to plan, problem solve, and follow classroom rules.

INHIBITION. Students are asked to respond accurately to a specific stimulus (e.g., butterfly), and withhold, or inhibit, a response to a different stimulus (e.g., bee). Scores reflect the student’s ability to respond accurately while inhibiting a response

WORKING MEMORY. Students are assessed on their ability to hold in memory 1 – 3 pieces of information in an increasingly complex setting. In this subtest, children recall where cars are parked in a garage. The number of cars and the number of parking spaces increases as the student progresses.

ATTENTION. Students are assessed on their ability to focus their attention, stay on task, as well as quickly and accurately focus on relevant features of the task. They are provided 2 minutes to make as many correct matches as possible between the target object, a flower, and five answer choices, other flowers.

SOCIAL EMOTIONAL COMPETENCE AND EMOTION MANAGMENT

The SOCIAL/EMOTIONAL COMPETENCE subtest focuses on children’s social and emotional skills within a classroom setting. The subtest evaluates children’s pro-social skills, approaches to learning, and emotion understanding.

The EMOTION MANAGEMENT subtest focuses on children’s ability to manage their emotions and respond appropriately to an emotional experience. They are evaluated on whether they can adapt to the demands of a classroom and school environment.

53 A variety of sources provided the information included in these descriptions. 1Brief_PhonologicalAwareness.pdf (d1yqpar94jqbqm.cloudfront.net) Phonological Awareness | LEARN - Children’s Literacy Initiative (cli.org), Kindergarten_TEKS_0820 (texas.gov), Phonics | LEARN - Children’s Literacy Initiative (cli.org), Reading Curriculum Ladders (fortheteachers.org)

54 Texas Kindergarten Entry Assessment (TX-KEA).https://www.texaskea.org/.

APPENDIX D, CONTINUED: DATA ANALYSIS AND ASSESSMENT TOOLS FOR FWISD DATA

Kindergarten-3rd Grade Reading Fluency55

LANGUAGE DOMAIN

MAP® READING FLUENCY™56 is an online screening and progress monitoring tool. The assessment establishes a benchmark oral reading fluency level for students, and depending on the level, assesses foundational or advanced reading skills. For kindergarten students, MAP® Reading Fluency™ assesses Language Skills (Listening Comprehension & Vocabulary) and Decoding Skills (Phonological Awareness & Phonics). For each skill, students are rated at, below, approaching, meets or exceeds grade level.

LISTENING COMPREHENSION includes the ability to listen to a passage that is read aloud and ask and answer questions about the passage, as well as the ability to repeat and follow sequenced, multi-step directions.

PICTURE VOCABULARY refers to the students’ ability to understand the meaning of spoken words (receptive vocabulary) by correctly identifying pictures related to the spoken words.

DECODING DOMAIN

PHONOLOGICAL AWARENESS is the ability to recognize and work with sounds in the spoken language, including skills such as: identifying rhyming words, recognizing words with same initial sound, blending syllables, and segmenting multisyllabic words.

PHONICS/WORD RECOGNITION refers to students’ ability to recognize the sounds of the spoken language and the letters and combinations of letters (words) that represent those sounds.

Kindergarten-3rd Grade Reading Growth57

MAP® READING GROWTH™58 is a standardized assessment that provides categorical (quintiles) and continuous scale (RIT) scores designed to measure achievement at a given point in time as well as growth over the school year and from one year to another. The quintile scores identify students as low, low average, average, high average, or high, relative to a national norm-referenced group, for overall reading achievement, foundational skills (Reading/Writing & Vocabulary) and Literal Comprehension skills (Analyzing Text & Composition.). MAP® Reading Growth™ also includes measures of each student’s projected growth (from fall to spring), identifies whether the student met that projection and provides a growth quintile based on their baseline beginning of year scores and the observed growth for similar students in the norm-referenced sample.59

FOUNDATIONAL SKILLS

BEGINNING READING AND WRITING skills include phonological awareness and phonics (described above) as well as correct use of oral language.

USE OF VOCABULARY refers to the students’ ability to understand how words can be used such as the ability to identify words with the same meaning or finish an incomplete sentence.

LITERAL COMPREHENSION

ANALYZING TEXT refers to students’ ability to understand key ideas, implied meaning, and details in an oral or written passage.

COMPOSITION: Inquiry and Research refers to the students’ ability to use critical thinking to analyze and summarize text.

55 A variety of sources provided the information included in these descriptions. 1Brief_PhonologicalAwareness.pdf (d1yqpar94jqbqm.cloudfront.net)

Phonological

Awareness | LEARN - Children’s Literacy Initiative (cli.org), Kindergarten_TEKS_0820 (texas.gov), Phonics | LEARN - Children’s Literacy Initiative (cli.org), Reading Curriculum Ladders (fortheteachers.org)

56 https://www.nwea.org/map-reading-fluency/

57 A variety of sources provided the information included in these descriptions. 1Brief_PhonologicalAwareness.pdf

Phonological Awareness | LEARN - Children’s Literacy Initiative (cli.org), Kindergarten_TEKS_0820 (texas.gov), Phonics | LEARN - Children’s Literacy Initiative (cli.org), Reading Curriculum Ladders (fortheteachers.org)

58 https://www.nwea.org/map-growth/

(d1yqpar94jqbqm.cloudfront.net)

59 NWEA. (2020). 2020 NWEA MAP Growth normative data overview https://teach.mapnwea.org/impl/MAPGrowthNormativeDataOverview.pdf

APPENDIX E: CFSRP AND COMPARISON GROUP DEMOGRAPHICS (FWISD, FALL 2022)

APPENDIX E, CONTINUED: CFSRP AND COMPARISON GROUP DEMOGRAPHICS (FWISD, FALL 2022)

APPENDIX F: YEAR-TO-YEAR COMPARISONS OF PREKINDERGARTEN READINESS RESULTS

(FALL 2017 – FALL 2022)

Comparison of Beginning of Year Pre-K Skills: CIRCLE Assessment Fall 2017-Fall 2022

For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

APPENDIX

G:

YEAR-TO-YEAR COMPARISONS OF KINDERGARTEN READINESS RESULTS (FALL

2018 – FALL 2022)

FWISD Kindergarten Entry Assessment (TX-KEA) Ratings: CFSRP Students and Comparison Group (Fall 2018 - Fall 2022)

For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

APPENDIX H: YEAR-TO-YEAR COMPARISONS OF KINDERGARTEN FALL MAP READING FLUENCY RESULTS (FALL 2020 – FALL 2022)

MAP Reading Fluency Scores: CFSRP Students and Comparison Group (Fall 2020 - Fall 2022)

PHONICS (DECODING)

For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

APPENDIX I: YEAR-TO-YEAR COMPARISONS OF KINDERGARTEN FALL MAP READING GROWTH RESULTS

(FALL 2020 - FALL 2022)

MAP Reading Growth Scores: CFSRP Students and Comparison Group (Fall 2020 - Fall 2022)

For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

APPENDIX J: YEAR-TO-YEAR COMPARISONS OF 3RD GRADE STAAR RESULTS (SPRING 2017 - SPRING 2022)

Note: third grade STAAR Reading and Math spring scores are not available for spring 2020 or spring 2021 due to pandemic disruptions.

For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

APPENDIX K: EARLY EDUCATION APPRENTICESHIP TRACKS

EARLY EDUCATION APPRENTICESHIP TRACKS

No prior work experience

Expired or no CDA or other qualifying related instruction

Prior related work experience

Active CDA or other related instruction

Complete EEI and concentration classes

Complete EEI and concentration classes

Request credit for prior work experience

Request credit for prior instruction

Obtain valid CDA

Obtain valid CDA

Complete EEI and concentration classes

Complete EEI and concentration classes

Complete remaining on-the-job learning (4,000 hours)

Complete remaining on-the-job learning (2,000 or 4,000 hours)

Receive credit for up to 2,000 hours of prior work experience

Receive credit for qualifying related instruction

Complete CDA, remaining education and on-the-job learning

Complete remaining education and on-the-job learning

EARLY CHILDHOOD EDUCATOR CERTIFICATE

2021-2022 DEMOGRAPHICS

Camp Fire School Readiness Program

Since 2005, the Camp Fire School Readiness Program (CFSRP) has helped children be ready to start school through improving quality of care at neighborhood childcare centers. CFSRP supports early education programs that feed into the Fort Worth Independent School District with professional development and individual coaching/mentoring focused on teaching practices that improve language and social-emotional skills among children age 0-5.

Our Students

726 students

45% 74% Student Retention

2021-2022 DEMOGRAPHICS Camp Fire School Readiness Program

Our Teachers and Directors

2021-2022 DEMOGRAPHICS

Camp Fire Early Education Apprenticeship Program

In 2020, Camp Fire began its Early Education Apprenticeship Program (EEAP), a 1-2 year program combining paid on-the-job learning, coaching, professional development, and professional certification to strengthen career and educational pathways for early childhood professionals. EEAP is an innovative approach to a critical need: stabilizing the workforce through improved wages and decreased turnover.

34 EEAP teachers and directors

16 EEAP Centers

100% Female

24% Hispanic

Ethnicity

35% African American 44% Caucasian

3% Asian

74% Teacher Retention

41%

American Indian or Alaska Native

The first U.S. Department of Labor-certified early childhood apprenticeship program in the state of Texas, EEAP is supported by:

GED High School Any college or postsec. training

Camp Fire School Readiness Program & Early Education Apprenticeship Program

Evaluation Report

2021-2022 School Year