2015-2016 Program Evaluation
Prepared by: CNM Connect AWare Research Solutions
It is with sincere appreciation that we thank the funders, community leaders and participants who have supported and advocated for this program and its foundational work over the course of the past 12 years. Without you, the seeds of Camp Fire School Readiness Program would never have been planted—and grown into the impactful community presence that it is today.
Camp Fire First Texas 817.831.2111 CampFireFW.org
Camp Fire School Readiness 2015-2016 Program Evaluation
EXECUTIVE SUMMARY BACKGROUND
Since 2005, Camp Fire First Texas has implemented a comprehensive, research-based program designed to improve the quality of early education and care facilities located in targeted Fort Worth neighborhoods within the Fort Worth Independent School District (FWISD).
The Camp Fire School Readiness Program (the Program or CFSRP) work focuses primarily on teacher professional development and mentoring related to developmentally appropriate teaching practices for young children. Each year, the Program partner facilities agree to participate in one of three intensity levels of professional development (Entry, Engaging, Excelling), ranging from 27 to 50 hours annually, in addition to on-site mentoring. For each level, teachers are expected to attend at least 80% of the sessions. Although all developmental domains are addressed in the professional development, the primary focus is on children’s social emotional and early literacy development.
OBJECTIVES
By improving the quality of education and care, the ultimate goal of the Program is for children to be prepared to enter pre-kindergarten or kindergarten and to experience continued academic success as they progress through school. With these expectations in mind, the Program evaluation addressed the following objectives: 1. To assess the changes over time in the quality of education and care in the partner facilities. 2. To assess the changes in children’s developmental skills while attending the facilities. 3. To assess the extent to which changes in quality of early learning and children’s developmental skills differ at facilities participating in professional development at the Engaging level (lower intensity) compared to the Excelling level (higher intensity). 4. To determine children’s school readiness and academic success in FWISD schools.
PARTICIPANTS •
•
During the 2015-2016 school year, a total of 302 staff and 1,808 children participated or were enrolled in a partner facility. o The majority of staff (73.5%) were employed during both semesters (an improvement from 2014-2015, when 64% of staff were retained throughout the school year). The majority of children were ethnic minorities (45% African American, 20.2% Hispanic/Latino). o Forty-one percent (N=752) of children attended partner facilities during both semesters.
4
Camp Fire School Readiness 2015-2016 Program Evaluation
KEY FINDINGS
Classroom Quality
Program classrooms showed improvements in teaching practices that support early literacy and social emotional development. •
•
Overall, teachers of infants, toddlers and three-and-four-year-old children increased their use of best practices that support early literacy and social emotional support.
Teachers of infants, toddlers and three-and-four-year-old children who completed the Excelling level (higher intensity) of professional development showed improvements in their overall use of best practices in the classroom.
Classrooms demonstrated quality teacher-child interactions. •
•
•
•
1
The classrooms of teachers of toddlers showed improvements in overall teacher-child interactions and met the quality threshold for teacher-child interactions that promote language development at the end of the year.
Classrooms of three-and-four-year-old children received high ratings in Emotional Support and Classroom Organization which persisted from preto post-test.
At the end of the year, teachers who completed the Excelling level (higher intensity) of professional development showed higher levels of classroom quality.
Compared to a state sample of pre-kindergarten classrooms 1, the classrooms of teachers who completed the Excelling level (higher intensity) of professional development showed higher scores for teacher-child interactions that “I can see growth. I can see we’re making promote Emotional Support and Instructional a difference.” – CFSRP teacher Support.
ICF International (2016)
5
Camp Fire School Readiness 2015-2016 Program Evaluation
Children’s Developmental Skills
The Program toddlers* and four-and-five-year-olds demonstrated gains in language skills. •
•
Toddlers showed gains in communication skills, problem solving and their ability to meet their own self-help needs (personal-social skills). Four-and five-year-old children showed gains in three areas of phonological awareness.
*includes 3-year-old children here and throughout
The levels of intensity of professional development influenced children’s developmental skills differently. •
•
•
Toddlers with teachers who completed the Excelling level (higher intensity) of professional development showed gains in communication skills, problem solving and skills related to meeting their own selfhelp needs (personal-social skills).
“I use a lot of calming strategies and developmental activities to better teach the need of every child.” – CFSRP teacher
The Program four-and-five-year-old children with teachers who completed the Engaging level (lower intensity) of professional development began the year exhibiting lower-levels of ageappropriate skills, and showed greater gains in phonological awareness.
However, a higher percentage of four-year-old children with teachers who completed the Excelling level (higher intensity) of professional development began the school year exhibiting age-appropriate skills in phonological awareness.
School Readiness and Continued Academic Progress
The Program evaluation looks at the impact on children’s school readiness and continued academic success using statistical comparisons of assessment scores of the children, once they entered FWISD, with the scores of a demographically similar group of children who did not attend one of the Program facilities. These scores were taken from assessments already in place at FWISD: an Early Skills Inventory for pre-kindergarten (PKESI) and kindergarten children (KESI); the CIRCLE Progress Monitoring Tool 2 for pre-kindergarten children; and norm-referenced achievement assessments (Stanford 10 3 and Iowa Test of Basic Skills 4) administered in kindergarten, first and second grade.
2 3 4
https://cliengage.org/public/tools/child-progress-monitoring-tools https://www.pearsonassessments.com/haiweb/cultures/en-us/productdetail.htm?pid=SAT10. http://www.hmhco.com/hmh-assessments/achievement/itbs
6
Camp Fire School Readiness 2015-2016 Program Evaluation
Analyses of the school readiness data revealed that the Program children entered school in 2016 better prepared in language and literacy than their demographically similar peers. • • •
Kindergarten children had significantly higher literacy skills than the comparison group.
Pre-kindergarten children had significantly higher Letter Naming, Vocabulary and Listening skills. The two groups had similar social-emotional skills on entry into pre-kindergarten and kindergarten.
Analyses of the SAT103 and ITBS4 data suggest the Program has a continued impact on children as they progress through their kindergarten and first grade years. •
In each of the past four years, the Program kindergarten children had higher Reading scores than the comparison group, and in three of those four years, the differences were statistically significant.
•
Similarly, in each of the past three years, the Program first grade students had higher Reading scores and for two of those years, the differences were statistically significant.
7
Camp Fire School Readiness 2015-2016 Program Evaluation
RECOMMENDATIONS
For continued improvement in classroom quality and child development outcomes, the evaluation team recommends the following: •
•
•
•
•
•
Continue to identify approaches that help enhance the social-emotional development focus in professional development. Continue to focus on more targeted professional development and mentoring, especially for teachers of infants. Identify funding opportunities or strategies that will help maximize professional development efforts for facility staff as well as mentors.
Convene with Program staff, mentors and evaluation team to set forth gain-focused performance targets that reflect the varied improvement capacities for children and teachers.
Continue to leverage CNM-pact to strengthen standardized data collection strategies and to engage in more frequent data monitoring to guide professional development efforts.
Consider additional analyses of the PKESI and KESI social competence measures to identify more detailed information about children’s readiness in the social development domain. The new information can then be used to target specific classroom teaching skills for future CFSRP professional development.
“A light bulb went off for her [the teacher]. She came back [from training] to the center and claimed that the child just needed more attention.” – CFSRP center director
8
Camp Fire School Readiness 2015-2016 hics
Participant Demograp
1852
hours of
1 Entry
Professional Development
Level
20.2 %
27.3 %
1808 Children 54%
Hispanic
Caucasian
Male
7.5 %
16 Engaging
Multiracial
African American
Level
2465
10 Excelling Level
302
hours of
106
&
ectors
30 %
52 %
Infants
Toddlers
PreSchool & Pre K
achers e T
66% Have only a high school diploma or GED
24%
14.9%
Have less than 1 year experience
Some College
Experience in the field
73.5%
Female
18 %
Mentoring
Classrooms
Dir
46%
45.0 %
48%
Have 4 or more years experience
10% Have an Associate’s, Bachelor’s or Master’s degree
28%
Have 1-3 years experience
The majority of staff (73.5%) were
employed during both the semesters,
an improvement from 2014-2015 when 64% were retained during the entire school year.
* 9.1% unknown