Evaluation of the Florida Child Welfare Pre-Service Training — Final Report by fsuchildwelfare

RESEARCH REPORT

Evaluation of the Florida Child Welfare Pre-Service Training Final Report SEPTEMBER 21, 2020 Principal Investigators: Amy Vargo, Ph.D. Pam Menendez

CONTENTS Executive Summary........................................................................................................................3 Knowledge Assessment.............................................................................................................3 Case Manager and CPI Focus Groups......................................................................................3

Project Team:

Self-Assessment Survey............................................................................................................3

Monica Landers

Family Functioning Assessments...............................................................................................3

Lisa Coy

Organizational Climate and Workplace Culture Assessment......................................................3

Cassandra Thomas Crystal Dharmasena

Policy Recommendations...........................................................................................................4 Introduction.....................................................................................................................................4 Study Sites Description..............................................................................................................4

Funded through a contract with the Florida Institute for Child Welfare

Evaluation Findings........................................................................................................................4 Knowledge Assessments...........................................................................................................4 Protocol Development.............................................................................................................. 5 Method...................................................................................................................................... 5 Results...................................................................................................................................... 6 Summary .................................................................................................................................. 8 Case Manager and CPI Observations and Focus Groups..........................................................8 Baseline Observations.............................................................................................................. 9 6- and 12-Month Focus Groups: Case Managers.................................................................. 10 Summary ................................................................................................................................ 14 6- and 12-Month Focus Groups: CPIs................................................................................... 15 Summary ................................................................................................................................ 22 Case Manager and CPI Self-Assessment Survey Results.......................................................22 6-Month Self-Assessment Results �� 22 12-Month Self-Assessments �� 24 Family Functioning Assessment ��25 Child Protective Investigators (FFA-I) �� 25 Case Management (FFA-O) �� 26 CPI and Case Manager Self-Assessment of FFAs ............................................................... 27 Comparison and Summary ................................................................................................... 27 Organizational Climate and Workplace Culture Three- and 12-Month Assessment Results .............................................................................28 Importance of Job Characteristics and Job Satisfaction........................................................ 28 Turnover Intentions................................................................................................................. 29 Organizational Culture and Workplace Climate Characteristics............................................ 29 Summary................................................................................................................................. 30 Training Perspectives Focus Groups.......................................................................................31 Finding 1: Training Content.................................................................................................... 31 Finding 2: Knowledge Assessment........................................................................................ 31 Finding 3: Training Structure.................................................................................................. 32

Finding 4: Family Functioning Assessments �� 32 Finding 5: Turnover �� 32 Finding 6: COVID-19 Changes �� 33 Discussion and Recommendations �� 33 Knowledge Assessment................................................................................................................................................................ 33 Case Manager Focus Groups....................................................................................................................................................... 33 CPI Focus Groups........................................................................................................................................................................ 34 Self-Assessment Survey............................................................................................................................................................... 35 Family Functioning Assessments.................................................................................................................................................. 35 Organizational Climate and Workplace Culture Assessment �� 35 Policy Recommendations.............................................................................................................................................................. 36

TABLES Table 1: Dates of Completed Evaluation Activities Across Sites: Months 1-3 �� 5 Table 2: Dates of Completed Evaluation Activities Across Sites: Months 4-12 �� 5 Table 3: Proportion of Correctly Answered Core Curriculum Knowledge Assessment Questions �� 6 Table 4: Specialty Track Modules Assessed....................................................................................................................................................... 6 Table 5: Average Module Scores for the CM Specialty Track Knowledge Assessment �� 6 Table 6: Average Module Scores for 6-Month CM Specialty Track Knowledge Assessment (n = 11) �� 7 Table 7: Average Module Scores for 12-Month CM Specialty Track Knowledge Assessment (n = 7) �� 7 Table 8: Average Module Scores for CPI Specialty Track Pre- and Post-Knowledge Assessments �� 8 Table 9: Average Module Scores for 6-Month CPI Specialty Track Knowledge Assessment (n = 9) �� 8 Table 10: Numbers of Observations by Type and Position �� 9 Table 11: Case Manager Observation Skill Counts............................................................................................................................................. 9 Table 12: CPI Observation Skill Counts............................................................................................................................................................. 10 Table 13: FFA-I Areas of Review and Percentage of Cases with Sufficient Information by Interval................................................................. 26 Table 14: FFA-Os Areas of Review and Percentage of Cases with Sufficient Information by Interval............................................................. 27 Table 15: Mean scores at each interval for Family Functioning Assessment section �� 27

FIGURES Figure 1: Importance of Job Characteristics at 3- and 12-Months Post Training �� 28 Figure 2: Satisfaction with Job Characteristics at 3 and 12 Months �� 28

APPENDIX Appendix A: Average Correctly Answered Core Curriculum Knowledge ......................................................................................................... 39 Appendix B: Observation Checklist for Case Managers �� 40 Appendix C: Field Observation Checklist for CPIs �� 42 Appendix D: Observation Checklist Objective and Corresponding Competency Key...................................................................................... 44 Appendix E: 6- and 12-Month Case Manager Focus Group Protocol �� 45 Appendix F: 6- and 12-Month CPI Focus Group Protocol �� 46 Appendix G: 6-Month Focus Group Coding Definitions – Case Managers �� 47 Appendix H: 6-Month Focus Group Coding Definitions – CPIs �� 50 Appendix I: Case Manager and CPI Self-Assessment Survey �� 52 Appendix J: Climate and Culture Survey �� 55

FLORIDA INSTITUTE FOR CHILD WELFARE

Executive Summary The child welfare pre-service training evaluation was designed to: 1) assess the readiness of case managers and CPIs to begin their job responsibilities, 2) determine whether pre-service training is at the level it should be, and 3) identify both environmental factors and workers’ individual coping strategies that facilitate and hinder knowledge acquisition and skill development while in the roles of case managers and child protective investigators. Five of the six regions in Florida were represented by nine study sites, comprised of six case management organizations, two DCF Circuits, and one sheriff’s office. This report concludes a two and a half year, indepth look at the child welfare pre-service training and the first-year trainees in the field. It is comprised of five core evaluation activities: 1) a knowledge assessment; 2) an observational checklist and focus groups; 3) a self-report survey; 4) a review of the Family Functioning Assessment (FFA) proficiency; and 5) an organizational culture and climate assessment. KNOWLEDGE ASSESSMENT The Knowledge Assessment component was developed to examine transfer of knowledge that occurred as a result of preservice curriculum training, in addition to knowledge retained over time. Findings indicate that knowledge was gained as a result of the CORE curriculum training as well as the Specialty Track training. Although some scores decreased by the 6-month followup assessment and again at 12-month assessment, overall, scores were still greater than those observed at pre-test. It is likely that some items included in the knowledge assessments are more applicable to case managers’ and CPIs’ roles than others, which may explain the fluctuation over time of correct responses. Participants also stated there may be a gap between what is learned in pre-service training and what supervisors support and prioritize once workers are in the field. CASE MANAGER AND CPI FOCUS GROUPS The purpose of the case manager and CPI 6- and 12-month focus groups was to examine how helpful the child welfare pre-service training was to case managers and CPIs in the day-to-day work they do. Overall, participants felt that their pre-service training was beneficial to developing the necessary skill set for their daily work as child welfare professionals. They indicated that information provided on identifying and understanding maltreatment was adequately covered. Areas of the pre-service training they found most helpful were role playing exercises, shadowing experienced child welfare professionals, and field days. Case managers felt that the CORE training was geared more toward CPIs and proposed several areas where additional training was necessary, such as navigating all aspects of the judicial process, case and safety plan updates, working in FSFN, aspects of adoption work, and time and stress management. Case managers also asked for more field days scheduled at times when experienced case managers were conducting home visits. Other suggestions included more role-playing exercises; jointly carrying an active case during pre-service training with an experienced child welfare worker; and providing a peer advocate for case managers to go to with questions once in the field. Some case managers expressed a reluctance to ask for help from supervisors who had indicated that pre-service training should have provided the necessary skill set. Case managers also reported that their understanding of FFA-Os and safety planning, two critical functions of their daily responsibilities, were not taught and/or practiced at a level that gave them enough knowledge or confidence once they were working directly with children and families. CPIs additionally asked for more practice with assessments and data systems that are required FLORIDA INSTITUTE FOR CHILD WELFARE

for daily use; and to make the training more comprehensive and realistic in addressing the wide variety of individuals and situations CPIs can expect to encounter in their work. CPIs reported that their understanding of FFA-Is and safety planning, two critical functions of their daily responsibilities, required more opportunities to observe and practice with “real” families in order to fully understand. In addition, first year turnover rates were high. Participants named several factors in their decision to remain employed in child welfare, such as loving the work they do, feeling that they made a difference, having a supportive supervisor and supportive co-workers, and that some case management areas are perhaps easier than others to stay in. Case managers, specifically, pointed mainly to the quality and accessibility of supervisors and the support of coworkers as major factors in their desire to remain in their positions. Alternately, reasons provided as to why peers left their case management position included emotional strain, an unsupportive supervisor, turnover as a domino effect, and low salaries/lack of recognition. SELF-ASSESSMENT SURVEY Case managers and CPIs completed an electronic, self-assessment survey at baseline (approximately 1 month), 6 months, and 12 months after their unit start dates. Overall, the responses by case managers on the 6-month self-assessment indicated ongoing challenges with key areas in transfer of knowledge that were also reported at the 1-month interval. These included: difficulty obtaining enough hands-on experience in pre-service training; a stressful work environment with unrealistic expectations for caseloads; and some discrepancies between expectations during pre-service training and actual practice, which may, in part, be due to different approaches to the work by superiors. With regard to supports in retaining knowledge and skills, case managers and CPIs consistently pointed to the benefit and importance of internal support by supervisors and other colleagues and mentors. FAMILY FUNCTIONING ASSESSMENTS A review was conducted of FFA-Is that were completed by CPIs to make initial case decisions and FFA-Os by case managers to make ongoing case decisions. Overall, reviews indicated a lack of understanding of how to accurately assess and document decisionmaking regarding child safety and risk, as well as insufficient documentation and evidence of information collection needed to make informed decisions in most areas and domains. In addition, only 35 percent of cases documented supervisory consultation before case plan approval and only 38 percent had evidence of regular consultation between supervisors and case managers; this declined further at 12 months. Although pre-service training appears to teach, within a classroom setting, the purpose of the FFA, assessment, engagement, and information collection and documentation, these skills and knowledge do not appear to translate to the field. ORGANIZATIONAL CLIMATE AND WORKPLACE CULTURE ASSESSMENT Participants completed the Organizational Climate and Workplace Culture Assessment survey at 3 and 12 months after pre-service training. It was commonplace for trainees to view a new child welfare job as only temporary until a better opportunity became available, with respondents expressing little aspiration for promotion within the field. The majority planned to leave their job to look for employment outside of child welfare. Staff remaining in their positions for 12 months or longer were more satisfied overall with their work than the 3-month respondents. Specific job areas that saw the highest gains in satisfaction as time progressed were satisfaction with 3

intellectual challenge, organizational climate, and opportunities for advancement. However, satisfaction over time regarding salary and benefits dropped, and emotional strain and burnout were common at both data points. POLICY RECOMMENDATIONS The following recommendations are made to improve the training experience and first year of employment for new case managers and CPIs: • Provide additional information to trainees regarding what types of challenges workers can expect to face during their first year (e.g., viewing a day in the life of an experienced case manager/CPI), so they may better determine whether or not the job is a good fit for them. • Consider what formal supports, such as field trainer positions, can be provided during a trainees’ first year to help staff better cope with emotional strain. • Consider additional support positions for supervisors that would assist with observations and feedback once trainees are back in units (e.g., Mentor, Field Training Coach, Advocate). • Incorporate time management or disaster planning/coping strategies into pre-service training and consider bringing in a more experienced case managers/CPI to a pre-service training day in order to talk about what strategies they developed to cope. • Continue emphasizing hands-on learning strategies in preservice training such as mock cases and trials, role plays, and FSFN practice. • Consider increasing the number of field days during preservice training, as well as a more formalized process for assuring that field days are scheduled on days and times that provide new case managers and CPIs with the ability to observe and learn from relevant activities. Participants stressed the need to learn under less scripted conditions than what the current role-playing exercises offered during preservice. • Consider scheduling field days directly after related training content, so as to better match training content with field experience. For example, if FFA-Is had just been reviewed in class, participants felt it would be helpful, then, to see a case manager complete one in the field. • Develop strategies across regions to provide realistic opportunities for interacting with families with diverse scenarios during training (e.g., “field house” model or role plays with outside personnel or volunteers). • Provide clarification during pre-service training on the function and importance of FFAs for CPIs and ensure dedicated inservice training on FFAs is available on a continuous basis. • Increase training content and duration on interviewing skills, specific to interviewing both parents and young children. • Increase training content on case management functions. Case manager trainees suggested spending more time on specific aspects of case management rather than the frontend maltreatment that caused the investigation or aspects of CPI work. Participants were interested in more detail on dayto-day activities, processes, and forms case managers need to complete. • Integrate supervisors into pre-service training processes a

to create a more fluid transition between knowledge gained during training and expectations in the field. • Consider ways to decrease the caseload size of new workers during their first year to prevent the cascading effects of turnover.

Introduction The Florida Institute for Child Welfare (FICW) at Florida State University (FSU) contracted with the Department of Child and Family Studies at the University of South Florida (USF) to conduct a study to examine the impact of child welfare pre-service training for newly hired child welfare professionals in Florida. This study was designed to: 1) assess the readiness of case managers (CMs) and child protective investigators (CPIs) to begin their job responsibilities, 2) determine whether pre-service training is at the level it should be at, and 3) identify both environmental factors and workers’ individual coping strategies that facilitate and hinder knowledge acquisition and skill development while in the roles of case managers and child protective investigators. This evaluation is comprised of five core evaluation activities: a knowledge assessment, an observational checklist and focus groups, a selfreport survey, a review of the Family Functioning Assessment (FFA) proficiency, and an organizational culture and climate assessment. Training perspectives from key training personnel were also assessed prior to and at the end of major data collection activities. This report provides a retrospective analysis of data collected during the entire study period, between February 1, 2018 and September 20, 2020. The pilot phase of the evaluation concluded on January 25, 2019 and all evaluation activities were completed by August 26, 2020 (see Tables 1 and 2). This report includes analysis of each study component, with an emphasis on patterns and changes that were observed among workers from the beginning of pre-service training through 12 months in their positions. STUDY SITES DESCRIPTION Full evaluation activities were conducted with nine study sites, including six case management organizations (CMO)s, two DCF circuits, and one sheriff’s office. Five of the six regionsa in Florida are represented. The Southern Region was excluded from the study due to changes in local leadership during the study initiation. The SE Region includes two sites. Two of the CORE trainings had a combined case management and child protective investigation model. The date range for study implementation was October 17, 2018 to February 18, 2019, meaning all sites took part in their first evaluation activity within this period. The class sizes for each site ranged from six to 14 participants, taking into account that some classes had case managers and CPIs combined for CORE. The evaluation team identified a key representative from each site for overall study communications, as well as training representatives and supervisors for communications regarding specific evaluation activities. Representatives from each site were extremely responsive and helpful in coordinating activities and data requests. As reported by the study sites, there were 14 case managers and 14 CPIs remaining from the original trainee cohorts (approximately 30 percent retention overall), while one site reported no remaining case managers.

Evaluation Findings

KNOWLEDGE ASSESSMENTS A primary goal of this evaluation was to demonstrate how much transfer of knowledge and skills occurred as a result of the pre-

Suncoast (SC), Southeast (SE), Central (C), Northwest (NW), Northeast (NE)

FLORIDA INSTITUTE FOR CHILD WELFARE

Table 1: Dates of Completed Evaluation Activities Across Sites: Months 1-3 Region

Site

Training

CORE Pre

CORE Post

Interviews

Central

Specialty

Specialty Track

1-Month

1-Month Self-

Culture/Climate

Track Pre

Post

Observation

Assessment

Baseline

Pilot

8/7/20

8/1/20

9/5/18

9/6/18

9/25/20

11/13/20

1/14/19

CBC

2/19/19

2/18/19

3/20/19

3/21/19

4/9/19

6/3/19

6/17/19

7/22/19

DCF

N/A

2/18/19

3/20/19

3/21/19

4/11/19

6/3/20

6/17/20

7/22/19

CBC

10/4/18

10/17/18

11/16/18

11/19/18

12/7/18

1/28/20

2/14/19

3/25/19

Central

CBC

1/15/19

1/3/29

2/18/19

2/11/19

3/15/19

5/1/19

5/13/19

6/17/19

CBC

2/19/19

1/29/19

2/26/19

2/27/19

3/22/19

5/1/19

5/13/19

6/24/19

CBC

1/22/19

2/20/19

2/21/19

3/11/19

5/1/19

5/13/19

6/10/19

SE 1

SE 2

DCF

N/A

1/22/19

2/20/19

2/26/19

4/1/19

6/3/19

6/10/19

7/29/19

CBC

1/22/19

1/7/19

2/4/19

2/25/19

5/6/19

5/13/19

5/26/19

Sheriff’s Office

2/19/19

1/22/19

2/28/19

3/18/19

4/4/19

6/3/19

6/10/19

7/8/19

Table 2: Dates of Completed Evaluation Activities Across Sites: Months 4-12 Region

Central NE

Site

6-Month Focus

6-Month

12-Month

Training Follow

Group

Self-

Specialty

Focus Group

Self-

Specialty

Culture/Climate

Up Focus Group

Assessment

Track Post

Assessment

Track Post

Pilot

N/A

CBC

10/21/19

11/6/19

11/11/19

4/20/20

5/6/20

8/25/20

DCF

10/21/19

11/6/19

11/11/19

4/20/20

5/6/20

8/25/20

CBC

6/3/19

6/24/19

7/1/19

2/20/20

4/1/20

5/6/20

3/26/20

8/18/20

Central

CBC

9/16/19

9/18/19

9/30/19

4/6/20

4/1/20

5/6/20

3/26/20

8/25/20

CBC

N/A

10/7/19

10/14/19

N/A

4/1/20

N/A

3/26/20

8/18/20

CBC

8/5/19

9/16/19

9/23/19

N/A

4/1/20

5/6/20

3/26/20

8/18/20

DCF

10/7/19

10/14/19

10/21/19

4/28/20

5/6/20

8/18/20

SE 1

SE 2

CBC

8/5/19

8/12/19

9/16/19

5/6/20

4/1/20

5/6/20

3/26/20

8/18/20

Sheriff’s Office

10/7/19

10/14/19

10/21/19

5/4/20

5/6/20

8/18/20

service training curriculum. To examine the transfer of knowledge, Knowledge Assessments were developed based on the content addressed in the CORE pre-service curriculum. Knowledge gained from trainees’ participation in the specialty track curriculum for case managers and child protection investigators that follows the CORE curriculum was also assessed in Knowledge Assessments tailored specifically to those curricula. Protocol Development The evaluation team partnered with a USF faculty member with test development expertise in order to produce a valid instrument that assessed knowledge and skills learned. The test development expert and evaluation team members reviewed all pre-service training materials in order to create a pool of potential assessment items. Specifically, for the CORE curriculum, 174 objectives situated within 10 modules were reviewed, resulting in a pool of 152 multiplechoice items. Team members with direct experience in child welfare then reviewed the items for accuracy and clarity to reduce the number of items to content that related more to practice knowledge (as opposed to general knowledge of the child welfare system). The resulting pool of questions was then reviewed by an external content expert who provided feedback on which items were stronger in terms of relevance and question structure (e.g., questions that were scenario-based were seen as better at assessing knowledge transfer) and suggested greatly reducing the number of items in

FLORIDA INSTITUTE FOR CHILD WELFARE

order to achieve a higher response rate. The study team made a strategic decision to ensure that the assessment represented the most important practical components of the curriculum and had a higher chance of being completed, rather than to ensure that it assessed a broad array of content. This process resulted in a 34item assessment which was then piloted in one training site before being finalized. Using the same process, training material for specialty track curricula for case managers (CM) and child protective investigators (CPIs) was reviewed and Knowledge Assessments were developed. Method Pre-service trainees completed the CORE Knowledge Assessment immediately prior to participation in the CORE training curriculum and following the commencement of the CORE curriculum. Then, just prior to the start of specialty track training, trainees completed a pre-test of the respective Knowledge Assessments for the case manager specialty track and the CPI specialty track. The Knowledge Assessment post-tests were administered immediately after training. Two follow-up assessments were administered at six months and 12 months after the specialty track training concluded. Together, these post-training assessment intervals were designed to give insight into how much the pre-service specialty track training led to knowledge gained, versus on-the-job training or experience.

Results

Table 4: Specialty Track Modules Assessed

CORE Pre-Service Training

CM Specialty Track Modules

The Knowledge Assessment developed for the CORE curriculum was informed by the following pre-service training modules: The Practice Module, Child Development, Trauma and the Child, Family Conditions, Understanding Child Maltreatment, Assessing and Analyzing Family Functioning, Safety and Risk, Safety Planning, and the CORE Readiness Assessment. Although the curriculum included two other modules on Florida’s Child Protection System and another that assessed CORE readiness, these were not included in constructing the Knowledge Assessment. Results of the Knowledge Assessment for the CORE curriculum were organized according to the modules assessed. Findings include data from 91 pre-service trainees across sites that completed the pre- and post-assessment. Paired t-test analyses for each module are shown in Appendix A. However, for easier interpretation across modules, Table 3 presents the percentage of questions trainees answered correctly at pre-test and post-test for each module. Overall, there was significant improvement in items correctly answered at the post-training assessment compared to the pretest. Trainees answered 70.5 percent of questions correctly at pre-test and 82 percent of questions correctly at post-test. Further, the percentage of questions answered correctly at post-test was greater than the pre-test for each module assessed. Significant increases in correct responses were observed in five modules: The Practice Module, Understanding Child Maltreatment, Assessing and Analyzing Family Functioning, Safety and Risk, and Safety Planning. The greatest proportion of correct responses following the CORE pre-service training was observed for Understanding Child Maltreatment (88%), Safety and Risk (87%), and Assessing and Analyzing Family Functioning (86.6%). The increase from pretest to post-test was greatest for the Safety and Risk module and the Assessing and Analyzing Family Functioning module. Findings from these Knowledge Assessments provided some evidence that knowledge was gained as a result of the CORE curriculum training. Table 3: Proportion of Correctly Answered Core Curriculum Knowledge Assessment Questions Pre-

Post-

Paired

Test

t-tests

(n = 91)

The Practice Module (4 items)

64.3%

77.8%

***

Child Development (3 items)

81.3%

84.3%

Trauma and the Child (3 items)

70.0%

74.3%

Family Conditions (4 items)

85.5%

88.0%

Introduction to Case Management Case Transfer Safety Management Out-Of-Home Care Family Engagement Standards: Preparation and Introduction Family Engagement Standards: Exploration Family Engagement Standards: Case Planning Evaluating Family Progress

CPI Specialty Track Modules

Introduction to Child Protection Investigations Assessment of Hotline to Assignments Commencement of the Investigation: Initial Contact and Present Danger Present Danger Assessment The FFA- Investigation and Safety Planning Developing In-Home or Out-Of-Home Safety Plan

Results of the Knowledge Assessments for the Specialty Track curricula are organized according to the modules assessed. Paired t-test analyses and repeated measures analysis (RMA) were used to compare module scores across data collection time points for each trainee. For easier interpretation across modules, reported findings detail the percentage of questions trainees answered correctly at pre-test and post-test for each module. Case Management Specialty Track Of the 59 trainees who completed the CM Specialty Track training, 89 percent (n = 53) completed both the initial pre- and post-Knowledge Assessments (see Table 5). Overall, there was significant improvement in items correctly answered at the posttraining CM Specialty Track assessment compared to the pre-test. Trainees answered 74.7 percent of questions correctly at pretest and 82.6 percent of questions correctly at post-test. There was an increase in correct responses for each module assessed. Significant increases, however, were observed with four of the eight modules: Case Transfer, Safety Management, Family Engagement Standards: Preparation and Introduction, and Family Engagement Standards: Case Planning. The greatest proportion of correct responses following the CM Specialty Track training was observed for Evaluating Family Progress (88.1%), Family Engagement Standards: Exploration (87.6%), and Case Transfer (84.9%). From pre-test to post-test, the greatest increase was observed for two of the Family Engagement Standards modules, specifically, Case Planning, and Preparation and Introduction. Table 5: Average Module Scores for the CM Specialty Track Knowledge Assessment Pre-

Post-

Paired

Test

t-tests

(n = 53)

67.9%

75.5%

ns *

Understanding Child Maltreatment (5 items)

58.2%

73.4%

***

Assessing and Analyzing Family Functioning (6 items)

72.0%

86.7%

***

Case Transfer (5 items)

75.9%

84.9%

Safety and Risk (5 items)

70.6%

87.0%

***

Safety Management (6 items)

77.7%

83.3%

Safety Planning (5 items)

68.2%

82.2%

***

Out-Of-Home Care (7 items)

72.0%

75.7%

Overall (35 items)

70.5%

82.0%

***

Family Engagement Standards: Preparation and Intro-duction (5 items)

69.1%

79.6%

Note: ***p < .001; ns = not significant

Family Engagement Standards: Exploration (3 items)

73.6%

75.5%

Specialty Track Pre-Service Training

Family Engagement Standards: Case Planning (7 items)

72.0%

87.6%

***

Evaluating Family Progress (7 items)

82.2%

88.1%

Overall (41 items)

74.7%

82.6%

***

In addition to the CORE training curriculum Knowledge Assessment, two additional assessments were developed specific to the CM Specialty Track and the CPI Specialty Track. Table 4 outlines modules from the specialty track curricula that were included in the Knowledge Assessments.

FLORIDA INSTITUTE FOR CHILD WELFARE

Introduction to Case Management (1 item)

Note: *p < .05; **p < .01; ***p < .001; ns = not significant

Trainees participated in follow-up assessments 6 and 12 months after completion of the specialty track curricula, responding to the same questions asked in the pre- and post- assessment. Unfortunately, the response rate for this assessment dropped drastically at the 6-month follow-up and again at 12-month followup. Due to turnover, low response rate, and missing data, findings only include data collected from 11 CM trainees across sites for the 6-month follow-up assessment and seven trainees at the 12-month follow-up assessment across sites. Data from 11 trainees who completed the CM Specialty Track training follow-up assessment at 6 months are shown in Table 6. Overall, there was significant improvement in items correctly answered at the follow-up assessment compared to the pre-test. Trainees answered 74.5 percent of questions correctly at pre-test and 83.6 percent of questions correctly at follow-up assessment. With the exception of the module on Family Engagement Standards: Preparation and Introduction, a greater proportion of items were answered correctly at follow-up compared to the pre-test for each module assessed. Modules with the greatest percentage of correct responses at follow-up assessment was observed for Case Transfer (100%), Safety Management (93.9%), and Introduction to Case Management (90.9%). According to the repeated measures ANOVA (RMA) analysis, significant increases were observed for two modules (bolded in Table 6): Introduction to Case Management and Safety Management. The percentage of questions correctly answered increased at each assessment time point. Although scores are increased for the module on Out-Of-Home Care at each time point, this increase was not significant. Table 6: Average Module Scores for 6-Month CM Specialty Track Knowledge Assessment (n = 11) Pre-

Post-

Test

Follow-

RMA

(83.7%). Scores decreased for the remaining modules with the most notable decreased observed for the Case Transfer and Family Engagement Standards: Exploration modules. However, overall, 12-month assessment scores were still greater than those observed at pre-test. According to the repeated measures ANOVA (RMA) analysis, significant change was observed for three modules (bolded): Introduction to Case Management, Case Transfer, and Family Engagement Standards: Preparation and Introduction. Table 7: Average Module Scores for 12-Month CM Specialty Track Knowledge Assessment (n = 7) Pre-

Post-

6 Month

Test

Post

Month

RMA

Introduction to Case Management (1 item)

28.6%

85.7%

100%

Case Transfer (5 items)

65.7%

85.7%

88.6%

71.4%

Safety Management (6 items)

73.8%

80.9%

88.1%

Out-Of-Home Care (7 items)

Post

75.5%

83.7%

Family Engagement Standards: Preparation and Introduction (5 items)

80%

65.7%

Family Engagement Standards: Exploration (3 items)

76.2%

71.4%

61.9%

Family Engagement Standards: Case Plan-ning (7 items)

65.3%

83.7%

79.6%

71.4%

Evaluating Family Progress (7 items)

83.7%

89.8%

75.5%

Overall (41 items)

73.2%

83.3%

82.2%

76.0%

Note: *p < .05; **p < .01; *** p < .001; ns = not significant

(6-month)

Introduction to Case Management (1 item)

36.4%

72.7%

90.9%

Case Transfer (5 items)

100%

96.4%

100%

Safety Management (6 items)

80.3%

81.8%

93.9%

Out-Of-Home Care (7 items)

71.4%

80.5%

81.8%

Family Engagement Standards: Preparation and In-troduction (5 items)

78.2%

81.8%

72.7%

Family Engagement Standards: Exploration (3 items)

75.8%

69.7%

78.8%

Family Engagement Standards: Case Planning (7-items)

71.4%

85.7%

84.4%

Evaluating Family Progress (7 items)

79.2%

89.6%

84.4%

Overall (41 items)

74.5%

83.6%

Note: *p < .05; **p < .01; ***p < .001; ns = not significant

Assessment scores continued to trend downward according to findings from the 12-month follow-up assessment. The proportion of questions answered correctly decreased from 82.2 percent at the 6-month assessment to 76 percent at the 12-month assessment. Scores remained the same or improved for Introduction to Case Management, Safety Management, Out-of-Home Care, and Family Engagement: Preparation and Introduction. Modules with the greatest percentage of correct responses at the 12-month follow-up assessment was observed for Introduction to Case Management (100%), Safety Management (88.1%), and Out-of-Home Care

FLORIDA INSTITUTE FOR CHILD WELFARE

Child Protection Investigation Specialty Track Forty trainees completed the CPI Specialty Track training and 85 percent of trainees (n = 34) completed both the initial pre- and postKnowledge Assessments. Table 8 shows the proportion of correct responses for modules examined from this curriculum. Similar to results of the CM Specialty Track assessments, overall, there was significant improvement in items correctly answered at the posttraining CPI Specialty Track assessment compared to the pre-test. Trainees answered 74.8 percent of questions correctly at pre-test and 86.9 percent of questions correctly at post-test. A greater number of items were answered correctly following the training for each module assessed. Significant increases were observed for all modules with the exception of the Assessment of Hotline to Assignments module. Although a significant increase was not observed with this module, almost 80 percent of trainees correctly answered these questions at pre-test. Following the CPI Specialty Track training, modules with the greatest percentage of correct responses was observed for Commencement of the Investigation: Initial Contact and Present Danger (100%), Introduction to Child Protection Investigations (94.1%), and Present Danger Assessment (93.1%). From pre-test to post-test, the greatest increase was observed for material assessed in the Introduction to Child Protection Investigations modules and the Present Danger Assessment module.

Table 8: Average Module Scores for CPI Specialty Track Pre- and Post-Knowledge Assessments

Table 9: Average Module Scores for 6-Month CPI Specialty Track Knowledge Assessment (n = 9)

Pre-Test

Post-

Paired

Pre-

Post-

Follow-

(n = 34)

Test

t-tests

Test

36.4%

72.7%

(n = 34)

(6-month)

Introduction to Child Protection Investigations (1 item)

73.5%

94.1%

Assessment of Hotline to Assignments (2 items)

79.4%

85.3%

Commencement of the Investigation: Initial Contact and Present Danger (3 items)

95.1%

100%

Present Danger Assessment (6 items)

76.0%

93.1%

***

The FFA- Investigation and Safety Planning (5 items)

82.9%

88.2%

Developing In-Home or Out-Of-Home Safety Plan (7 items)

58.0%

74.4%

***

Overall (24 items)

74.8%

86.9%

***

Note: *p < .05; **p < .01; ***p < .001; ns = not significant

Trainees participated in a follow-up assessment 6 and 12 months after completion of the specialty track curricula, responding to the same questions asked in the pre- and post- assessment. Unfortunately, the response rate for this assessment dropped drastically at the 6-month follow-up and again at 12-month followup. Due to turnover and missing data, findings only include data collected from nine CPI trainees across sites for the 6-month followup assessment and one CPI trainee at the 12-month follow-up assessment across sites. Data from nine trainees who completed the CPI Specialty Track training follow-up assessment at 6 months are shown in Table 9. Overall, there was improvement in the proportion of items correctly answered at the follow-up assessment compared to the pre-test. Trainees answered 72.7 percent of questions correctly at pre-test and 75.0 percent of questions correctly at follow-up assessment. About 85 percent of questions were correctly answered at posttest, resulting in a significant decrease in scores by follow-up assessment. With the exception of the module on Developing InHome or Out-Of-Home Safety Plan, a greater proportion of items were answered correctly at follow-up compared to the pre-test for each module assessed. In fact, scores on the Developing InHome or Out-Of-Home Safety Plan decreased significantly from 76.2 percent answered correctly at post-test to 49.2 percent at follow-up assessment. Modules with the greatest percentage of correct responses at follow-up assessment were Commencement of the Investigation: Initial Contact and Present Danger (100%), Present Danger Assessment (85.2%), and Assessment of Hotline to Assignments (83.3%). At each time point, scores on the Developing In-Home or Out-Of-Home Safety Plan module were the lowest. Summary Taken together, findings from Knowledge Assessments administered to pre-service trainees provide initial evidence that knowledge was gained as a result of the CORE curriculum training as well as the Specialty Track training. Trainees correctly answered significantly more questions at post-test compared to pre-test for the CORE Knowledge Assessment and both specialty track assessments. Further, for most modules assessed, a greater percentage of items were correctly answered at post-test than at pre-test for each curriculum. Although some scores decreased marginally by the 6-month follow-up assessment and again at 12-month assessment, overall, scores were still greater than those observed at pre-test. Due to turnover and missing data, there was a significant amount of attrition over time. The sharp decrease in response rate by follow-up assessment is a major limitation to the reliability of these findings. FLORIDA INSTITUTE FOR CHILD WELFARE

RMA

Introduction to Case Management (1 item)

90.9%

Case Transfer (5 items)

100%

96.4%

100%

Safety Management (6 items)

80.3%

81.8%

93.9%

Out-Of-Home Care (7 items)

71.4%

80.5%

81.8%

Family Engagement Standards: Preparation and In-troduction (5 items)

78.2%

81.8%

72.7%

Family Engagement Standards: Exploration (3 items)

75.8%

69.7%

78.8%

Family Engagement Standards: Case Planning (7-items)

71.4%

85.7%

84.4%

Evaluating Family Progress (7 items)

79.2%

89.6%

84.4%

Overall (41 items)

74.5%

83.6%

Note: *p < .05; **p < .01; ***p < .001; ns = not significant

CASE MANAGER AND CPI OBSERVATIONS AND FOCUS GROUPS One goal of the study was to assess the skill competency of case managers and CPIs in the field at multiple intervals throughout their first year on the job. Study team members originally planned to conduct in-person observations at each site at 1 month, 6 months, and 12 months post-training. These in-person observations took place at the one-month point for all sites but one. An observation tool was developed using the Florida Child Welfare Competencies as a guide, focusing most concretely on skills related to family engagement, risk and safety assessment, and identification of strengths and needs (see Appendices B and C for the observation checklists). The checklist includes 13 items for case managers and 11 for CPIs. Each item corresponds with a particular competency and one or more objective from the CORE pre-service curriculum (see Appendix D for a key matching these items). For in-person observations, instructions were to indicate whether a skill was demonstrated by marking “Yes” or “No,” or to mark “N/A” if there was not an opportunity for the skill to be observed. The observer was asked to include an example for each of the skills demonstrated and to remark on any other factors that were important to note for the observation. For focus groups, the items from the checklist were adapted to questions, with the facilitator probing for examples of each competency or skill. After the study team recognized that coordinating observations for individual participants across multiple agencies and regions was a challenge and burden for the sites, in-person focus groups were conducted at the 6- and 12-month intervals instead. Focus groups allowed participants to share feedback on how they understood and integrated specific competencies into their practice in an environment where participants could learn from and build off of each other’s input. The Six-month focus groups were conducted in-person, and the 12-month focus groups were conducted virtually due to the COVID pandemic. A member of the research team coordinated with site contacts to schedule focus groups that included as many case managers and CPIs from the original training cohorts as possible. Focus groups were recorded with participants’ permission, and digital audio recordings were sent to a professional transcription company to produce electronic documents of all focus group discussions. Study team members developed a coding scheme based on the focus group protocol used to guide questions (see Appendices E 8

and F), which were modified from the core competencies used in the one-month observations and the electronic self-assessment surveys completed by case managers and CPIs at 1, 6, and twelve months after unit placement (see Appendices G and H). Baseline Observations One-month observations took place between January and June 2019 with a total of 37 participants across the five study site regions, including six CMOs, two DCF circuits, and one Sheriff’s Office. Multiple types of interactions were observed, including home visits, court hearings, and staffing meetings (see Table 10). For case managers, the evaluation team observed 22 home visits, two court hearings, and one permanency staffing. A focus group with case managers was also conducted during one site visit when local site management felt that parents would not be comfortable with an additional person observing during a home visit. For CPIs, the evaluation team observed six home visits and one transfer staffing. Table 10: Numbers of Observations by Type and Position Site Visit Type

CMs

CPIs

Total

Home Visit

Court Hearing

Permanency Staffing

Transfer Staffing

Total

Case Manager Observations Results from the case manager observations are provided in Table 11 below. Scores indicate that case managers demonstrated desired skills the vast majority of the time (95%) and did not demonstrate skills in only nine instances, or two percent of the time. However, it is important to note that interactions were not applicable or observable 36 percent of the time. Most often, this was because the purpose of the home visit or other interaction did not warrant demonstration of the particular skill. The most frequently demonstrated skills were related to utilizing interviewing techniques and strategies, engaging and assessing families from a strengths-based perspective, and discussing progress in recommended services. Examples of these skills include the following: “Case manager asked open-ended questions, repeated information back, and had a conversation– not just note taking,” and “Case manager was strengths-based with the youth, very encouraging and pointed out many areas where the youth is doing well.” Instances in which case managers demonstrated lack of skills include examples such as not being very engaged with the caregiver, only being strengths-based with some family members or caregivers, and not discussing the safety plan during the visit.

services based on the FFA, and demonstrating knowledge and application of Florida’s Child Welfare System Guiding Principles. Examples of these skills include: “CPI was very strengths-based and acknowledged positive steps Mom has taken already,” “Discussed appropriate referrals and available services,” and “Discussed child safety and goal to keep family together.” For instances in which observers did not see sufficient skill being demonstrated, some of the reasons included that the CPI was focused on completing a form and missed some opportunities to delve deeper; the CPI used a document to guide the interview and did not ask a lot of probing questions; or not accommodating a family that spoke another language, relying instead on the family member being investigated for translation. Table 11: Case Manager Observation Skill Counts Yes

N/A

Demonstrates sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.).

Communicates with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs.

Interacts with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a nonconfrontational manner.

Utilizes interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and con-duct age-appropriate interviews with the child, caregiver, and family.

Interprets the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations.

Conducts or has conducted for this case ongoing, ageappropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

Engages and assesses families from a strengths-based perspective.

Develops and implements (or has developed and implemented) a case plan/safety plan based on strengths-based assessment.

Identifies and facilitates referrals to appropriate services based on the Family Functioning Assessment.

Discusses progress in recommended services.

Demonstrates active and ongoing assessment of the safety management plan.

Demonstrates use of safety analysis and planning criteria needed to modify safety plans.

Demonstrates knowledge and application of Florida's Child Welfare System Guiding Principles.

179

136

Total

CPI Observations

Scores from the CPI Observations are provided in Table 12. Skills were observable 61 percent of the time. Out of observable skills, CPIs demonstrated proficiency in desired skills 83 percent of the time and showed lack of skill demonstration 17 percent of the time. The areas where skills were most frequently demonstrated include engaging and assessing families from a strengths-based perspective, identifying and facilitating referrals to appropriate

FLORIDA INSTITUTE FOR CHILD WELFARE

Table 12: CPI Observation Skill Counts Yes

N/A

Demonstrates sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.).

Communicates with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs.

Interacts with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a nonconfrontational manner.

Interprets the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations.

Conducts or has conducted for this case ongoing, ageappropriate assessment activities that ascertain if a child’s

Engages and assesses families from a strengths-based perspective.

Develops and implements (or has developed and implemented) a case plan/safety plan based on strengths-based assessment.

Identifies and facilitates referrals to appropriate services based on the Family Functioning Assessment.

Demonstrates knowledge and application of Florida's Child Welfare System Guiding Principles.

Communicates clearly the information collected during the investigation that has been used to determine the need for case management.

Total

physical, mental, social, and educational needs are met.

6- and 12-Month Focus Groups: Case Managers The purpose of the case manager 6- and 12-month focus groups was to examine how helpful the child welfare pre-service training was to case managers in the day-to-day work they do. Case managers from six sites participated in the focus groups. Focus groups ranged from one to 10 people, with smaller groups at the 12-month mark. The small number of participants is likely due to a number of factors, including high turnover rates by the 12-month point, which ranged from 58 percent to 100 percent among case management sites, with an average rate of 81 percent. During the focus groups, case managers were asked for their opinions and experiences about the impact of the child welfare pre-service training on their work, since at the time of the focus groups most had been working for one year or longer as a case manager. Participants were also asked, specific to each domain, what other factors helped them develop their skill sets in each area and to what extent those factors compared to the pre-service training.

or discussing culturally specific services. Both 6- and 12-month responses to this question indicated that case manager do feel comfortable working with clients of different cultural backgrounds than their own. The majority also agreed that the training had covered this topic, although responses were mixed regarding whether it was briefly covered or dealt with more specifically and/or more in-depth during training exercises such as role playing. Case managers specifically felt that the pre-service training helped them better understand differing customs and etiquette within the home and gender and power differences they might encounter when on a home visit, and that the training fostered a value base of openmindedness toward each family they served. Many case managers felt that their demonstration of cultural competency with clients had been more influenced by other experiences they had, such as previous jobs, coworkers, supervisors, and learning while gaining more experience in their case management work. Six-month focus group participants also mentioned knowledge gained about interpretation services, different languages, and dialects, in addition to a greater sensitivity and awareness of the implicit bias around issues of poverty and race. Communication Case managers were asked if they felt the child welfare preservice training helped them communicate with family members and collaterals in order to discuss their needs and concerns and were asked to share examples. Both 6- and 12-month participant responses were generally split between two sentiments: that the pre-service training helped by way of role playing, teaching a strengths-based focus, and interviewing techniques, and that the training did not match up with the reality case managers faced once in the field. Regarding aspects of the training that did help case managers communicate, role playing exercises were most frequently discussed. Some participants added that the role playing helped to prepare them to some extent: “I run into really ridiculous scenarios, so you can’t prepare for everything, you really can’t. But as best as the role playing could help, absolutely as best as it could.” In the 6-month focus groups, participants expressed the need to be more prepared for families yelling at them and being angry. Participants shared that the training perhaps helped them have the expectation that this might happen, but it did not necessarily really help them develop the skills to use in the moment when anger or even aggression occurs. Regarding the second primary sentiment expressed, many case managers felt a disconnect between the training provided and the situations they encountered in the field. Sentiments ranged from the simple reality that every family is different, to statements such as, “It’s rough, it’s overwhelming, it’s frustrating. It’s all of the above. And no matter what kind of degree you have, it does not prepare you for reality,” and “Most of the time, for me, the training kind of went out of the window and I had to think outside of the box for that moment.” However, focusing on how the training could improve, some respondents offered that the scenarios presented seemed to be best case scenarios and fairly “cookie-cutter,” indicating that increasing the frequency, diversity, and complexity of the training scenarios would be helpful. Increased preparation for handling parent hostility was another area that was highlighted as needing to be augmented.

Cultural Competency

Interviewing Techniques and Strategies

Case managers were asked if they felt that the child welfare preservice training helped them show sensitivity to cultural and ethnic differences among their clients, such as respecting a client’s communication style, acknowledging their value system,

Next, case managers were asked if they felt that the child welfare pre-service training helped them develop interviewing techniques and strategies to conduct age-appropriate interviews with children, caregivers, and families, such as demonstrating respect for clients,

FLORIDA INSTITUTE FOR CHILD WELFARE

practicing active listening, and using exploring and focusing skills. The majority of 6- and 12-month case managers who participated in the focus groups agreed that pre-service training helped them with their interviewing techniques. Example interview questions provided during pre-service training were mentioned as a helpful source case managers could review and utilize once in the field. Role playing specific to interviewing children was brought up as being helpful. Field days were listed as helpful, particularly in terms of highlighting personal differences in interviewing styles across experienced case managers. Participants’ views were split on whether interviewing skills could be taught in class or if they could only be taught in the field. Other comments regarding the impact of pre-service training on interviewing techniques had to do with the necessity of learning as you go and having different experiences in the field with cases, as well as learning from coworkers who might be better at communicating with certain age ranges of children. Unique to the 6-month groups, one region from the study utilized a training program called Avatar, which is intended to be a simulation of actual interactions between case managers or CPIs and child welfare involved families. The program uses a combination of onscreen “avatars” and live feedback from actors to most closely represent “real life” experiences. Case managers from this region described this program as being helpful in enhancing their interviewing skill set and said that it increased potential to help them if more time was allowed to interact with the simulation family. In addition, several case managers at the 6-month point discussed a disconnect in philosophies of care as demonstrated in how each case manager approached working with families and to what extent they were able to be strength-based. Case managers shared that they felt pre-service training promoted belief in a strengths-based perspective, but when they joined the active case managers, they encountered more of a child safety focus above family advocacy. Family Functioning Assessment Case managers were asked if they felt that the child welfare preservice training helped them interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations and referrals to services. In regard to helping them interpret FFAs, the majority of case managers at both 6- and 12-month focus groups felt that they had not been adequately prepared by pre-service training in this area. Some stressed that more time spent on FFAs was needed, with a clarification being that the current training helps them know what to expect from the FFA process but does not demonstrate the complexities of completing one correctly. A suggestion was made that it would be helpful if trainees could complete an FFA-O for an actual case under the guidance of an active case manager during their pre-service training. Another suggestion was for pre-service training in this area to cover other parent home assessments and requirements for supporting documentation. The 6-month groups also shared that they felt challenged by CPIs not finishing the initial FFA, caseworker turnover, and in both cases, the resulting blank information spaces on the FFAs as a new case manager took on the case. When case managers were asked if the pre-service training helped them make appropriate referrals for services, the responses were generally “no.” Case managers clarified that while pre-service training did help them recognize what issues were troubling a family based on the FFA results, it did not prepare them for the referral process. New case managers turned to coworkers and their supervisors for help understanding the referral process and availability of services to refer families to. Some agencies also provided new case managers with lists of services and points of contact, which was seen as helpful. In addition, several respondents mentioned that they routinely use FFAs in conjunction with other sources of information to develop a plan for families. These other FLORIDA INSTITUTE FOR CHILD WELFARE

sources included shelter orders; comprehensive behavioral health assessments; and intake, triage, and referral (ITR) staffings. Unique to the 6-month focus groups, the FFA and referral process domain was also an area where case managers shared feelings of emotional strain related to their jobs. Participants again raised the issue of a significant disconnect between the pre-service training and the situations and work demands they encountered once in the field. Age Appropriate Assessment Case managers were asked if they felt that the child welfare preservice training helped them conduct ongoing, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met. In terms of ways the training helped case managers understand how to conduct age-appropriate assessments, most participants at both the 6- and 12-month point felt that it had given them a basic skill set, particularly with younger children, but many suggested they learned this while providing case management services afterwards and/or they learned this from a school or internship program prior to their pre-service training. Strengths-Based Perspective Case managers were asked if they felt that the child welfare preservice training helped them engage and assess families from a strengths-based perspective. In terms of approaching interactions from a strengths-based perspective, case managers at 6 and 12 months agreed that their pre-service training did help them, providing some examples detailed in the next paragraph. The two caveats were that sometimes the nature of case management made strengths-based approaches challenging, and many times case managers learned to incorporate the strengths-based philosophy of care as they experienced more cases with guidance from their supervisors. First, case managers did agree that the importance of viewing families from a strengths-based perspective was covered in preservice training; “I think they did discuss it a lot about focusing on the family’s strengths,” one case manager stated. However, all case managers acknowledged that while the training did support a strengths-based perspective, sometimes this was challenging to actually put into practice; “It’s very, very hard sometimes. Very, very difficult for me to think of some strengths for some of these cases. It’s very, very difficult, especially when you’re going through a TPR [termination of parental rights] trial and stuff like that and everyone’s focusing on the negatives. You forget all the strengths that a family could have,” another participant explained. Another sentiment consistently shared was that pre-service taught what ideally should be done, but that once in the field, other demands on case manager time took priority. In addition, case managers described how limited time spent on cases can inhibit their ability to be strengths-based. Some case managers also felt that while pre-service trainers discussed what a strengths-based perspective was, being able to apply it came more from ongoing, on-the-job experience. Guidance from supervisors was also mentioned. One nuance to this was that the 6-month group reported that they sometimes experienced disconnect between the strengths-based philosophy of care taught in the training and the child safety driven practice found in the field. Case and Safety Plan Implementation Case managers were asked if they felt that the child welfare preservice training helped them develop and implement case plans/ safety plans based on strengths-based assessment. Generally, participants at both data points acknowledged that the pre11

service training helped lay a foundation but did not actually lead to development of this skill set, specifically. Supervisors were often turned to for guidance on case and safety plans post-training. One case manager shared, “I don’t remember that much about safety plan in pre-service. Again, I think, for me, a lot of the information that I gained has definitely been post pre-service with my supervisor.” Suggestions were made by the 6-month group that pre-service spend more time on FSFN training and having trainees carry a case during training with the supervision of an existing case manager. The sense was that this type of case carrying, even one case, might help trainees know whether or not they were well matched to the work ahead. Florida’s Child Welfare Guiding Principles Case managers were asked if they felt that the child welfare pre-service training helped them demonstrate knowledge and application of Florida’s Child Welfare System Guiding Principles. Six- and 12-month participants indicated that this was included in pre-service training and they were shown where to find this information in writing when needed. Six-month case managers also reported using this information in practice, providing examples of mitigating their own bias, treating clients with respect, and sharing instances where codes and statues needed to be referenced and shared with parents on issues such as necessary background checks. Discussing Progress with Clients Case managers were asked if they felt that the child welfare preservice training helped them feel confident in discussing progress with clients in their recommended services, and what other factors may have helped with this. In terms of how case managers felt that pre-service training helped build confidence in discussing plans with families, results at each data point were mixed between those who felt it was very helpful and those who felt they learned by doing, more so than the pre-service curriculum. Also, helpful was preparing case managers to expect different communications patterns, needs, and styles from parents. For example, some parents might need daily assurance, others might curse a case manager out, and still others might participate but want as little contact as possible. Other case mangers acknowledged that the training covered this area briefly but could not attribute their skills in this area to the preservice training. In addition, 6-month case managers explained they also came out with a sense of where to draw the line in terms of the extent to which they could help parents who were not ready to help themselves. A case manager explained, “I’m always straight up with my parents on their progress because a lot of them don’t really take it all that serious. And I’m like, ‘Well this is your child that’s on the line.’ That’s my job, is to assess safety of the child, help the parents. But you can only do so much at the end of the day for them.” This was a common sentiment in reference to parents failing drug screens and not showing up for hearings or treatment. Others clarified that the pre-service training covered rapport-building with families at the front end of investigations and case management, but how to move a case forward so that parents regained custody of their children was not covered, and fell into the area of skills they needed to learn as they went along. Active and Ongoing Assessment Case managers were asked if they felt that pre-service training helped them to demonstrate active and ongoing assessment of their clients’ safety management plans, and what other factors may have helped with this. In terms of training, both 6- and 12-month

FLORIDA INSTITUTE FOR CHILD WELFARE

responses generally indicated, again, that a foundation had been provided but nothing further. Others mentioned the demands on their time as a barrier to conducting ongoing assessments on a case. Some 6-month focus group participants needed to be provided a description of what a safety plan was during the focus group in order to answer the question, and others indicated that they did not have much experience with them. Modification of Safety Plans Next, case managers were asked whether they felt that the child welfare pre-service training helped them make good use of safety analysis and planning criteria needed to modify safety plans. In terms of impacting their ability to modify safety plans, the majority of case managers at 6 and 12 months felt that pre-service training was helpful to some extent. One case manager stated, “Yeah. I think we went over that. And again, I like to kind of go back to the, like, handouts we’re given and to go back to those resources when evaluating safety plans. So, I think we were definitely given the resources to do so.” Another recalled their trainer focused on the five safety criteria that they needed to always re-evaluate, particularly with in-home cases. However, other case managers did not feel that pre-service training had adequately prepared them in this area. One stated, “I remember coming outta training I was a little confused on safety plans. Like I feel like I didn’t learn how to modify, or when to modify safety plans until it was discussed with me with my supervisor.” Again, a clear trend was seen in this domain as was seen in others: pre-service laid a basic foundation, but fieldwork provided skill set development. In addition, 6-month focus group participants felt that the training had helped them use collaterals to determine when a parent was not being honest about abiding by an established safety plan. Improvements Finally, case managers were asked how they would improve the child welfare pre-service training, such as areas to add, lengthen, or shorten in the curriculum or its delivery. The following curriculum recommendations come directly from case managers during these 12-month focus groups. Increase Field Days Case managers were also interested in more and better organized field days that are scheduled earlier in the week when there are case managers working on active cases that can be shadowed. A case manager explained, “The best thing is more of those field days. Because those field days are really where you’re going to see, I mean you don’t know what’s going to come up during, you know while you’re shadowing and what’s going to come up with the family while you’re observing.” Participants stressed the need to learn under less scripted conditions than what the current role-playing exercises offered during pre-service. Another case manager shared about pre-service, “It just wasn’t authentic enough. It helped as best as it could…it was very robotic mostly.” Shadow Experienced Case Managers Case managers also felt that having an actual case that they could work on with an experienced case manager throughout their training would be beneficial. Participants felt that examples during training were helpful, but without the complexities of real life, they felt artificial to an extent. In addition, it is important to stress that case managers felt the existing schedule of shadowing and field days could be improved in that they were normally sent to case managers on Fridays when the case managers were not working, so trainees were left to sit and read their pre-service manuals. A 12

proposed option for better scheduling field days would be to pair field experiences directly after related training content. For example, if FFAIs had just been reviewed in class, participants felt it would be helpful, then, to see a case manager complete one in the field. Navigating the Judicial Process Case managers at 6 months, specifically, were interested in spending more time during pre-service training on navigating the judicial process; for example, what to expect at a shelter hearing. A case manager shared, “There’s a video that we were shown to prepare for court, but it doesn’t prepare you for court.” Case mangers felt thrown off by the presence of attorneys in the courtroom and had not been sure what to focus on in order to prepare for Court. One participant mentioned they did not know they needed to follow the recommendations in court orders or given in Court verbally (there was an issue mentioned that many judges and magistrates were giving orders in Court verbally but not putting them into written orders and new case managers were getting caught in the cross fire of people arguing about what was said and ordered and whether or not the case manager followed the orders or followed up on the parents following the orders). Increase Training Content and Duration on Interviewing Skills Twelve-month focus group participants specifically requested that more time be spent in pre-service on interviewing parents and children. They indicated that enough time had been spent on interviewing other parties, and that if needed, there might be preservice content areas that could be shortened with more time spent on interviewing skills. More time learning how to interview younger children was also requested. Increase Training on Case Management Functions Some responses centered around wanting more time on specific aspects of case management rather than the front-end maltreatment that caused the investigation or aspects of CPI work. Participants were interested in the details of day-to-day activities and the processes and forms case managers need to complete. Specifically, they asked for more content on handling drug screens and other parent home assessments (e.g., if parents are not living together and the child is spending time in each home). In addition, some case managers that ended up focusing on adoption felt that the pre-service training needed enhancement in this area, or possibly a third specialty track. Increase Training on Time Management Inclusion of experienced case managers and supervisors in the preservice training to discuss stress management and how to manage workloads and schedules was requested by case managers at 6 months, specifically. In addition, it was suggested that including individuals in these positions may help mitigate the disconnect in philosophies of care that newly trained case managers face when they actually initiate case work post-specialty track training. Case Manager Peer Advocate Another suggestion made specifically by 6-month case managers was to establish a peer advocate for new case managers in addition to their supervisors. Case managers repeatedly mentioned that supervisors and lead case managers in their units were already shouldering a lot of responsibilities, and in some cases, were near a breaking point. In other instances, case managers had been made to feel by their supervisors as though they should not need much FLORIDA INSTITUTE FOR CHILD WELFARE

help after having already gone through pre-service training. While there were many case managers who heralded their supervisors as being very helpful, there were also participants who indicated that their supervisor was either “maxed out” or did not see it as the supervisor’s role to guide new case managers in their case work. This finding seems to support the suggestion of a neutral person who could also provide some ongoing training support as needed to the new case managers, without fearing judgement or overworking a coworker. One case manager wished, “If we just had somebody that we could just go to. You know, just say, ‘This is what I’m dealing with today,’ to kind of let your hair down. You know what I mean? And not feel like it’s going to count against you.” Case Manager Turnover Unique to the 12-month focus groups, participants were asked to reflect on why some of their initial training cohort chose to leave their case manager positions, and why others chose to stay at their jobs. The FICW specifically requested that this question was added for the 12-month focus groups. Participants named several factors in their decision to stay employed as a case manager, such as loving the work they do, feeling that they made a difference, having a supportive supervisor, having supportive coworkers, and some case management areas perhaps being easier than others to stay in. First, participants who remained in their positions talked about needing to find something about the job that they really loved, and with this attachment came the willingness to stay. One case manager explained, “It’s easy to get bogged down by the negative part of this job, and so if you don’t really love what you do then it’s easy not to stay,” Participants talked about the joy that came from helping children, helping women dealing with an abusive partner, helping families better cope with economic strain, and often just giving someone at a high stress point the gift of having another person to just sit down and take the time to listen. Second, those who remained in their roles as case managers at 12 months talked about feeling like they were making a difference in the lives of children and families. Participants discussed successful reunifications and adoptions. A case manager explained, “When you see a child that’s been in care for a while and you actually get them an adoptive family and now, they’re gonna have a forever home. Those small moments kinda outweigh the times when I’m like, ‘Oh my god, this job is too much.’ And those are the moments that you stay for.” Any time a child was successful, and a case manager could see and believe they would no longer need services, this helped workers get through more difficult times. As described, this type of hope and satisfaction existed even though only a small percentage of cases fit this pattern. Participants who remained after the one-year mark expressed that any evidence of successes in a child’s life was enough to stay. Third, focus group participants were very clear that a supervisor could “make or break” the journey as a new case manager. Those taking part in focus groups seemed to reflect favorably on their supervisors, while describing peers who had quit as those who often did not have the benefit of a good supervisor. A case manager explained, “It depends on the supervisor you get. I have an awesome supervisor, so I always feel supported and I always feel like if I don’t have the answer, she does.” Another reiterated a similar feeling: “I think a lot of this job has to do with who your supervisor is and the relationship you have with them, because you need a lot of help in this job and a lot of support.” Fourth, those who remained after one year of employment expressed that they had supportive coworkers. Participants shared that sometimes support came from having colleagues take over tasks, such as in-office work when case managers were deluged with home visits or attending home visits when case managers had to be in the office. Participants also shared stories of mutual 13

emotional support, in that coworkers would check in with each other, make sure they were taking care of themselves, allow each other to vent, etc. Another key area of support coworkers provided was sharing information about case management tasks and how to best accomplish tasks. Finally, a theme emerged—some areas of case management might be easier to work in than others—in a few ways. First, direct statements from participants such as, “I’m in adoptions, so I am in a much happier end of it, for the most part. I mean, I have my crazy cases as well, don’t get me wrong, but I think it’s easier for people to stick around in adoptions than primary.” Second, participants recounted that some peers had been trained for one area of case management that was perceived as easier (e.g., prevention services), but then once released to their agency, there was a greater need in dependency. They shared that this had led to turnover and emotional strain and perceived it was because the peers had trained and sought out one type of work, but then had been assigned to a more challenging area. Finally, participants were asked if they knew where their peers had gone to work after leaving the agency, and some recounted that peers were in social work or mental health roles, just in positions that are not as taxing as what they experienced as a child welfare dependency case manager. More specifically, reasons provided as to why peers left their case management position included emotional strain, an unsupportive supervisor, turnover as a domino effect, and low salaries/lack of recognition. First, participants discussed, at times, feeling overwhelmed due to caseload size, feeling under paid, and feeling like they were not appreciated. Typically, these feelings led to leaving the job, but there were also reports of peers needing to seek medical treatment for resulting anxiety. Some members of the focus groups felt that the key to not getting overwhelmed was the ability to be in the moment, working step by step, and not being as concerned with work that was left for the next day. Others explained that empathy towards clients, while beneficial to case work, sometimes worked against individual case managers in that it led to emotional strain. Participants also explained that the reason coworkers left their position was because of an unsupportive supervisor. As one case managed stated, “Some of the supervisors are a lot more helpful. And others are just like, ‘Well, good luck.’” Those who stayed seemed to have supervisors who were seen as mentors with open-door policies and a willingness to pick up where pre-service training left off. This was described as a future-oriented attitude, in terms of taking the time to train new case managers to do well. Another difference among supervisors that impacted turnover was a perceived flexibility toward work hours versus requiring case managers to be in the office during office hours in addition to evenings and weekends for home visits. Third, focus group participants talked about the impact of turnover on those who remained in case manager positions, in that it tends to have a domino-like effect, causing more strain on those who remain, and more guilt on the part of those who leave; turnover was perceived as a “vicious cycle.” Case managers described situations where workers left because, not only were they new and lacking a helpful supervisor, but they were also, at times, thrust into taking the lead on tasks and cases they “knew almost nothing about.” Fourth, there was a consistent theme that case managers get a lack of recognition and salaries are too low for the work hours and emotional strain of the work. Some participants talked about being able to stay as long as they had because they did not yet have families with children or partners that depend on them financially. They mentioned that it was tough for single parents to support their children on a case manager’s salary. Others discussed peers wanting to further their education and not being able to do that alongside case manager responsibilities.

FLORIDA INSTITUTE FOR CHILD WELFARE

Impact of COVID-19 Pandemic The 12-month focus groups were conducted either before or during the first few months of the COVID-19 pandemic while shelter-inplace orders were in effect throughout the state. At the request of the FICW, participants were asked how the emerging pandemic was impacting their case management work. Not surprisingly, case managers experienced confusion and uncertainty, particularly as the pandemic developed. Some case managers talked about wishing their agency had been better prepared for a pandemic scenario and how agencies made the choice to send case managers home before lockdowns were enforced. Case managers shared that they were initially expected to do a screening measure of families for COVID-19 symptoms in the home without having personal protection equipment such as masks. These items, as well as conducting home visits on Zoom and other remote platforms, seem to have become routine as the pandemic matured. Case managers were generally happy that they were able to conduct work at home but noted that it did come at a potential cost—both to families and to case managers. Families were no longer going through drug screens in many cases, which potentially left openings for increased relapse and noncompliance. Case managers who were in the 12-month focus groups seem to have already established a network of colleagues who provide support to each other, for example, via text messaging, but they noted that new case managers may or may not have the skillset to establish this online network of support. Focus group participants advised that continuing to have peers to ask questions and vent to will be helpful to new workers as they come on board. Summary The purpose of the 6- and 12-month case manager focus groups was to gather input from each cohort of trainees that completed preservice training on the extent to which they felt that pre-service or other supports or training helped them develop skills related to core competencies. Overall, case managers at 6 months felt that their pre-service training was beneficial to developing the necessary skill set for their daily work as child welfare professionals. Notably, they indicated that information provided on identifying and understanding maltreatment was adequately covered. Areas of the pre-service training participants found most helpful were role playing exercises, shadowing experienced child welfare professionals, and field days. Case managers also identified other experiences as having had a strong influence on their skill set, in some areas, more so than the pre-service training. These types of influences were most commonly noted as previous work experience serving children (e.g., previous child welfare experience, teaching, childcare, or serving as a Guardian Ad Litem), previous education (e.g., a social work degree program), on the job experience in the last 6 months commonly referred to as “learn as you go,” and help provided by supervisors and coworkers over the past 6 months as needed. One theme that emerged from the 6-month data was a perceived lack of curriculum on specific case manager functions and how to complete them. Case managers felt that the training was geared more toward CPIs and proposed several areas they needed more training on, such as navigating all aspects of the judicial process, case and safety plan updates, working in FSFN, aspects of adoption work, and time and stress management. Case managers also asked for more field days scheduled at times when experienced case managers were conducting home visits. There was an indication that field days were sometimes scheduled on days when experienced workers were not available to shadow, causing trainees to miss out on valuable in the field experience. Other suggestions included more role playing exercises, jointly carrying an active case during pre-service training (with an experienced child welfare worker), and providing a peer advocate 14

for case managers to go to with questions once in the field, as some expressed a reluctance to ask for help from supervisors who had indicated that pre-service training should have already provided the necessary skill set. Another common theme across focus groups was that the examples presented in training seemed to represent best case scenarios that did not showcase the complexity of issues families and child welfare professionals face in reality. Participants recommended either more field time and hands-on work to address this, or more complexity added to the examples within existing training curriculum. Overall, case managers at the 12-month mark indicated that they were provided a foundational understanding of their job responsibilities through pre-service training. However, they found that it was difficult to apply the knowledge and skills they gained to the field when they returned to their respective units. Specifically, case managers reported that their understanding of FFA-Os and safety planning, two critical functions of their daily responsibilities, were not taught and/or practiced at a level that gave them enough knowledge or confidence once they were working directly with children and families. 6- and 12-Month Focus Groups: CPIs As with the case manager focus groups, the purpose of the 6-month focus groups with CPIs was to examine how helpful the child welfare pre-service training was to CPIs in their daily work. During the focus groups, CPIs were asked for their opinions and experiences about the impact of the child welfare pre-service training on their work after 6 months of being in the field. Participants were also asked, specific to each domain, what—if any other factors—had helped them develop their skill sets in each area and to what extent compared to the pre-service training. Participants from all three CPI study sites participated in the 12-month focus groups, including seven participants in total. Attrition from these cohorts ranged from 30 percent to 56 percent, with an average of 47 percent across all three sites. Focus group participants shared feedback on the influence that pre-service training and other factors continued to have on core competencies after 6 months and one year on the job. In addition, at 12 months, CPIs were asked to discuss the impact of the COVID-19 pandemic on their work, as well as their perceptions of turnover among colleagues. Findings are shared here by domain, with examples quoted when relevant to enhance context. Cultural Competence The cultural competence domain refers to how pre-service training helped CPIs learn to show sensitivity to cultural and ethnic differences among clients, such as respecting clients’ communication style, acknowledging their value systems, or discussing culturally specific services. Responses at both 6 and 12 months were similar, in that CPIs found training around this topic helpful and recalled several ways that pre-service training explicitly addressed cultural competency, such as discussing the importance of respecting different cultural beliefs and practices and approaching families in a non-judgmental way. Specific examples that CPIs remembered from pre-service training were understanding that they may need to remove their shoes before entering a house, working with parents who may not allow pictures to be taken of children, and adjusting interview questions to accommodate different religious beliefs. Respondents reported that understanding differences in parenting and disciplinary norms helped CPIs interact more openly with families, have a more flexible understanding of their actions, be prepared to encounter a diverse clientele, and better assess danger threats. It was also suggested that learning about other trainees’ backgrounds during pre-service was beneficial for developing an appreciation of cultural differences.

FLORIDA INSTITUTE FOR CHILD WELFARE

Although respondents agreed that some components of pre-service training were beneficial in developing cultural competence skills, there was wide agreement that previous work and life experiences were at least as helpful, if not more so. Being naturally observant of people, respecting other’s decisions and lifestyles, and having existing exposure to different types of people and communities were all seen as enhancing CPIs’ capacity for cultural competence. Many respondents indicated that their clientele was particularly diverse, and while pre-service training gave them a framework for cultural competence and a sense of what the Department’s expectations were, the hands-on aspect of working directly with clients contributed more concretely to actually developing their skills. Age Appropriate Assessment Assessment refers to the ways pre-service training helped CPIs conduct appropriate assessment activities to determine if a child’s physical, mental, social, and educational needs are met. At both 6 and 12 months, the majority of CPIs felt that foundational concepts of assessment were reviewed in pre-service training. Some of the key concepts participants remembered from pre-service include the following: the settings and strategies that work best for assessing children (e.g., school may allow children to open up more because they are not as concerned about their parents’ reactions), the importance of understanding children’s developmental stages and how this impacts their ability to respond to questions, how to identify disabilities in children and adults, and how to assess home environments to gauge well-being. One CPI brought up an example of a helpful activity from training in which trainees broke into different groups to practice assessments, each group with a different age group focus. Another respondent found it helpful when trainers integrated examples of how a child’s age and maturity level should be taken into consideration at various points throughout the curriculum. However, many CPIs said that the most influential factor in helping with age-appropriate assessments was on-the-job experience, as it allows them to see children with different needs, hear the different ways children respond to assessment questions, and gain experience assessing home environments. One respondent noted that the child protection team (CPT) was useful in serving as a second opinion on cases and in helping to determine whether appropriate assessments were made: “[CPTs] help, because sometimes with us in just talking with the kids, we may not get all the information you need, whereas CPT, they’re more trained.” Another CPI pointed out that school contacts are helpful in understanding a child’s status, academically and developmentally. Some respondents had previous experience working with or raising children that they felt helped them understand children’s developmental stages and needs. One respondent also provided an example of learning to use the words and phrases children and clients are familiar with while asking questions in order to increase understanding. Some CPIs received informal training on assessments through division meetings, where coaching was provided on tasks like conducting home studies, completing safety plans, and carrying out removals. One area of assessment that CPIs felt warranted more attention was home studies, which require CPIs to assess for hazardous conditions and overall safety in the home. It was agreed that there was very limited information included in pre-service training on conducting home studies, and the content consisted mostly of explanations of the different types of home studies. One respondent said that colleagues were a key source of informal training on these assessments, as well as a field training investigator (FTIs) who created templates that could be used.

Safety Plan Implementation CPIs were asked about how pre-service training or other factors helped them develop and implement safety plans based on strengths-based assessment. At both intervals, CPIs recalled components of the training that addressed safety planning, suggesting that mock safety plans and individual feedback from trainers were especially helpful. However, at the 6-month point, there was significant feedback about the lack of preparedness CPIs felt in independently completing safety plans. As one CPI noted, “My first safety plan was on a piece of notebook paper. Because I was that unprepared.” Furthermore, CPIs suggested that the safety plans that were provided as examples in training were not realistic or representative of complex cases; for instance, when both parents have co-occurring disorders and a removal is required. In addition to these types of cases in which there were numerous risk factors for maltreatment, other respondents suggested it would be beneficial to do scenario-based training during pre-service on the types of cases that have less obvious signs of danger, such as behaviors or indicators in the home that take more insight to recognize. There were several other procedural issues with safety planning that CPIs identified as lacking in pre-service training, such as the type of language required in safety plans, (e.g., using “monitor” instead of “supervise”) and generally needing to be very specific and concrete in documenting supports or actions that will be taken. In addition, it was noted that the training did not help CPIs understand how to navigate situations in which parents may be resistant to signing a safety plan or when a plan may require numerous attempts and revisions, and it did not prepare them for understanding how much time they would spend “micromanaging” the safety plans once they were in place. Finally, respondents shared that the pre-service training did not prepare them to appropriately transfer cases for ongoing services with the local community-based care agency. Respondents from the 12-month focus group had less feedback in general about safety plans, and although they felt the importance of safety plans was conveyed during training, they had lingering concerns about effectiveness of learning safety planning during training because the scenarios were not “real” and less was at stake than interacting with real families. In terms of other supports that helped CPIs understand how to develop safety plans, respondents at both intervals said that they learned these skills “on-the-job” and from their supervisors and other agency support staff. While respondents generally spoke of the benefits of working closely with supervisors or other staff, they also highlighted ways that actual processes deviated from what they learned in pre-service training because many of these processes, like safety planning, depend on supervisor preferences. Furthermore, once they were in their units, they questioned the legitimacy of some of the guidelines they learned in pre-service or other early trainings, such as the 14-day timeframe that safety plans should remain in place. Many felt that this was far too little time for many cases, and they learned to adapt their explanation of the policy they tell to parents by saying that after 14 days, they will reassess their case and determine whether children can come home, knowing that cases can be active for up to 60 days. Twelve-month respondents emphasized the benefits of other trainings after they started on their units, as well as shadowing veteran workers. One CPI who was a self-described visual learner pointed out that “sitting at a table learning and taking notes” was much less impactful than directly observing processes. Communicating Investigation Information to Case Managers This domain refers to focus group discussions of how pre-service training helped CPIs clearly communicate the information collected during an investigation that was used to determine the need for case management. At the 6-month point, there were mixed responses on this topic. Some participants remembered practicing case staffing in the training and felt that it was helpful in learning about the needs of FLORIDA INSTITUTE FOR CHILD WELFARE

case management. Other CPIs said that pre-service training did not help them in their ability to assess the need for case management and that this was learned solely on the job. For instance, one respondent gave an example of having to go back to a house several times because they neglected to get pertinent information such as fingerprints or pictures. Several respondents shared that this was another aspect of training that is very different in practice. By the 12-month point, CPIs shared that the majority of what they learned about the case management process was learned after pre-service training. Some respondents recalled how the training provided information on what types of services would be needed for different types of cases, and in understanding the differences between judicial and non-judicial cases. One CPI said that discussions with a supervisor were helpful in determining present danger, and if present danger is determined, discussing the case at regular intervals with a supervisor is important. Others shared that it was “difficult to connect” ways that CPIs would need to use information about case management in their positions at the time they were in pre-service training. Several respondents agreed with the sentiment that “you can only do so much when you’re in preservice” for certain skills and components. Communication CPIs were asked how pre-service training helped them communicate with family members and collaterals in order to discuss their needs and concerns. Feedback was consistent at both 6 and 12 months. The topics most often referred to as valuable for communication skills included gathering information from collaterals, differentiating questions for different types of people, and asking open-ended questions that would result in better information (e.g., assessing mental health by prompting, “Tell me about a time in your life when you were happy”). Some CPIs that took part in the Avatar simulation training and had fictional clients who were verbally abusive and resistant, felt that it helped them prepare for “difficult people,” and helped them improve their patience, de-escalation skills, and their ability to relate to families. Role play gave CPIs a better understanding of how to address specific scenarios, like when people refuse contact, or when a case had a major unknown person involved in the investigation. Hearing about actual experiences and observing trainers in the field provided additional insight into the wide variety of people and circumstances CPIs may encounter on the job. Other feedback on this topic indicated that several additional factors influenced CPIs’ ability to communicate effectively with families, including previous work experience in human services, observations of experienced staff and support staff such as FTIs, and experience in the field. In one instance, a CPI’s supervisor had them shadow everyone in their unit at least once and then work with an FTI; from these experiences, the CPI recalled learning a great deal about communicating with different types of people. Several respondents said that certain personal qualities are helpful in building rapport with families, such as being naturally friendly and personable. For instance, one CPI shared that her role as a parent allowed her “find common ground” with parents and show empathy to those who may feel judged or embarrassed by investigations by acknowledging how hard parenting can be, especially for those with strenuous circumstances: “…especially if they’re a single parent. I say, you know, ‘This is tough for me and I’m married. So, I can understand as a single parent this must be really hard for you.” Some CPIs commented that their communication skills improved after practicing independently and by learning to modify their responses when interacting with individuals with challenging behaviors. Some suggested that greater experience in working with a variety of clients and on numerous cases leads to greater ability to know what to look for, what questions to ask, and how relationships may have an impact on what information is disclosed. The importance 16

of working closely with supervisors to ensure CPIs were gathering sufficient information was also discussed. Some respondents pointed out that the introductory nature of many topics covered in pre-service training was difficult to transfer to practice because every case and every person is different. CPIs spoke of the need to “think fast” about the best ways to interact with families and find their own way of asking questions, noting that a generic set of questions is ultimately not very useful. This point was echoed by training personnel during the training perspectives focus groups, who shared that critical thinking was crucial in being able to apply what trainees learn about how to handle one case to different types of scenarios. Interviewing Techniques and Strategies Focus group respondents were asked about how pre-service training helped them develop interviewing techniques and strategies to conduct age-appropriate interviews with children, caregivers, and families. Specific skills that were emphasized as helpful during the 6-month focus group include: demonstrating respect for clients, showing active listening skills, and using exploring and focusing skills. Similarly, 12-month respondents felt that the most poignant lessons they learned from pre-service training were how and why to ask open-ended questions, making sure not to lead children towards particular responses, and how to interview different age groups. There was wide agreement at both intervals that the practice-based lessons during pre-service were helpful, and that role plays, in particular, stood out as beneficial in understanding how to approach and interact with difficult people and with children, and how to modify body language and expressions. However, some feedback suggested that role plays are limited because they are done in a “controlled environment,” and the way trainers respond during the exercises may be very different from how clients respond in the field. As one CPI pointed out, “The level of stress, anxiety, and confidence is completely different when you’re out on your own in front of someone.” One respondent suggested trying to find a way to replicate the unpredictable nature of client responses, such as having outsiders come in to play the role of family members so that trainees are not as comfortable as they are with trainers, with whom they typically had a strong rapport. Videos were also highlighted as effective in portraying ways that clients might react to different investigation approaches, followed by discussions of how CPIs could handle those situations. Some respondents referred to general messages they learned in pre-service about how to interact with clients, such as to try to relate to families however they can, to be compassionate and “considerate of what’s going on in their lives,” and to be approachable but not become overly emotional during interviews so that they can remain focused on their goals of obtaining necessary information. Several CPIs also remembered guidelines for conducting interviews, such as getting down to children’s eye level when talking with them, sitting if clients sit or standing if they stand, and trying different strategies of engagement if clients are unresponsive. Other lessons respondents said they remembered were not “pushing” when a child is unresponsive and using play-based strategies to help the child feel more comfortable before asking questions. For the group that received Avatar training, there was mixed feedback at the 6-month point. Respondents generally suggested that the training had the potential to be helpful with practicing interviewing techniques and preparing trainees for interacting with different types of people, while some comments highlighted fewer effective aspects of the program. For instance, it was suggested that it was overall a very negative experience because there was only negative feedback offered and no indication of what CPIs were doing correctly. On the other hand, however, some CPIs felt the program was helpful in being able to practice their interviewing skills using FLORIDA INSTITUTE FOR CHILD WELFARE

different types of responses. Overall, there was wide agreement that pre-service training would benefit from more hands-on experiences that allowed more direct input, such as having trainers go out in the field with trainees to give them real-time feedback. Respondents at the twelve-month focus group commented on other factors that enabled them to successfully interview clients, such as being a parent or having other experiences interacting with children on a regular basis. Coworker support was also brought up as a significant factor, as strong connections with coworkers allow for frequent advice-sharing about how to handle different situations. Some CPIs said that ongoing training, such as monthly voluntary “refresher” workshops, sometimes covered interviewing skills. CPIs reiterated, though, that it was difficult to replicate the effectiveness of gaining experiences from on-the-job learning when it comes to interacting with families. Family Functioning Assessment CPIs were asked how pre-service training or other factors helped them interpret the results of the Department’s Family Functioning Assessment (FFA) to make appropriate child safety determinations or referrals. At the 6-month point, most of the feedback indicated that materials that covered FFAs were not sufficient in preparing CPIs to adequately complete them in practice. Several comments suggested that the training was limited to learning the domains of the FFA, which is not enough to prepare CPIs for actually completing the assessments in cases with a variety of circumstances. It was also widely agreed that the training provides examples “by the book,” but in practice, processes are not as neat. One CPI commented that the training did help with understanding various issues of functioning, such as developmental delays in children, but in practice, older children and adults often do not answer some of the questions on the FFA honestly, so it then becomes an ineffective tool. Other areas respondents saw as insufficient at 6 months were how to do referrals, what wording to use, how to enter information into FSFN, and how to navigate the FFAs for families with a large number of individuals. One respondent commented, “I got one household with 11 people. Five adults, six kids. I’m going to cry when I have to write that one.” It was acknowledged that providing effective training on FFAs is difficult because “everything is different for each family,” which means that each FFA is different and one cannot necessarily be used as a template for another. Furthermore, CPIs expressed frustration that supervisors expected CPIs to demonstrate skills in the field that they didn’t have the proper context for in pre-service training: “…when I ask for help [with the FFA] and they’re like, ‘You didn’t already know this?’ And I’m like, all right, well now I feel kind of stupid because how am I supposed to know about all of these things when it’s so complex and I’m training?” Another respondent had similar concerns: “…when you get on the floor, there are different, you know, supervisors that like things done differently. You learn one way in training, and then you get on the floor and they’re like, ‘Oh, no, it doesn’t go that way, it goes this way.’” With regard to wording, some respondents shared concerns that finding the right language to use can be “tricky” because it can lead to a different determination of safety than what CPIs may feel is accurate, and that different supervisors had different preferences for wording. There was also consensus among the groups that the training did not provide enough guidance on how to do referrals, as indicated by the following excerpt: “There’s so many referrals out there, but we never learned them in training…It’s just like, ‘Okay, refer it.’ But who do I refer to? How do I know who to refer to? What’s the appropriate referral?... The FFA doesn’t tell me that. And we didn’t learn that in training either.” This was further complicated by a lack of knowledge of agencies that provide services for “high risk” cases. Although several respondents recalled learning about different agencies during pre-service training, they noted that it was not realistic for them to retain or understand how to apply that knowledge without 17

having begun case work or to have known what the referral process would entail. During the 6-month focus groups, most CPIs discussed ways that they learned to do FFAs or referrals in the field or “on-the-spot” after very minimal examples or oversight. Other CPIs noted that they had more ongoing support from their supervisor or other support staff, with one highlighting that an FTI took the time to teach the FFA process thoroughly rather than just providing answers. At the twelve-month point, supervisor guidance was emphasized even more as a key factor in successfully completing FFAs, with regard to providing individualized feedback, giving specific instructions for completion, and determining if CPIs made the right decisions on the tool. Several respondents said that expectations for completing FFAs depended largely on supervisors’ preferences, which may differ between supervisors as well as from what trainees learned in pre-service training. Hands-on experience completing FFAs with a diverse array of families and cases was also described as helpful in becoming competent with FFAs. In terms of making referrals, CPIs at twelve months generally seemed more confident in this process and more aware of appropriate services than during the 6-month focus groups, some of which may be owed to the additional experience they had becoming familiar with provider agencies. One respondent suggested that coverage of the referral process during pre-service training was more limited for CPIs because it was geared more towards case managers. At the 12-month point, some of the feedback suggested that CPIs who were able to do more hands-on practice during pre-service felt that the training covered what they needed to know when they were in the field. One CPI spoke to the effectiveness of doing FFAs on mock cases, and another recalled watching a movie that featured child abuse and doing a practice FFA afterwards, with individual feedback provided by trainers. However, many respondents expressed that there was not sufficient time dedicated to a tool that was a central part of their daily practice, and that the lack of adequate focus given to learning to do FFAs contrasted with what several CPIs saw as “the whole focus of most investigations.” Feedback from both intervals highlighted CPI concerns with the utility of the FFA and its impact on other work processes. Some of the feedback about the FFA at the 6-month point indicated that there was a lack of understanding of why the assessment was important, or that it seemed redundant when other processes required the same information. In numerous instances, CPIs suggested that the FFA was not relevant to child safety, as elaborated by one respondent, “In my opinion, the FFA has nothing to do with the safety of the child because the FFA you can do basically two weeks after you decide whether the child is safe or not.” Some CPIs shared that the FFA seems like additional and unnecessary work: “I feel like we’re doing the same thing over and over, ‘cause I feel like when I gather my information, it’s right there in the PDA [present danger assessment] and then the same things that I’m putting in the PDA, I’m putting into the FFA. I’ve already come to that conclusion that the child is safe…” The perception of the FFA as unimportant was shared by other respondents, though in another example, the relevance of the FFA was questioned when a child is determined to be unsafe, since the CPI already has the information they need for court, making it unclear what value the FFA contributes. These examples suggest the possibility that a foundational understanding of the purpose of the FFA is not effectively conveyed during preservice training, or that the purpose of the FFA as it is communicated in training conflicts with the practical use of the assessment in the field. One respondent recommended unifying FFA training efforts across pre-service and other training and supports that occur after CPIs start working in their units so that workers, trainers, FTIs, and supervisors are all working from the same conceptual and procedural frameworks. Overall, focus group participants at 6 months described the FFA FLORIDA INSTITUTE FOR CHILD WELFARE

as a “sore spot” and voiced many concerns with expectations for completing them. For instance, several CPIs shared that the lengthy process of the FFA can make it difficult to close cases in a timely manner. In addition, several CPIs felt they were expected to use the results of the FFA to refer families to services before they had been able to fully assess whether there was an ongoing problem, as opposed to an isolated event. One respondent explained, “So, every maltreatment, they’re like, ‘Okay, did you make referral to here? Did you make referral for that?’ But then if you communicate, like, ‘I don’t really feel like it’s necessary,’ then afterward comes a whole bunch of ‘Why?’” Some CPIs felt this was contrary to how the process should work, which is that the family’s overall needs should drive referrals to services. During the 12-month focus groups, questions about whether preservice training adequately prepared CPIs to complete FFAs led to discussions of broader challenges with the tool. Some respondents explained that the interview sheet used for investigations was very extensive and includes all of the information needed to determine functioning, and therefore it is redundant to enter this information in another place as well. In addition, it was suggested that CPIs generally have an idea of what services they will refer families to based on pre-commencement information, such as prior offences, allegations, and criminal histories. CPIs in the focus groups widely perceived FFAs as mostly helpful in making safety determinations for unsafe children, but for safe children, they found them to be additional work that did not provide value, since safety determinations were made prior to completing the FFAs. Given the high caseload sizes CPIs carry, any components of the job that felt redundant or unnecessary were also seen as contributing of the problem of high stress. Some CPIs felt that for cases in which there were no safety concerns, the cases could otherwise be closed, but because the FFA still needs to be completed, they felt it added to an unnecessary workload. Florida’s Child Welfare Guiding Principles This domain refers to the extent to which respondents perceived pre-service training as helping them demonstrate knowledge and application of Florida’s Child Welfare System Guiding Principles. During the 6-month focus group, there was some confusion about exactly what the Guiding Principles referred to (e.g., methodology, practice model, policies), but the majority of respondents agreed that there was basic coverage of this material in training, in terms of understanding what warrants removals and how to go about those processes. For the most part, respondents said they generally had a basic understanding of the Guiding Principles but that they did not apply the Guiding Principles on a daily basis, (or if they did, they may be unaware of it). It was suggested that CPIs learn the “dos and don’ts” quickly on the job, and that they are familiar with requirements for interacting with parents and understanding procedural boundaries, but they may not be able to cite directly from the Guiding Principles. Some responses indicated that CPIs saw “policies and principles” as being more in the purview of managers and supervisors, while CPIs focused more on family safety. During the 12-month focus groups, there was little discussion of the extent to which Guiding Principles were incorporated into pre-service training or how knowledge of them influenced their work. Most CPIs simply said that they recalled learning about them, but that it is a daily process to incorporate them into work, something that is continuously evolving. Strengths-based Perspectives CPIs were asked how pre-service training helped them engage and assess families from a strengths-based perspective. At the 6-month point, some respondents were not clear about what “assessing families from a strengths-based perspective” meant, and one group 18

didn’t recall learning specific concepts or strategies described as “strengths-based” in their pre-service training. However, many CPIs seemed to have a firm grasp of this concept and described it as highlighting the strategies or supports that were already in place for families rather than just focusing on what parents are not doing right. In one group, the CPIs recalled a particular strategy that trainers taught that was memorable and helped them incorporate a strengths-based approach: “What we were taught to do is to look for the “blue” and not for the “red”…What that means when we get into the home, and it’s pure chaos, look for the strength of the family…” Other respondents shared a similar understanding of strengths-based concepts, suggesting that this framework helped them see the bigger picture outside of the limited report they receive on a person’s allegations, and helped them use families’ strengths as “glue” to help them function better and be more cohesive. Several CPIs gave examples of what they might say to families to demonstrate this approach, such as “Hey, you know, you’re talking to me. You’re willing to do these services, so that’s a good sign. You’re showing me that you want change.” They also acknowledged the difficulty on the family’s part of allowing a stranger to come into their house and make a judgment about them, noting that having a strengths-based framework also allowed them to approach their work as helping parents through a temporary setback so they can be in a better place to eventually regain custody of their children. At twelve months, respondents were better able to relate strengthsbased practice to what they learned from training. For the most part, respondents said that pre-service training helped them understand the importance of making families feel comfortable and showing them that they are there to help, despite many showing resistance to DCF presence. Furthermore, it was instilled in CPIs during training that treating families well will help investigations go smoothly. However, one respondent noticed that some experienced CPIs do not always uphold a strengths-based approach, and instead make assumptions about families prior to investigations, causing a discrepancy in training ideals versus practice ideals. Improvements At both intervals, CPIs were asked if they had any ideas or recommendations to improve pre-service training based on their experiences. Participants across all groups gave numerous suggestions that spanned issues from training structure to specific tools and practices. Training Structure Much of the feedback provided by CPIs related to the structure of pre-service training. One overarching recommendation at the 6and 12-month points was to provide more opportunities for handson experiences that would be more representative of what CPIs encounter in the field. Field days were a topic of frequent discussion at 6 months—there was wide agreement that there should be more field days in pre-service training so that CPIs could practice a multitude of skills, not only related to interacting with families but also in terms of learning data systems and assessment processes. Some respondents commented that pre-service training felt like trainees were “being taught to pass a test,” rather than learning a full range of concrete skills that they would need to know when they began field work. There were also recommendations to better prepare for field days so that CPIs could have meaningful learning experiences. One respondent described what felt like a particularly unproductive field day due to lack of preparation: “The supervisor at our agency, she forgot we were even coming. So, we sat there for three hours observing visitations. And then when she got there, she had nothing for us to do, so she gave us like a big court case and said, oh, you all can review this.” Other responses indicated that the timing of field experiences didn’t always allow for application FLORIDA INSTITUTE FOR CHILD WELFARE

of knowledge and skills learned in training, “…say, for instance, you learn something today, I feel like you should get the handson training like that same exact week.” It was suggested that field experience opportunities should be better coordinated with the actual material learned in training. Some respondents shared that they only had two field days, and many felt that having field days once per week throughout training would be more appropriate and allow for more opportunities to practice skills. There were several recommendations to incorporate more opportunities for feedback throughout training in order to confirm understanding of concepts and build confidence in applying knowledge and skills. It was frequently reported that CPIs felt they should have more knowledge and ability when they started field work than they actually did, and one CPI noted that this adds more responsibility to support or training staff in the units who then have to “teach more than field work.” Another respondent suggested that it would have been beneficial to review processes from the case example used throughout the training as a whole group, rather than just individually. This may help CPIs retain information and understand various elements of a case that need to be considered while trainees are still developing knowledge of processes. Several CPIs agreed that there was very little insight in pre-service training about the job functions of case managers, especially as they relate to interactions with CPIs, and because of this, they sometimes had difficulty understanding what case managers needed when cases were transferred. Respondents gave input on how this lack of understanding led to difficulty once they were in their units, which sometimes led to communication challenges or a lack of consistency in family engagement when cases were transferred. One suggestion for this was to rearrange the pre-service training structure so that specialty tracks come before CORE; therefore, better preparing CPIs to know what to look for during shadowing. This would also help prevent instructors telling either case managers or CPIs to only pay attention during certain sections of training (e.g., trainers would tell CPIs “this is for case managers only” during certain sections). Twelve-month feedback emphasized the need to have more opportunities to shadow experienced CPIs on different types of cases (e.g., child-to-child abuse, human trafficking) and having more “in field” training so that the in-class training is more applicable. Some respondents added that more observations and hands-on learning would help knowledge retention and passing the certification test: “…it would be, you know, in the field training so that way when we feel ready, we take the test, we would be able to master the test and then we could just go in the field and be the best that we can be.” In general, CPIs recommended better integrating the strategies, concepts, and methodologies taught in pre-service with what is practiced in the field. For regions that have Field Training Investigators (FTIs), it was recommended that pre-service trainers train FTIs to avoid divergent messages and practices, since that is who the CPIs will ultimately work with for on-the-job training. Some CPIs also recommended that trainers and supervisors go into the field more often to understand how practice is being carried out and how it relates to their training. Knowledge Retention During the 6-month focus groups, several respondents gave recommendations for improving processes for knowledge retention during and after pre-service. Key among these recommendations was to include more hands-on training activities, which may include participating in role plays, practicing assessments in class (with feedback), practicing using data systems, in-class practice making individual interpretations based on available information, and having more opportunities to observe interactions between experienced workers and families or to engage with clients. Scenario-based 19

exercises were also seen as helpful in retaining knowledge, though CPIs generally wished to see more varied types of scenarios that could be applied to different cases and different types of maltreatment. There were suggestions to include more visual materials and to make those that were included (like videos) more realistic and less “friendly.” One respondent who had previously participated in a different pre-service training recalled visual materials from that past training that had a lasting impact, such as watching videos of detectives’ interview children and parents on sexual abuse cases. It was generally agreed that, though the classroom portions of the training provided a necessary foundation for understanding core components of the work, there was a lack of engaging materials that would help with retaining information longterm. Assessments During the 6-month focus groups, suggestions for improving training around safety plans were to utilize more specific scenarios or templates for different circumstances. Because different types of maltreatment warrant different safety measures, it was suggested that multiple types of safety plans be taught, as described by one respondent: “I feel like they should teach us safety plans per maltreatment…Like, how does the safety plan look for household violence, how does it look for physical injury or environmental. You know? Because all safety plans don’t look the same and they only showed us how to do one.” This idea was echoed by others who suggested that safety plan training should be more comprehensive and intensive, and it should cover legal issues as well as provide more practice on the language that is required for the plans. One CPI also suggested that safety plan training should be given once CPIs are in their units so that they can better focus on applicability. Several respondents at 6 months suggested that FFA templates for different types of cases would help in better navigating the tool and making the process more efficient, as many reported spending a significant amount of time on completing the assessment. Some CPIs received or made their own templates with de-identified information and noted that the time it took to complete them after using the templates went from approximately two hours to 45 minutes. With regard to referrals, it was agreed by numerous respondents that trainees need to have a better understanding of what the referral process entails, as well as what types of agencies and services are available for referral (e.g., provider agencies, government departments, early learning coalitions). One CPI suggested that it would be helpful to have a document or other type of resource that outlines referral processes and agencies for different needs that is specific to each region or circuit. Incorporating information on how health insurance factors into clients’ ability to participate in services was also an improvement that CPIs recommended. Florida Save Families Network Issues related to FSFN garnered a lot of attention at both intervals. At 6 months, CPIs requested more training on FSFN during preservice so they could feel competent in basic use of the system when they began their field work. Several respondents discussed needing significant guidance on using the system while they were trying to manage their first cases, and many expressed frustrations with the system and the requirements for entering information on timelines that were perceived as unrealistic. One suggestion was to have a full week dedicated to FSFN that would include practice logging in, entering notes, submitting FFAs, and distinctions between types of notes (e.g., medical notes versus the file cabinet). At 12 months, several CPIs discussed ways that pre-service training could be enhanced by increasing time and attention to FSFN. FLORIDA INSTITUTE FOR CHILD WELFARE

Some ways they envisioned better integration of FSFN was by starting practice with it from the beginning and doing a mock case— completing each step of a case on a weekly basis throughout the training. CPIs said this was important because it is a tool that they use daily in practice, and there were many details that were not covered in pre-service training, leading many to feel lost or unsure how to enter case information when they started their position. This added to the challenges of learning new responsibilities as a new hire and for some, they could not always find someone who was available to help them when they needed it. Investigation Processes CPIs at the 6-month focus groups expressed a need for pre-service training to address situations where allegations are made that likely do not have a basis. Several respondents said they spend a lot of time investigating allegations that they feel are made out of spite and are generally unfounded (one CPI estimated that this was half their current caseload). This was seen as contributing to high caseloads because CPIs still have to “go through the motions” even though the cases often do not result in a need for further investigation. CPIs recommended having a better screening process from the hotline, as well as also guidance on how to approach these types of cases in pre-service training. In addition, respondents gave feedback at the 6-month point on improvements to training on legal and court processes, which many saw as lacking. One CPI gave an example of having to participate in a mock hearing during pre-service training in which a magistrate asked trainees questions about legal content of a case that they had not fully been trained on or that they felt was not relevant to their role (e.g., on case management). It was suggested that trainees would benefit from more training on legal processes like learning how to prepare a petition for removal and conduct home studies, beyond just a basic overview of these topics. Other processes that 6-month respondents wished they knew more about prior to starting work in the field included reading and interpreting criminal records, verifying allegations through outside sources, and navigating data systems such as the Florida Crime Information Center (FCIC), the National Crime Information Center (NCIC), and the Comprehensive Case Information System (CCIS), and Odyssey. There were numerous calls for including more content on how to gather information from other sources, such as calls to law enforcement in order to better understand whether previous disturbances correlate with current allegations, or from collaterals to help inform CPIs of information relevant to family assessments. One CPI commented, “…we need to know what we’re doing, what we’re looking for, and how to read [the information], you know...” This respondent suggested having a session in pre-service dedicated to reviewing a case using these different sources of information. Caseload Size, Turnover, and Retention Caseload size, timelines, and turnover were widely discussed at both the 6- and 12-month points, and recommendations related to pre-service training centered largely around conveying more realistic expectations of these issues in training. At 6 months, CPIs shared that they had unrealistically high caseloads, and it was suggested that one way pre-service training could help improve the management of these caseloads would be to include more training on time management and how to work more effectively within the timeframes required by the Department. In addition, many respondents commented that frequent turnover rates contributed to their high caseloads, and in fact, several respondents noted that many, if not most, of the other trainees from their cohorts were no longer in their units at 6 months due to turnover. Many CPIs felt they were being inundated with new cases before they would really 20

gained a strong foundational knowledge of all of the processes they needed to understand to effectively work cases. One CPI expressed frustration that they were being questioned by their supervisor about why their caseload was so high and why they were working overtime, when it felt like they were still learning and “playing catch up.” One CPI commented that training felt like “false advertising” compared to what the actual work felt like, and suggested that more accurate work expectations should be conveyed in pre-service training so that it may lead to a more stable workforce by helping to ensure that those who pursue the child welfare profession do so with full awareness of the responsibilities. Twelve-month responses on this topic centered largely around the need for taking measures to improve employee retention. One area of concern was staff morale; one respondent shared that the negative environment they experienced early on in their job was palpable and may influence workers to leave prematurely. It was also suggested that agencies take explicit steps to show workers they are valued, and to find ways to be supportive of new employees to ameliorate problems of turnover. One respondent pointed out that in some places, there are efforts to hire employees one month prior to pre-service training so that they have an introduction to the system and shadowing experience prior to entering training, which was perceived as helpful in providing knowledge about what to expect and what kinds of information employees need from the training. CPI Turnover Because employee turnover is a significant problem across child welfare systems, during the 12-month focus groups, the study team asked respondents about their reasons for staying in their position and what their perceptions were of why others left within the first year. Overall, CPI turnover was approximately 38 percent at the 12-month mark, so asking these questions allowed us to understand the issue from those who had received the same training and worked in the same environment for the same timeframe. Respondents provided a great deal of insight into the challenges new investigators face. CPIs who were still in their positions more than one year after training provided many reasons for staying in their jobs. Several respondents spoke to the personal benefits they feel from helping children and families, from “excitement and joy,” to genuinely liking the work, to satisfaction in making a difference, especially in instances where they felt that “a child was actually saved.” For one CPI, the work was particularly meaningful for personal reasons: “I’m passionate about this job. I came, myself, from an abusive home, so I could understand the child where, you know, what’s going on with them. I can relate to them.” Another common trend among respondents was that they felt their team and supervisors were supportive and responsive to their needs, for the most part, and they had gained a sense of camaraderie with their coworkers. Several respondents discussed how crucial it was, especially, to have a “good” supervisor. The primary reason for high turnover rates, as perceived by respondents, was that caseload sizes were too high, which resulted in overwhelming stress and anxiety and inability to “stay ahead” of caseloads. The constant flow of cases was seen by many CPIs as unrealistic to keep up with without jeopardizing some aspects of their job or their well-being. Several CPIs across different sites reported receiving an average of 2 cases per day at many points, which led to so much work that it interfered with CPIs’ ability to effectively engage with families. This pressure may lead to other problems that occur from trying to expedite work. For instance, being unable to give cases appropriate attention may lead clients to feel they are being treated poorly, which may exacerbate negative perceptions of CPI work. One respondent explained, “…you don’t FLORIDA INSTITUTE FOR CHILD WELFARE

really have the time to put into the families like you should. So, that’s why a lot of these families have negative views of the department, because it’s just like, ‘Oh, I’m just another case. Like, they don’t really care.’” In addition, the problem of high caseloads was seen as contributing directly to people quitting soon after starting their position, as they were caught in a seemingly inescapable cycle of stress and turnover. It was widely agreed that the extent of job-related stress was not fully acknowledged by superiors. One respondent shared an experience of asking for support in temporarily alleviating their caseload during an especially challenging time and being met with little understanding or effort in providing concrete help. This made clear the inherent contradiction in what CPIs were being asked to do for families versus what agencies were doing for CPIs: “I’m here, you know, getting up at three in the morning, picking up phone calls, making sure that I don’t miss a call or there’s a child that is in danger, but when I was in danger with my mental health, I didn’t get that support…” Other CPIs said they observed coworkers experience physical and mental health problems that were reportedly related to stress from their jobs. The intensity of caseloads led many CPIs to feel that they were not able to take time off, or if they did, the anxiety of knowing their work would build up often prevented them from doing so. Some CPIs expressed a lack of adequate support and guidance from supervisors or administrators, especially during intense periods. Not only was this seen as contributing to a poor work environment, but it also added to fears of making mistakes, which, as one respondent pointed out, could have severe consequences for the children they work with: “…when you have that much responsibility on your hands, it could be scary not having the support, not having the proper guidance, not knowing what to do. It could definitely be scary and challenging, and I can see why people would just rather quit and not deal with the stress…” Another related theme that emerged from these discussions was the challenges of balancing work with personal or family life. Several respondents had families, including young children, that they felt pulled away from because of work responsibilities. One respondent mentioned having it “drilled in” that to be successful in this job, you need to have a strong support system. Despite knowing this, however, it was still challenging for several CPIs to try to uphold the ideology of family life being most important when work duties were unending or could not be put on hold. Many respondents said they did have sufficient family and social support, but this did not counteract the negative effects of the job. Many CPIs shared the sentiment that, at times, they did not feel valued, knowing that another cohort was always being trained, and new workers would always be available. They felt unprotected in general, being inundated with cases without any concrete measures of relief. Many respondents suggested that supervisors and administrators should be more attuned to the ways that job stress impacts CPIs work and that there needed to be more efforts to provide support in order to motivate workers to stay in a challenging environment. Impact of COVID-19 Pandemic Because twelve-month focus groups were conducted shortly after the COVID-19 outbreak, while statewide shelter-in-place orders were still in place, the study team added a question to the focus group protocol to learn more about how CPIs’ jobs had changed as a result of service changes in the child welfare system. CPIs across all sites were working mostly from home, though some still went out on calls and occasionally to their office for printing or to supervise visitations or other essential duties. One of the main changes shared by respondents was a reduction to workload. Several CPIs said their caseloads were lower and more realistic, with one describing the new work environment as more “ideal.” Respondents said that this change was favorable not 21

only because there was less stress and pressure, but because it allowed them to do their work more thoroughly and provide better service to the families they worked with. Several CPIs agreed that the decrease in workload helped ensure they could give adequate attention to case details, reducing their concern about making mistakes. Some respondents said that working from home most of the time was more relaxing in some ways, in that it lessened the tension they often felt about being away from their children so much because of their work. However, some respondents did point out the added challenges of trying to “homeschool” or help their children with online school while also trying to work fulltime. One CPI noted that this added responsibility did not impacted their work performance overall, but there were discussions of needing to be even more flexible with schedules and work later hours to meet obligations. With regard to interactions with clients in the COVID-19 environment, one change brought up was that many clients do not want CPIs to enter their homes, for fear of potential virus contamination. Because CPIs need to assess home environments for safety concerns, many said that when they did walk-throughs, they may take less time than normal, and they may need to reassure clients that they are safe. One respondent discussed fears about potentially becoming infected, emphasizing that, “out there...it’s just so unknown.” It was also noted that CPIs were taking appropriate safety precautions such as wearing masks and gloves and frequently using hand sanitizer and obtaining information by phone when possible. Some other challenges involved the temporary closure of schools and other agencies. Because some service providers stopped working for a short period, CPIs had to figure out what other options were available for their clients. Also, because school personnel are the child welfare system’s “main reporters,” there were concerns about children who may be at risk by being home without that oversight. So even though the slower pace of cases was a welcomed change for many CPIs, the knowledge that it was potentially because of less abuse being reported weighed heavily on them. Summary Generally, respondents at the 6- and 12-month points spoke very positively about trainers and their competencies, often referring to them as “amazing” or as deserving of “high praise.” Many CPIs felt that pre-service training provided a strong foundation of knowledge of core principles, especially with regard to understanding and assessing different types of maltreatment. They acknowledged throughout the focus groups that they will encounter a wide range of families and scenarios in their jobs that it is nearly impossible to teach everything they need to know before starting actual investigations. Respondents were also relatively consistent with key areas they saw as needing improvement, which were to include more opportunities for hands-on learning, including more field days; to provide more practice with assessments and data systems that are required for daily use; and to make the training more comprehensive and realistic in addressing the wide variety of individuals and situations that CPIs can be expected to encounter in their work. CASE MANAGER AND CPI SELF-ASSESSMENT SURVEY RESULTS In order to assess case managers’ and CPIs’ perceptions of their competency and identify specific challenges and supports, a selfreport survey was distributed electronically to all participants at each of the three intervals—case managers and CPIs completed a self-assessment via electronic survey at baseline (approximately one month), 6 months, and 12 months after their unit start dates. The items on the self-assessment survey mirrored the observation checklists and were derived from the Florida Child Welfare FLORIDA INSTITUTE FOR CHILD WELFARE

Competencies, focusing on skills related to family engagement, risk and safety assessment, and identification of strengths and needs. The self-assessment included 13 items. Each item corresponds with a particular competency and one or more objectives from the CORE pre-service curriculum, with an additional section on the Family Functioning Assessment competency that included five Likert-scale questions, and one section on Transfer of Learning, which included five open-ended questions that assess supports, barriers, and other feedback on how knowledge and skills are applied after training (see Appendix I). At each interval, participants were given three weeks to complete the survey, with two one-week reminder emails. Case managers and CPIs were asked to reflect on their most recent family visit when answering questions, and to indicate whether a skill was demonstrated by marking “Yes” or “No,” or to mark “N/A” if there was not an opportunity. They were also asked to include an example for each of the skills demonstrated. 6-Month Self-Assessment Results At 6 months, the survey was sent to 62 participants in the study, and a total of 18 respondents completed the survey (nine case managers and nine CPIs), resulting in a completion rate of 29 percent. Results are reported in each section below and are often compared to results from baseline responses for context. Practice Competency: Case Managers Baseline responses indicated that case managers perceived that they were demonstrating the majority of skills listed during their most recent family visit (85% of the time), particularly with regard to respectful communication. Several respondents provided examples of ways they demonstrated these skills. For instance, when describing how they interacted in a non-confrontational manner, one case manager said, “When addressing the allegations with the mother, I remained mindful of my facial expressions, tone, and body language. I also allowed for her to explain her side before asking questions.” Another respondent noted ways they engaged with and assessed families from a strengths-based perspective: “I always ask the clients what do they enjoy doing and what are their strengths. If they struggle, I point out accomplishments such as keeping a job, maintaining a clean home, etc.” For instances in which case managers felt the skill could not be assessed, they marked “N/A,” which accounted for 10 percent of responses. The most common item for which respondents said they did not demonstrate the skill was on developing and implementing a case plan/safety plan based on a strengths-based assessment. There were no comments to explain the responses. The higher number of “Yes” responses overall compared to the observers’ marks is likely due to the greater knowledge case managers have of their own cases than what evaluation team members were able to observe. From the 6-month self-assessments, case managers indicated that they demonstrated appropriate competency skills and knowledge with more frequency at baseline (94% of the time compared to 85% of the time). One case manager referenced the importance of understanding cultural differences in disciplining children, and another explained their approach to demonstrating sensitivity to cultural differences: “I do not force my beliefs on someone and I do not let what I believe hinder me working with clients and how they act or what they believe.” One case manager discussed the way they establish non-confrontational communication: “[I] allowed [the] client to answer questions in a way that had them initiate conversation with regards to the maltreatment, and then was able to move forward from there. I was only able to do so after engaging the parents in a way that built trust and established respect.” Respondents discussed specific interviewing skills they used, such as active listening and using “10-step” interview questions for children. Case managers generally discussed how strengths-based 22

approaches to working with families was most successful when they collaborated closely with families to understand how they see their own strengths, and then built on these self-identified strengths through the case plan. Out of all six instances in which case managers marked “No,” one person accounted for four of these instances, and there were no comments to offer insight into why respondents did not demonstrate the skills or knowledge for these items. There were no instances in which respondents marked “N/A.” Transfer of Learning: Case Managers When asked what areas of pre-service training were most helpful in preparing them for working in the field, baseline respondents gave the following responses: techniques for interviewing and relating to different clients; discussions on policy and “real-life” cases; FSFN training; explanations of the process from FFI, to FFA, to case plan, to judicial review, to Termination of Parental Rights (TPR)/ Permanency; and shadowing and field observations. These responses highlight the value of hands-on components of preservice training, a finding that was consistent at both the one-month and six-month self-assessments. At 6 months, case managers said the training helped them to understand the “basics of the job,” the requirements for and process of reunification, the court process, the child welfare system in Florida, Avatar (for one site), and FSFN. Some respondents also commented that even though the training helped with basic understanding of these aspects of their work, this knowledge did not prepare them for what it was like “out in the field.” When asked about the least helpful aspects of pre-service training, baseline respondents indicated that more time should be spent on FSFN training (at least five days), car seat practice, and the specifics of case management. Some said that more training on specific forms or processes was needed, like what to include on the POS form or how to make referrals to specialists; and one suggested that trainees need more information on all the services offered throughout their county. At 6 months, case managers commented that the field days were not as helpful as they were intended to be because of difficulties accessing families during the designated timeframe for field days. They suggested that field days be planned in a way that will better assure a family will be available or that trainees act as “a secondary” on a current caseload to be able to ask questions and help work through the case. One case manager pointed out what they perceived as an imbalance of the topics addressed in the curriculum: “The least helpful [aspect] was the extent that maltreatments were covered and there was less time on FFA-Os, Case Planning and Safety Planning.” Case managers were asked about the types of resources and supports available to them to help them apply pre-service learning to everyday work. At baseline and 6-month assessments, supervisor and peer support were frequently mentioned, as well as support from other staff, such as management, mentors, team leaders, and career development coaches (CDCs). One respondent noted, “A CDC that I wasn’t assigned to help me to stop overthinking outcomes for a case plan. I was really stuck in a rut and she took the time to help me overcome that hurdle. I don’t have that problem anymore with outcomes.” To explain the importance of support by mentors and experienced workers, one respondent said, “Mentor[ing] by a current employee gave me someone who I could ask questions [to] and [who could] walk me through the various aspects of the job.” Other responses identified ongoing trainings and resource guides as additional supports during the first year of employment. In addition, one respondent mentioned that having

FLORIDA INSTITUTE FOR CHILD WELFARE

a reduced caseload and having training through case manager labs and academy days were also helpful, while another said that they learned the most by asking questions to superiors and other colleagues. When asked about barriers to practice as a new case manager, respondents at baseline and at 6 months said that having backto-back cases while still learning the job was very stressful, in addition to widespread challenges such as low salary, turnover, caseload size, records management, internal communication gaps, and lack of one-on-one supervision. One respondent described ongoing challenges with learning processes at baseline: “Not knowing exactly what to do, how to do it, or when to do it can be a barrier. Having multiple cases with different requirements can be confusing for someone just getting started.” At 6 months, case managers described feeling overwhelmed and having stress, and some pointed out how turnover leads to unrealistic work demands: “Getting cases that belonged to someone else prior, and they didn’t do a good job, and you’re left starting over if the person didn’t engage in services...” Other challenges reported were a lack of consistency and communication between systems such as DCF and the Courts, feeling that completing documentation is emphasized over quality case work, and making new referrals. One respondent felt there was a disconnect between how family advocacy is presented in training versus what the realities of the job allow case managers to do: “Training tells us that we have the right to advocate for the families but if the manager or supervisor has a different view, you must do what the Department is requiring or you may lose your job.” Other case managers also expressed concerns with disconnect between training and case work, such as the limited amount of training on judicial reviews and having better guidance on resolving issues when managers have competing methods for finishing tasks. Finally, case managers were asked what suggestions they had for improvements with pre-service training based on their experiences. Responses to this question centered around providing more hands-on training at all levels and protecting caseload sizes for new hires. As with responses from the one-month assessment, case managers felt that the field days needed to be better utilized during pre-service training. One respondent suggested that supervisors and experienced workers should attend pre-service training at fiveyear intervals: “I believe the training should be given to manger, supervisors, and case managers that have been employed for five years or more. This will allow them to become familiar with the current changes in processes and establish a better working relationship with new case managers.” Another case manager, who started the job with a caseload of more than 20 cases, noted that new hires should be given a smaller caseload, starting with five cases then increasing. One case manager expressed the point of view that it was not the interactions in the field that were challenging, but “All the things that [have] to be done in addition to being in the field.” Overall, the responses from case managers on the 6-month selfassessment indicate ongoing challenges with key areas in transfer of knowledge that were also reported at the one-month interval. These include: difficulty obtaining enough hands-on experience in pre-service training, a stressful work environment with unrealistic expectations for caseloads, and some discrepancies between expectations during pre-service training and actual practice, which may, in part, be due to different approaches to the work by superiors. With regard to supports in retaining knowledge and skills, case managers consistently pointed to the benefit and importance of internal support by supervisors and other colleagues and mentors.

Practice Competency: CPIs At baseline, CPI respondents indicated that they demonstrated most of the skills in the assessment during their last family visit with high frequency (75% of the time). Skill areas with the highest marks include utilizing interviewing techniques and strategies, identifying and facilitating referrals to appropriate services, and demonstrating knowledge and application of Florida’s Child Welfare System Guiding Principles. Some examples were provided for context. For demonstrating cultural competency, one respondent noted, “Being an African American CPI working with clients from a diverse population, I look at parenting differently from how I was raised than some of the clients I work with without judgment.” For communicating with family members to identify, discuss, and prioritize concerns and needs, one respondent did this by “asking parents what do they love about being a parent and what are their struggles as a parent,” and another by “asking the family what is it that I can do to better assist the family.” Respondents felt that the skill area was not applicable to their interactions 18 percent of the time. All but one of the “No” responses came from one individual, indicating that the majority of CPIs who took part in the survey are confident in their abilities to carry out necessary skills. At 6 months, CPI respondents indicated a higher frequency of demonstrating the skills in the assessment the majority of the time (94%) compared to baseline. CPIs gave brief examples of their interactions with families for several items. In terms of demonstrating sensitivity to cultural differences and ethnicity, respondents commented that this may include understanding how different forms of discipline are used, understanding various religious beliefs, and acknowledging the families CPIs interact with are from different places and have different cultures. Responses across several items suggested that CPIs saw respectful communication and “friendly,” family-centered approaches as important in their interactions with families. For instance, CPIs said they make sure to “listen to the family’s point of view,” ask families to tell their stories from their perspective, or they ask families to tell the CPI what they feel will help them function better. One CPI emphasized the importance of being “non-judgmental.” Overall, however, explanatory comments by CPIs were much more limited than those given by case managers. Transfer of Learning Protective Investigators were asked the same five open-ended questions as case managers to assess transfer of learning. The first question pertained to the most helpful aspects of pre-service training. At baseline, most responses indicated that pre-service training was helpful to the CPI’s job, particularly elements such as role play, the breakdown of Department policies, and creative ways of quizzing. However, one respondent disagreed, commenting, “I do not feel as if the pre-service helped me for this current position. Once on the floor, everything was different.” At 6 months, some CPIs generally commented that the CORE portion of pre-service training was helpful or that “everything in the class was helpful.” One respondent pointed to the interviewing component as beneficial, and four respondents said that the field days were the most helpful. In terms of the least-helpful aspects of pre-service training, baseline respondents agreed there should be less “book work” and more hands-on activities and field experiences. One CPI said that the time spent learning how to write FFAs and PDAs was ineffective because there was no feedback given on them, and because the processes were different once in the field. Six-month respondents did not provide feedback in this area. In terms of resources and supports for the job, at the baseline self-assessments CPIs indicated peer and supervisor support as helpful during the transition from trainee to independent field work,

in addition to receiving files with sample “how-to” forms to use as a guide. From the 6-month assessments, responses fell into multiple areas that suggested CPIs saw particular tasks as helping them retain training knowledge, including having assistance with referrals and home studies, interviewing people, completing the pre-commencement process and commencement case notes, and completing the FFA. One respondent emphasized the importance of having support from colleagues and supervisors: “The support of the office - my coworkers have been life-saving. I would’ve quit by now without the support of my supervisor and other coworkers.” Barriers discussed at baseline included being given large caseload sizes at a fast pace. One respondent commented, “Every case is different, some cases need more work than others. There is so much work to be done, not enough time to complete.” In addition to suggestions previously mentioned, CPI respondents said that improvements to pre-service training could be made by doing more interviewing practice, having more hands-on field experience, and communicating better with agencies on what new employees will be doing once they start. At 6 months, CPIs responded similarly to case managers, referencing high turnover rates, demanding caseloads, and burnout. One respondent elaborated on how case work expectations were perceived as unrealistic: “Sometimes I feel like I can’t get everything done in a reasonable amount of time. There are just so many aspects to the job and what’s required on each case.” For the final question, respondents were asked what improvements they recommended for pre-service training; CPI responses were mixed for this question. Five respondents reiterated that there needed to be more field days or that trainees needed more field experience during pre-service training. One of these respondents added that “Pre-service has provided good structure for the job,” but that that on-the-job training is necessary to have a more “complete” understanding of the work. One respondent suggested that there are no changes that need to be made to the pre-service training. Summary The small number of responses and examples by CPIs on the 6-month self-assessment are a limitation to having a more in-depth understanding of their perceptions of how skill competence and knowledge transfer change over time. However, a few themes that are consistent across both groups and both data points are that new hires desire more hands-on experience during pre-service training; caseloads are perceived as overly demanding during the first 6 months, and support from management and coworkers is seen as crucial to continuance in the field. 12-Month Self-Assessments At 12 months, the survey was sent to 50 participants in the study,a and a total of 10 respondents completed the survey (eight case managers and two CPIs), resulting in a completion rate of 20 percent. As an effort to avoid sending surveys during the most intensive period after COVID-19 services changes, surveys were sent in April and May 2020. However, it is possible that the transition to new work arrangements and re-prioritization of responsibilities may still have contributed to the low response rate, in addition to the study team not being aware of several workers who were no longer employed by study sites. Because of the very small sample size for this group, case manager and CPI responses were aggregated. Results are provided by domain below.

a It is likely that some of these participants were no longer employed at the time surveys were distributed, so the number of participants the surveys reached is likely lower. A formal count of remaining employees was conducted in early July 2020.

FLORIDA INSTITUTE FOR CHILD WELFARE

Practice Competency During their most recent family visits, respondents indicated that they demonstrated appropriate competency skills and knowledge 85 percent of the time. There were limited examples provided for responses. One example of demonstrating sensitivity to cultural differences was providing culturally responsive services to Caribbean families. With regard to utilizing interviewing techniques and strategies, one respondent said they “speak to younger children in a way they can understand.” For the item assessing strengthsbased perspective, one respondent said they explicitly asked parents about their strengths during the interview process. It is unclear why some items were marked “No,” (N/A was also an option if there was no opportunity), but it is possible that the action was not warranted for their particular visit or for the person’s position. Transfer of Learning When asked about the most helpful aspects of pre-service training, one respondent commented that particular field days were especially valuable: “The structured field days, like “legal day” and going to the courthouse to observe various types of hearings and going to the county jail. Also, another favorite field day was visiting the local service providers. That was very helpful to see what resources are available in the community.” Other 12-month respondents discussed several areas they felt were especially helpful from pre-service training, including role playing scenarios, going out “in the field,” obtaining necessary referrals, and submitting judicial reviews. When asked about the least helpful aspects of pre-service training, responses varied; one comment referred to the need for all field days to be structured, because when they were not, supervisors were not prepared to have trainees in the field and they lost a valuable opportunity for hands-on learning—one that respondents throughout this study emphasized as especially beneficial. A respondent who was in adoptions said that the FFA portions of the training were not directly helpful, as this is not a component of their position. And one respondent noted that all of the material was helpful, but also seemed rushed. In terms of supports and resources that helped new workers in their daily practice, several respondents commented that coworkers and supervisors were frequent sources of support, and that this was especially true given that “most of what you learn is ‘on the job’.” This is consistent with feedback from the 12-month focus groups, in which several respondents asserted that these supports in child welfare work were crucial for longevity. Other responses pointed towards ongoing trainings, particularly for newer workers, and concrete resources like a binder with information on service providers that could be used for referrals. There were mixed responses when asked about barriers to case managers’ and CPIs’ jobs. Some respondents answered that they did not perceive any barriers, while others said that caseload size and turnover were significant problems, as explained by this comment, “Caseloads and turnovers are the worst. When people leave, everyone that stays absorbs the rest of their cases.” Another comment by a CPI highlighted the problems of not being able to provide services to families in a very in-depth way. This was also brought up in the 12-month interviews with CPIs in direct relation to intense caseloads that did not allow for the time CPIs thought was necessary to work with families. Finally, respondents were asked what improvements they would make to pre-service training. There were few recommendations for improving pre-service training, but of those that were provided, comments centered around hands-on aspects of the training: “More time on the field and set appointments for those ‘field days’ so that they are not simply spent in the office. Additional role playing as FLORIDA INSTITUTE FOR CHILD WELFARE

well and with an equal amount of focus on interviewing younger children and teens as our class primarily focused on interviewing younger children.” Another respondent shared that there should be more time spent on learning and practicing FSFN during pre-service training. These suggestions are in alignment with feedback provided during the 12-month focus groups. Summary Overall, responses from the 12-month self-assessment indicate increased competency in completing FFAs over time, and a continuation of frequent skill and competency integration into practices. Though open-ended responses were limited, there were frequent references to the benefits of hands-on learning strategies (e.g., role plays, interviewing skills, field days) and the need to ensure that opportunities for field days are as structured as possible so that trainees can benefit from direct observation. It was also apparent that caseload size and high turnover rates continue to have a negative impact on workers who remain in their jobs. Finally, peers and supervisors were identified as key supports and helpful in applying skills and knowledge. FAMILY FUNCTIONING ASSESSMENT The Family Functioning Assessment (FFA) is a tool used by child welfare professionals to gather, analyze, and assess information to determine child safety in the household where alleged maltreatment has occurred. CPIs complete the initial FFA (FFA-I) and case managers are responsible for conducting ongoing FFAs (FFA-O). This section includes a review of FFAs that were completed by CPIs to make initial case decisions and by case managers to make ongoing case decisions. During this study period, Action for Child Protection (ACTION) was conducting quarterly statewide reviews of FFAs for the Department of Children and Families. Therefore, to avoid overlap, ACTION drew from a random sample of the study cohorts on USF’s behalf. The USF team submitted a list of trainees at both the 6- and 12-month mark of this evaluation. The 6-month analysis was included in the previous quarterly report; this final review includes analysis of the 12-month review of FFAs, a comparison over time, and summary of overall findings. In March 2020, the names of CPIs and case managers who remained from the study cohorts (n = 54) were provided to ACTION for a randomized review. Reviews of completed FFAs for individuals in the sample were conducted by ACTION in April/May 2020, at which point all workers had been in their positions for at least 12 months. ACTION reviewed 18 CPI FFA-Is and 8 case manager FFA-Os. Specific areas of review for CPIs and case managers are listed in Table 13 and Table 14 below. Child Protective Investigators (FFA-I) Present Danger (PD), an immediate, significant, and clearly observable danger threat, was identified in 8 of 18 CPI cases. The reviewer determined that 50 percent of the cases included sufficient information within the FFA-I to appropriately assess for PD. In six of the 18 cases (33%), the decision around PD was either inaccurate or there was not enough documentation to determine if appropriate decisions were made. In all of the cases where PD was identified, a safety plan was initiated; however, 75 percent of safety plans were not sufficient to control the danger threats that were identified. Of the cases in which PD was determined, only one provided sufficient information/documentation to support decision-making. There are six domains of information collected in the FFA-I. Documentation within these domains identify danger threats and inform whether a child is safe or unsafe in the home. Domains in which CPIs were most likely to gather and document enough 25

information were Extent of Maltreatment (59%) and Surrounding Circumstances of Maltreatment (41%). The remaining domains decreased significantly: Child Functioning (18%), Assessments of Adult Functioning (6%), General Parenting (12%), and Discipline or Behavior Management (6%). This suggests that documentation declined drastically in these areas. Information collected by CPIs on Caregiver Protective Capacities indicates that only 6 percent of cases had sufficient information. None of the cases reviewed documented enough information to support decision-making or determining impending danger. CPI supervisory consultation, guidance, and coaching are critical to ensure timely and accurate decision-making occurs. Initial case consultations occurred in 94 percent of the cases. These consultations are primarily used to review the initial information gathered during the Present Danger Assessment and Present Danger Safety Plan, and to guide the investigator in the collection of sufficient information in all six information domains. Precommencement consultations occurred in only 28 percent of the cases. These consultations are related to specific case practice issues and provide an ideal instructional opportunity for both assessing and developing worker competencies including, but not limited to, analyzing known information, guiding information collection, and planning initial investigative activities. Further review of case notes and supervisory notes by ACTION indicated that only 11 percent reflected that a supervisor provided ongoing coaching and mentoring to the CPI to ensure accurate and timely safety decisions were achieved. When comparing the 6-month and 12-month reviews of the FFA-I there was a significant deterioration of sufficient documentation to support decision making in the field, as indicated in Table 13 below. Table 13: FFA-I Areas of Review and Percentage of Cases with Sufficient Information by Interval FFA-I Areas of Review

6 Months

12 Months

Present Danger

73%

50%

Present Danger Decision Accuracy

73%

67%

Present Danger Safety Plan

25%

Extent of Maltreatment

82%

59%

Surrounding Circumstances of Maltreatment

91%

41%

Child Functioning

73%

18%

Adult Functioning

45%

General Parenting

55%

12%

Discipline or Behavior Management

55%

Caregiver Protective Capacities

45%

Impending Danger Decision Accuracy

45%

18%

Supervisory Initial Consultation

82%

94%

Ongoing Supervisory Consultation

27%

28%

During the 6-month review, sufficient information and documentation were highest when assessing for Danger (73%), Extent of Maltreatment (82%), Surrounding Circumstances of Maltreatment (91%), and Child Functioning (73%). In about half of the cases, CPIs adequately documented Adult Functioning (45%), General Parenting (55%), Discipline or Behavior Management (55%), Caregiver Protective Capacities (45%), Impending Danger (45%), and Safety Decisions (45%). Supervisory involvement is critical to the oversight, guidance, and mentoring of CPIs. During the 6-month review, initial consults occurred 82 percent of the time; however, ongoing supervision dropped significantly to 27 percent of the time. FLORIDA INSTITUTE FOR CHILD WELFARE

In the most recent review, at 12 months, findings showed a significant decline in accuracy of decision making and sufficiency of documentation in all areas. Most significant were in the areas of Extent of Maltreatment (59%), Surrounding Circumstances of Maltreatment (41%), Child Functioning (18%), Adult Functioning (6%), General Parenting (12%), Discipline or Behavior Management (6%), Caregiver Protective Capacities (6%), Impending Danger Decision (18%), and Adequate Safety Plans (25%). When reviewing supervisory oversight, similar to the 6-month review, documentation was seen in 94 percent of the cases for initial consultation and then fell significantly to 28 percent for ongoing supervisory oversight, coaching, and mentoring. Case Management (FFA-O) Data from the eight case managers that completed an FFA-O were reviewed. Case managers receive cases after the CPIs determined that a family is in need of ongoing services. Ongoing services are provided to intact families (in-home) as well as to families where the children were determined unsafe, and removal was necessary (out-of-home). Case managers continually assess families utilizing the FFA-O. The FFA-O is an ongoing assessment process that provides case managers with information that informs their actions throughout the case. The assessment is formally documented on a regular basis in the Florida Safe Families Network (FSFN), due to the agency’s official position regarding the current status of impending danger threats, child well-being, and safety analysis. Case notes should include the efforts of the case manager to gather the information included in the FFA-O. The FFA-O begins with engaging the family to help them transition from investigations to ongoing services. Engagement should provide families with opportunities to discuss their concerns, ask questions, receive answers, and establish rapport. Of the eight FFA-Os reviewed, none indicated that the case manager began the process by engaging the family. Overall, there were significant deficits noted on all domains and sections of the FFA-O reviewed. Twenty-five percent of cases in the Child Functioning domain reflect adequate documentation on the child’s general behavior, emotions, temperament, development, academic status, physical capacity, and health status. Adult Functioning (13%), General Parenting (13%), and Discipline/ Behavior Management (0%), which are all used to inform Caregiver Protective Capacities, lacked sufficient information. In all cases reviewed, none contained adequate case notes to support informed decision-making. Danger Statement (0%) and Family Change Strategy (13%) are critical elements needed to develop adequate safety planning and case plan goals and outcomes; however, very few case managers in the sample appear submit notes and documentation in practice. Documentation indicated that case plans were developed with the family in 13 percent of the cases reviewed, and only 25 percent of the FFA-O documentation supported the case plan outcomes. Twenty-five percent of FFA-Os indicated that Safety Plans were actively managed and that Conditions for Return statements were identified and supported. Case manager supervisors are required to have ongoing consultation with case managers. Supervisor consultations promote and develop case managers’ understanding of their responsibilities, skills, knowledge, attitudes, and adherence to ethical, legal, and regulatory standards in the practice of child welfare services. Guidance and oversight of FFA-Os, and ultimately approval, is the supervisor’s responsibility. Case managers’ regular consultation was evidenced in 13 percent of the cases reviewed. Similar to the FFA-Is, the differences in the review of FFA-Os from the 6- to 12-month review was significant, as shown in Table 14.

Table 14: FFA-Os Areas of Review and Percentage of Cases with Sufficient Information by Interval FFA-0 Areas of Review

6 Months

12 Months

FFA-O Family Engagement

29%

Child Functioning

53%

25%

Adult Functioning

24%

13%

General Parenting

18%

Discipline or Behavior Management

12%

Case Notes Support Decision Making

15 to show differences in self-perception of FFA ability compared to formal assessments by ACTION. Table 15: Mean scores at each interval for Family Functioning Assessment section Item

1-Month

6-Month

Mean

12-Month Mean

13%

Score

(n = 21)

(n = 18)

(n = 10)

I understand all of the components I am required to complete in the Family Functioning Assessment.

3.24

3.39 (+)

3.7 (+)

I can provide sufficient descriptive information when completing the Family Functioning Assessment.

3.33

3.44 (+)

3.8 (+)

I am confident in my ability to rate the functioning level of children.

3.10

3.10 (+)

3.8 (+)

Caregiver Protective Capacities

18%

13%

Child’s Needs Assessment

53%

25%

Danger Statement

12%

Family Change Strategy

24%

13%

Case Plan Developed with Family

24%

13%

FFA-O Supports Case Plan Outcomes

12%

25%

Supervisory Case Consultation Prior to Approving Case Plan

35%

13%

I am confident in my ability to rate the caregiving capacity of adults.

3.10

3.33 (+)

3.7 (+)

Safety Plan Actively Managed

65%

25%

2.81

2.67 (-)

3.5 (+)

Conditions for Return

10%

25%

I have enough time to fully complete the Family Functioning Assessment.

Regular Supervisory Case Consultation

38%

13%

FFA-Os at the 6-month review lacked documentation and evidence of sufficient information collection needed to make informed decisions in most areas and domains. Some of the most significant deficits were seen in Family Engagement (29%), Adult Functioning (24%), General Parenting (18%), and Parenting Discipline/ Behavior Management (12%), with the review of the case files at 12 months showing that none of the cases reviewed contained sufficient documentation that would support decision-making. The supervisor’s role is to provide ongoing oversight, coaching, and mentoring. However, only 35 percent of cases documented supervisory consultation before case plan approval and 38 percent had evidence of regular consulting occurring between supervisor and case manager. At the 12-month point, further decline in sufficient information and documentation was noted. None of the eight files reviewed reflected that family engagement occurred at the time the case manager began the FFA-O. Significant deficits in information collection and documentation was also noted in Child Functioning (25%), Adult Functioning (13%), General Parenting (13%), and Parenting Discipline/Behavior Management (0%), with the review of the total case file showing that none of the cases reviewed contained sufficient documentation that would support decision-making. Supervisory involvement also declined, with both consulting prior to case plan approval and regular consulting both falling to 13 percent. CPI and Case Manager Self-Assessment of FFAs As part of the self-assessment sent to case managers and CPIs at baseline (one month), 6 months, and 12 months, respondents were asked to complete a section focused on the FFA. Respondents were asked to rate their level of ability to understand and complete the FFA based on a four-point Likert scale (where 4 = Strongly Agree and 1 = Strongly Disagree). Although the survey was sent to the same cohorts at each interval, only a small number of the respondents completed both surveys (n = 10), while most completed only one or the other or did not participate, and therefore a formal test of significance could not be completed due to inability to appropriately match enough responses at both points. However, a summary of mean scores across each interval is provided in Table

FLORIDA INSTITUTE FOR CHILD WELFARE

At each assessment period, the majority of respondents agreed or strongly agreed with each item, demonstrating a high level of self-reported competence with FFA completion. At the 12-month interval, mean scores show an increase compared to both 1-month and 6-month scores. Though only incremental and based on small sample sizes, these consistently higher scores across intervals indicate a steady increase in self-perceived competency over time during the first 12 months in their position. Comparison and Summary Assessment of FFA-Is and FFA-Os occurred around the 6- and 12-month timeframe for both the self-assessment and the external review. Due to staff turnover, the sample size of the external review significantly decreased from 77 at the 6-month review to 54 at the 12-month review. The results of both reviews revealed an overall lack of understanding of “how to” accurately assess and document decision-making regarding child safety and risk. At all self-assessment points, respondents reported a high level of competency in completing the FFAs, which contrasts greatly with findings from the formal assessment by ACTION. At both the 6- and 12-month points, where CPIs and case managers indicated increased competency, ACTION’s review of cases indicated decreased competency. It is important to note, however, that because of the random selection process by ACTION, it is not possible to know whether individuals who responded to the selfassessment survey were also the ones whose cases were reviewed. Also, the two assessments used different areas of analysis and different methodologies, and they cannot be directly compared. Furthermore, the small sample size of respondents, particularly at the 12-month point, makes it difficult to draw conclusions about the data. However, one insight we may draw is that the general patterns of self-perceived FFA competency and competency determined by the external review differed among remaining workers in the study. According to ACTION’s analysis, information collection and documentation within FFA-Is and FFA-Os, as well as supporting documentation within the official record (FSFN), was not sufficient to inform decision-making. Equally as concerning is the lack of supervisory involvement. Although pre-service training appears to teach, within a classroom setting, the purpose of the FFA, 27

assessment, engagement, and information collection and documentation, these skills and knowledge do not appear to translate to the field. Further supporting this assessment were the comments from CPIs and case managers who participated in focus groups and indicated they were not adequately prepared to complete FFAs once in the field.

respondents felt was least important was level of responsibility (2.83), in comparison to the 3-month group respondents, who felt that intellectual challenge (3.23) was lowest in importance. Figure 1: Importance of Job Characteristics at 3- and 12-Months Post Training

ORGANIZATIONAL CLIMATE AND WORKPLACE CULTURE THREE- AND 12-MONTH ASSESSMENT RESULTS One of the primary goals of the pre-service training evaluation is to identify environmental factors and individual coping strategies of workers that facilitate and hinder knowledge acquisition and skill development while in the roles of case managers and child protective investigators. The organizational climate and workplace culture analysis is intended to examine perceptions of organizational culture and climate among case managers and CPIs, as well as the job turnover intentions of case managers and CPIs. A survey developed by Glisson and colleagues was adapted for use in this project in order to assess the climate and culture of each localized child welfare system (See Appendix J).1,2,3 Specifically, this study adapted subscales of the Children’s Services Survey which measure organizational climate and culture. Domains pertaining to organizational culture include, for example, role conflict, role overload, depersonalization, and emotional exhaustion. Domains related to organizational climate include rigidity, proficiency, engagement, and functionality. Domain labels were omitted from the survey so that the negative wording (e.g., work overload and rigidity) did not impact participants responses. These scales have sound psychometric properties, and a number of studies in varied workplace environments found the scales to be reliable. The Organizational Climate and Workplace Culture Assessment was administered to cohorts of case managers and CPIs who completed pre-service training and had been in the field for 3 months and for 12 months. The survey was sent to each participant via Qualtrics, a web-based survey platform. This report includes comparisons and trend seen across 3- and 12-month data collection points. In addition, although the survey was sent to participants at all sites, a somewhat small sample size (n = 7) occurred due to worker turnover by 12 months (68% of newly trained case manager and CPIs combined were reported to have left their positions by the 12-month mark) and likely the heavy demands on the remaining workers. While repeated attempts were made to send reminders to complete the survey, workers may have generally prioritized their case work before a survey, as well as any additional burden of the COVID-19 pandemic that occurred during this time. While comparisons were made to show trends between the 3- and 12-month data points, tests of significance were not done due to the small sample sizes.

Next, respondents were asked to rate how satisfied they were with the above characteristics specific to their current job. In Figure 2, satisfaction levels at 3 and 12 months are compared. At 3 months, survey respondents were most satisfied with the contribution their work was making to society (3.38) and the degree of independence they had in their daily work (3.31), followed by the intellectual challenge of the position (3.15) and the level of responsibility they had been given (3.08). Three-month respondents were somewhat satisfied with their level of job security (3.00), while they were somewhat dissatisfied with benefits (2.85) and organizational climate and supports (2.85). Figure 2: Satisfaction with Job Characteristics at 3 and 12 Months

Importance of Job Characteristics and Job Satisfaction Respondents were asked, when thinking about a job, how important several factors were to them. Figure 1 shows 3-month post training results compared to 12-month post training results. On a scale of one to four points, the most important job factors listed at 12 months were salary (4.0) and job security (4.0) followed by benefits (3.83), in comparison to the most important factor at three months being organizational climate and support (3.92), followed by benefits (3.84) and salary (3.76). Opportunities for advancement and level of responsibility were not as important at 12 months (3.33 and 2.83 respectively) as they were at three months (3.76 and 3.46 respectively). Contribution to society and degree of independence were rated similarly at 12 months (3.55 and 3.33 respectively) and 3 months (3.61 and 3.46 respectively). At 12 months, the job characteristic FLORIDA INSTITUTE FOR CHILD WELFARE

Three-month respondents were least satisfied with opportunities for advancement (2.77) and salary (2.23). In comparison, 12-month respondents were most satisfied with the intellectual challenge (3.83) of their day-to-day job, more so than they were at three months (3.15). Respondents were highly satisfied with the 28

contribution their work made to society, results indicating that those still employed at 12 months felt slightly more satisfied (3.5) than the cohort at 3 months (3.38). From 3 to 12 months, additional gains in satisfaction were also seen in degree of independence (3.31 to 3.5), job security (3.0 to 3.33 respectively), level of responsibility (3.08 to 3.17 respectively), opportunities for advancement (2.77 to 3.17 respectively), and organizational climate (2.85 to 3.17 respectively). Level of satisfaction decreased slightly for benefits (2.85 to 2.67 respectively) and salary (2.23 to 2.16 respectively) between the two periods. Survey respondents were also asked to provide an overall job satisfaction rating at 3 and 12 months. Three-month respondents rated their overall job satisfaction at an average score of 2.61, while 12-month respondents rated it at 3.0. Turnover Intentions Respondents were asked how likely it is that they will leave their job to pursue another position within the field of child welfare in the next year. While at the 3-month mark the majority of respondents (85%) believed it was somewhat to very likely they would leave their jobs, at the 12-month mark 67 percent believed it was only somewhat likely. No 12-month respondents felt that leaving their job was very likely. Respondents were also asked how likely it was they would leave their current jobs to accept work in a field other than child welfare. At 3 months, 77 percent of respondents believed it was somewhat to very likely they would leave child welfare to pursue another vocation. In comparison, at 12 months, 83 percent of respondents felt it was somewhat to very likely they would leave child welfare to pursue another vocation. Although these findings should be interpreted within the context of somewhat small sample sizes, they do show that most individuals in the position of a case manager or CPI desired to leave the field of child welfare. This may be reflective of the challenges frontline workers face and/or the perception that frontline work is not a long-term job. It is also important to note that a limitation of the survey sample is that those who already left their positions were not contacted to complete the survey. Organizational Culture and Workplace Climate Characteristics The specific survey domains we used to assess organizational culture and workplace climate were rigidity, proficiency, resistance, stress, engagement functionality and morale. This section explains each domain and includes any trends that were apparent from the data. Job Flexibility Rigidity Job flexibility rigidity refers to things such as how much flexibility and discretion an employee feels in their work, whether or not they feel they have input in the decision-making processes in their work environment, and the extent to which rules and bureaucratic processes govern an organization versus being able to tailor case decisions to individual family need. This last trait is often talked about in terms of centralization versus formalization within a work environment. In a highly centralized environment, decision making authority would be concentrated in the hands of a few people versus diffused authority at different levels of an organization. Formalization has to do with how many policies and procedures govern employee behavior. First, at both the 3-month and 12-month marks, the majority of respondents (79% and 71% respectively) agreed that they had the flexibility to make case decisions independently, but the intensity of that agreement varied (Somewhat Agree 36%; Agree 36%; and Strongly Agree 7% for the 3-month group, and 14% Somewhat Agree and 57% Agree for the 12-month group). When asked if they felt they could make case-specific decisions or if all cases had to be handled in the same way, there was an increasingly FLORIDA INSTITUTE FOR CHILD WELFARE

positive trend toward making case-based decisions that matched family need represented by a somewhat to strong agreement rating by 71 percent of 3-month respondents and 86 percent of 12-month respondents. When asked if staff felt their input was solicited and valued by leadership, 3-month results were split down the middle with 43 percent agreeing their input was valued, 43 percent feeling their input was not valued, and 14 percent not yet sure how they felt. At the 12-month mark, 57 percent felt their input was valued by leadership and 43 percent were not sure how they felt. Proficiency The second domain included in the organizational culture and workplace climate survey is proficiency. Proficiency within an organization encourages norms and behaviors that place families and quality of care first above timelines and numbers served. In child serving organizations it is often associated with higher levels of professionalism, being responsive to clients, implementation of EBPs, maintenance of updated treatment models, and improved outcomes for youth served. First, all respondents agreed that they were encouraged to be responsive to the unique needs of their clients (36% Strongly Agree; 50% Agree; 14% Somewhat Agree at 3 months and 71% Agree; 29% Somewhat Agree at 12 months). Respondent agreement also continued to be strong when asked if they were expected to apply up-to-date knowledge of EBPs and best practices (36% Strongly Agreed; 57% Agreed; 7% Somewhat Agreed at 3 months and 14% Strongly Agreed; 43% Agreed; 14% Somewhat Agreed at 12 months). Respondents were also asked whether they felt that emphasis was placed on quality of the services they provided versus meeting certain quotas or timelines. Respondents at 3 months were split on this issue: 43 percent agreed that there was a focus on quality, while 43 percent disagreed that quality was valued over quotas, and 14 percent were not sure how they felt yet. At 12 months, respondents showed more neutrality than disagreement on whether there was an emphasis on quality over quantity; in the 12-month group, 56 percent of respondents agreed to some extent that quality was prioritized, while 43 percent at 12 months were not sure whether or not creativity was valued. It is possible that those who strongly felt that quality was not valued over quotas already left their positions by the 12-month mark. Finally, respondents were also asked about whether they were encouraged to be creative in how to best serve families. Results were again mixed at 3 months; 64 percent agreed, 21 percent disagreed, and 14 percent were not sure yet whether or not creativity in service provision was encouraged. However, 100 percent of the 12-month group felt that their creativity in service provision was encouraged. Resistance The third domain included in the organizational culture and workplace climate survey is resistance. Organizations that score higher on resistance are those that are not as open to changing their service delivery or case practice models. Resistant organizations often encourage apathy or passivity toward change and suppress behaviors that introduce new practices and innovative service approaches. Behaviors might range from a simple inactivity toward change to direct opposition and criticism of individuals promoting change. Specific to the survey, respondents at 3 months were mixed regarding whether or not new ideas and innovation were supported by administrators (50% Agreed; 43% Disagreed; 7% Neutral). However, a positive trend was seen over time as 86 percent of the 12-month group felt that new ideas were supported, while 14 percent disagreed. Results were also mixed regarding whether staff felt that they were expected not to make waves within their organization, although a positive trend was seen over time in that more experienced workers felt more comfortable voicing opinions (28% Agreed; 21% Disagreed; 51% Neutral at 3 months 29

and 14% Agreed; 57% Disagreed; 29% were Neutral at 12 months). Another positive trend was seen regarding respondents’ feelings about whether their organization was quick to adopt new EBPs and promising practices (35% Agreed; 35% Disagreed; 29% Neutral at 3 months compared to 71% Agreed; 14% Disagreed; 14% Neutral at 12 months). Finally, a positive trend was also seen over time regarding whether or not organizations were perceived not to be addressing service barriers and developing new ways of providing services (42% Agreed; 28% Disagreed; 29% were Neutral at 3 months compared to 28% Agreed; 43% Disagreed; 28% Neutral at 12 months).

Functionality

The fourth domain included in the organizational culture and workplace climate survey is stress. A stressful work climate is one in which workers feel emotional stress and burnout either due to role conflict, competing demands, insufficient resources, and/or not enough time to get their assigned work done. Continuing with the positive trend, while 79 percent of respondents at 3 months agreed that paperwork and other bureaucratic concerns superseded client interests (the remainder of this response was 21% Neutral), only 28 percent of 12-month respondents felt this way (the remaining were 57% Disagree and 14% Neutral). A clear finding at the 3-month mark was that the majority of respondents (86%) felt emotionally drained and burned out without enough time to do their work, with 7 percent neutral and 7 percent not reporting such stress and time shortages. This was also generally true of the 12-month group, however only 71 percent felt this way and the remaining 28 percent did not feel drained. Likewise, while the majority (64%) of 3-month respondents felt that their workload and job expectations were not reasonable (the remainder of this response was 29% Neutral and 7% Disagree), at 12-months this group dropped to 28 percent, with 43 percent feeling that their workload and job expectations were indeed reasonable, and 28 percent felt neutral. Finally, this pattern of a positive trend did not hold true for respondent perception of adequate resources and services to meet their clients’ needs. At both 3 and 12 months, the data remained the same–57 percent agreed that there were adequate resources, while 28.5 percent disagreed, and 14 percent responded as neutral.

The sixth domain included in the organizational culture and workplace climate survey is functionality. Functional work climates are those where employees feel they understand the vision, mission, and goals of an organization; understand their place and roles within the organization; and feel a sense of cooperation from coworkers and administrators in supporting their organization’s work toward such goals. Job performance standards are clear, appropriate feedback is given to employees, and opportunities for personal growth and development exist and are clearly presented to staff. First, the majority of both 3- and 12-month respondents agreed to some extent that their organization provided advancement opportunities if they worked toward them (71% and 86% respectively). In addition, the majority of 3- and 12-month respondents felt that they had a clear understanding of their work responsibilities (79% and 86% respectively). In terms of how well staff perceived that coworkers functioned together to serve families, a clear positive trend was seen in that the majority of 3-month respondents (79%) and all 12-month respondents agreed to some extent that their coworkers collaborated to improve the effectiveness of services delivered to families. A positive trend across data points was seen when staff were asked if this type of positive performance was recognized by their organization and rewarded. Half of the 3-month respondents felt that due recognition was given, 29 percent did not, and 21 percent were neutral. In comparison, 71 percent of the 12-month respondents felt that their organization rewarded positive performance, 14 percent did not, and 14 percent were neutral. Results indicated that at 3 months, most staff (79%) reported having a clear understanding of their organization’s mission, vision, and strategic plan; and at 12 months, all respondents had a clear understanding. Results were very mixed at the 3-month mark regarding whether or not the work atmosphere encouraged open communication. Fifty-seven percent of respondents agreed that it did, and 43 percent indicated that they had not encountered a work environment where open communication was encouraged. However, at 12-months 86 percent of respondents felt that open communication was actively encouraged. The portion of respondents who felt they were given accurate and timely feedback also increased between the 3-month and 12-month group (71% to 86% respectively).

Child and Family Engagement

Staff Morale

The fifth domain included in the organizational culture and workplace climate survey is engagement. Engaged environments contain workers who perceive themselves to be doing something worthwhile and feel a personal connection to the work they do with families. Engagement within an organizational climate typically includes personalization, the extent to which staff feels involved with families, personal accomplishment, and the level of success staff feel they have in their work with families. At 3 months, there was a moderate trend across the engagement domain that respondents felt connected to the families they worked with, and they perceived themselves as having success with serving children and families. At 12 months, ratings generally increased in a positive direction. For example, at 3 months, only 57 percent of respondents felt they were making a difference in contrast to 86 percent of the 12-month group. At both 3- and 12-month data collection points, all respondents agreed that they care about their clients and treat them with respect. However, while all 3-month respondents agreed to some extent that they worked with clients to address their families’ needs, 28 percent of the 12-month group felt neutral (with 71% still agreeing). Finally, the majority of respondents agreed to some extent that their work offered them opportunities to make a difference in the lives of children and families (93% Agreed; 7% Disagreed in the 3-month group and 86% Agreed; 14% Neutral in the 12-month group).

The seventh domain included in the organizational culture and workplace climate survey is morale. Morale relates to employees’ attitudes about their job and career path, their level of job satisfaction, and their level of commitment to an organization and its mission. Participants were asked how they felt about their salary; at 3 months, results were mixed regarding whether respondents felt that they were compensated fairly. Half of respondents did not feel like they were compensated fairly, 43 percent did, and 7 percent were neutral. At the 12-month point, a different trend emerged in that only 14 percent of respondents felt that they were compensated fairly. In a clear positive trend, 79 percent of 3-month respondents and 86 percent of 12-month respondents liked their job and the agency they worked for. However, 57 percent of both 3-month and 12-month respondents indicated they often thought about quitting their job. The question was also posed to respondents, if they were offered a different job that paid the same salary, would they leave? At the three-month mark 86 percent believed they would. At the 12-month mark this dropped to 56 percent.

Job Stress

FLORIDA INSTITUTE FOR CHILD WELFARE

Summary The Organizational Climate and Workplace Culture Assessment was administered to cohorts of case managers and child protective investigators who completed pre-service training and had been in the 30

field for 3 months and for 12 months in order to examine perceptions of organizational culture and climate among case managers and CPIs, as well as job turnover intentions. It was commonplace at the 3-month mark to view a new child welfare job as only temporary until something better came along, even with little aspiration for promotion within the field. Eighty-five percent of respondents planned to leave their positions, and 77 percent of respondents were looking to leave child welfare all together. However, the 12-month data provides additional insight. For example, at the 12-month point, only 56 percent of respondents believed they would leave child welfare if the opportunity arose, and for those staff remaining in their positions for 12 months or longer, they were more satisfied overall with their work than the 3-month respondents. Specific job areas that saw the highest gains in satisfaction as time went on included: satisfaction with intellectual challenge, organizational climate, and opportunities for advancement. However, just as important to note, satisfaction over time regarding salary and benefits dropped. Gains across time were seen in the domain areas of flexibility, proficiency, and resistance. Twelve-month respondents were overall more likely to feel they had flexibility in making case-based decisions (rather than employing a cookie cutter approach to case work), their input was more valued by leadership, their agency emphasized quality of service provision over meeting quotas, creativity in service provision was encouraged, and that new ideas were supported. Additional areas that showed gains over time were that workers felt more comfortable voicing disparate opinions, they felt their agencies were more willing to adopt new EBPs, and they perceived their agencies as better addressing service delivery barriers. Additional gains across time were seen in the domain areas of job stress, client engagement, functionality, and staff morale. Specifically, 12-month workers were more likely to feel their job expectations were reasonable, and less likely to feel that paperwork and bureaucratic concerns superseded client interests. The 12-month group was also more likely to feel that they were making a difference with clients, had a clearer understanding of their work responsibilities, felt that coworkers functioned well together to serve families, that positive staff performances were rewarded by their employer, and that open communication was valued. While emotional strain and burnout was lower in the 12-month group, possibly indicative of a more resilient respondent base, it was still high, with 71 percent of 12-month respondents feeling emotionally strained and overtaxed. In addition, the positive trend seen across time did not hold true for respondent perception of adequate resources and services to meet their clients’ needs. TRAINING PERSPECTIVES FOCUS GROUPS After 12-month activities were completed, the study team conducted focus groups with training personnel who took part in initial training initiatives interviews. Two sessions were convened across all sites, and one or more representative from each of the nine sites participated in the focus groups, with seven participants in the first group and six in the second. The intent of the focus groups was to provide a brief summary of some of the key findings from the study to training managers and to garner their feedback on some of the strengths, challenges, and opportunities for improvement for pre-service training that were identified by participants throughout the study. Findings were summarized into six key domains and presented to the participants during the focus group. The evaluation team then elicited feedback from the groups in response to each finding. The domain related to FFAs was added during the second focus group after recognizing the importance of the results in that area.

FLORIDA INSTITUTE FOR CHILD WELFARE

Finding 1: Training Content The study team shared the finding that case managers and CPIs said that some of the content from pre-service training that was the most beneficial to them was knowledge of child development and age-appropriate assessment, interviewing skills, and role-playing scenarios to help with communication skills. Conversely, many felt that more time should be dedicated to practicing FFAs, FSFN (CPIs), and safety planning (case managers) during pre-service training. Focus group participants were asked how pre-service training can be adapted to respond to those suggestions and what barriers exist to enhancing those components of the training. According to participants, this finding was consistent with feedback across sites and was heard from trainees, supervisors, and field trainers, and many sites continued to conducted assessments and developed strategies to address this gap. They pointed out that trainees may already have some knowledge in foundational areas from their education and backgrounds in social work, human services, and child development, making that information easier to grasp, and pre-service trainers are constrained by how much practice with technical processes they can incorporate into the training. Several sites reported making recent efforts to include more hands-on training, such as having trainees work a “mock case” using FSFN Training Sandbox all the way from pre-commencement to the FFA. It was suggested that critical thinking may be one of the missing links, in that comprehensive practice with one type of maltreatment or danger threat may not translate to being able to sufficiently do an FFA on another type, for instance, so trainees needed a better way to understand how to transfer those skills across different scenarios. One participant added that an additional barrier is that with new legislation, critical child safety practice experts have been delegated to a new quality office and away from sites, and these positions were instrumental to some sites in helping with the safety planning modules during pre-service. Finding 2: Knowledge Assessment Throughout the study, trainees responded to surveys that assessed knowledge gained and retained as a result of the CORE training and specialty-track trainings. Findings showed that trainees scored better on the assessments immediately following the training. However, when trainees were re-assessed 6 months later, in some cases, Knowledge Assessment scores were slightly lower than before trainees were in the field. Trainers were asked whether those findings were surprising and why they thought that pattern may exist. Participants shared that there is a heavy emphasis on preparing trainees to pass their test, so after that point, the emphasis on “book knowledge” may decrease as skill-based practice becomes more of a priority. They also suggested that there may be a gap between what is learned in pre-service training and what supervisors support and prioritize once workers are in the field, and there may be room for supervisors and support staff such as field training investigators to play more of a role in reinforcing the knowledge gained in preservice. Some participants pointed out that differences in local policies, procedures, and practices once trainees are in the field may also heighten discrepancies in knowledge gained from pre-service (e.g., timeframes that were learned in pre-service may be shorter in practice to ensure agencies meet performance measures). Supervisor expectations may also differ from what is taught in pre-service. Several participants agreed that the 6-month point, in particular, is typically where stress levels are highest because new CPIs and case managers are coming off of a protected caseload and beginning a full rotation, and therefore have pressure to close cases and address new ones; work-life balance challenges are also often intense at this point.

Finding 3: Training Structure The evaluation team shared the finding that case managers and CPIs expressed a need to have more hands-on training that represented “real life” scenarios, such as more field days (with better coordination) and more opportunities to observe experienced case managers and CPIs in the field. Focus group participants were then asked what some opportunities for enhancing these aspects of preservice training might be, and what the barriers are to implement some of those changes. Many sites agreed there is a constant struggle between training needs (trying to provide sufficient opportunities for observation and shadowing) and operation needs (needing to have workers in their units and be done with training) that contributed to challenges with field training. Some barriers to building in adequate field days are lack of willingness by experienced workers to be shadowed, not having enough workers available when needed, difficulties ensuring that trainees will observe valuable interactions when they do have pre-scheduled field days, and not having enough time to provide adequate training. One comment received widespread agreement: “If we gave them all the field experience that they needed we’d have them in class for six months.” Even when shadowing was not done actively in the field (e.g., in offices), these opportunities were helpful in seeing a realistic day-in-the-life of a CPI or CM, in understanding work prioritization and time management skills, and in gaining rapport with others in the trainees’ unit. Sites shared many of the efforts they made to address this challenge, including the following: assigning trainees to someone from their unit prior to pre-service so they are “attached” to a mentor throughout the process; continuously encouraging staff to help with training; assigning trainees to an administrative team (director and supervisor) that is responsible for managing field training opportunities; ensuring “after hours” opportunities are available during pre-service training; providing other types of “real life” scenario training, such as virtual reality platforms, a training “field house” with live actors to conduct mock home visits, and incorporating mock court processes in a real court to become familiar with the court environment; and inviting children, parents, foster parents, and caregivers to training to share their experiences in interacting with child welfare professionals. Respondents said it was evident that, in instances where trainees had more time to shadow, such as interns who needed to fulfill hours, they were much more prepared to be in the field. Finding 4: Family Functioning Assessments The study team discussed the statewide review of FFAs that ACTION conducted during the evaluation, and that USF partnered with them to incorporate some of their reviews on workers that were in the study cohort. Findings were shared that in the first round of case reviews of FFA-Is, less than one half of the cases reviewed for the study cohort demonstrated sufficient information needed to assess and make accurate decisions for impending danger and safety. Similarly, in case reviews of FFA-O’s, less than one third of the cases reviewed reflected adequate information, with the most significant deficits noted in family engagement, case plan support, and conditions for return. The study team also discussed the data point that supervisory oversight was insufficient in approximately two thirds of the cases reviewed. Focus group participants were asked what their reactions to these findings were, whether they felt the pattern related back to training, and whether training should be changed or enhanced to improve these findings. Participants agreed that this finding was not surprising and was consistent with sites’ own reviews of FFAs. Several participants offered the rationale that the practice model is very in-depth and

FLORIDA INSTITUTE FOR CHILD WELFARE

complex, and it takes a lot of time to master (it was suggested for CPIs, it may take upwards of a year to have a consistent work product with FFAs). One site mentioned completing frequent booster trainings improved FFA sufficiency. It is important to note that the findings shared with participants represented data from approximately the 6- to 9-month point on the job, yet the new analysis of FFA data from the 12-month point presented in this report shows a decreasing level of sufficiency in FFA completion. If familiarity with the practice model was a key variable that influenced sufficiency rates, it would be expected that sufficiency would increase over time, but our findings suggest otherwise, meaning that other variables may be responsible for the low levels of proficiency. Finding 5: Turnover The following turnover rates and insights were shared with focus group participants. Out of 93 case managers and CPIs across all sites, 28 remained at the 12-month point, leading to an overall turnover rate of approximately 70 percent for this group (75% for case managers and 53% for CPIs). When asked about reasons for staying, case managers and CPIs said they had supportive supervisors and coworkers, they enjoyed the work and felt they made a difference. Case managers and CPIs suggested that key reasons new workers leave is that caseload sizes are too high, the environment is too stressful and/or emotionally draining, the expectations for work processes are unrealistic, they do not have adequate supervisor support, they have little work/family balance, and frequent turnover rates have a disproportionate effect on new workers, in particular. Some respondents suggested that pre-service training can help provide a more realistic understanding of what to expect in the field as a way to reduce frequent turnover. Training personnel were asked to discuss what ways pre-service or other early in-service training might better help alleviate the problems associated with high turnover rates and stressful environments. Sites acknowledge the tremendous amount of time and resources that go into pre-service training. Participants shared that, in order to protect time and resources, they made many efforts to try to provide a realistic understanding of the work expectations for case managers and CPIs at many different points in the hiring and training process. Some of these efforts included the following: conducting an orientation prior to training to have an honest conversation about work-life balance and the nature of shift work; bringing supervisors into pre-service to discuss expectations and lessons learned so trainees can have awareness as they go through training and to have a more tangible connection to supervisors so they are not an vague “external entity”; having candidates view video on a day-inthe-life of a CPI and discussing the challenges of the work during hiring interviews, such as work-life balance, attrition rates, missing family events, and childcare challenges; and connecting trainees to staff from individual agencies throughout training so they have a better understanding of the different organizational cultures and expectations at the agency they will be placed with. Some of these efforts were helpful in allowing candidates or trainees to recognize ahead of time that the job was not a good fit for them, and for others it allowed them to organize necessary supports they need to be successful. However, many participants shared that, despite these efforts to provide a realistic and honest portrayal of the challenges of the job, even workers who say they are confident that they will be able to manage the responsibilities often quit several months later because of work-life balance problems. In general, the groups discussed a systemic tension between the needs of training and the needs of operations, acknowledging the need for new hires to complete training and be in the field. At the same time, they sometimes struggled to be able to provide the significant amount of training new hires needed in the time allotted to them.

Finding 6: COVID-19 Changes The study team shared feedback from case managers and CPIs on how COVID-19 impacted their jobs. These findings suggested that for some workers, their caseloads had decreased, allowing them to spend adequate time on cases and to have more work/ family balance since they were working from home and not traveling as much; for others, not much had changed since they were still doing investigations and safety checks, although there were some concerns about virus exposure. Others shared that it was difficult to maintain boundaries while everyone worked from home, as work occurred at all hours. Focus group respondents were asked what ways pre-service training was impacted by COVID-19, and what the potential long-term consequences or opportunities are as a result of these changes. Several sites discussed having to switch to virtual training in the middle of a class when shelter-in-place orders began; since then, several training classes have been conducted fully remotely and in some cases, classes have resumed in-person or in hybrid formats, but most are still fully remote. Some challenges that participants highlighted with virtual training are: ensuring sufficient engagement by participants (although sites reported requirements to have videos on and some form of tracking engagement such as trainees submitting assignments on what they have learned) as well as incorporating adequate opportunities for movement and variations in learning format (e.g., not just sitting in front of a screen the whole time); scheduling training in a consistent way that also allows trainees to meet personal obligations such as childcare needs while children were/are not able to be at school or daycare in-person, and generally being able to help trainees develop much-needed time management skills when the hours and responsibilities all blend together without a nine-to-five structured schedule. It was also noted that there was lack of time for informal opportunities for trainees to talk during lunch, debrief, and bond and lack of options for having one-on-one discussions and other regular interactions with fellow trainees that cannot be replicated online. Some participants discussed the hurdle of developing new skills for online training, which required knowledge of technological capacity of remote platforms, more creative strategies for engagement and balancing live interactions with independent work and managing exhaustion from meeting the demands of the new way of training as it emerges. It was also widely agreed that significant attention needed to be given to ensure trainees were still adhering to professional standards as preparation for client interactions, including meeting dress code requirements, not having family members or pets in the background, and sufficiently engaging in training. A final, but significant, concern with the online platform was losing opportunities for “real” field days, although efforts are still made to have trainees join virtual home visits or unit meetings such as case staffings. Participants discussed the strengths of trainers and other staff in adapting to virtual training and ensuring processes ran smoothly and trainees were engaged, and it was also recognized that many of the trainees seemed adept at remote learning, perhaps due to familiarity with online college courses. Some adaptations made over the past few months have been to utilize break-out rooms to increase opportunities for small group interactions, limiting live instruction to 2-hour periods and balancing that with more “prework” or “homework” to ensure mastery of content, and having several levels of oversight to ensure trainee engagement. Some ongoing concerns were that some elements of training just cannot be replicated online and there may be a significant loss of informal aspects of training, both between participants and between trainers and participants, and that lack of opportunities for in-person observations and shadowing may have very real consequences in worker preparedness. Many sites saw opportunities for embracing aspects of remote learning and adapting to hybrid models when it was safe, which may include more content-based modules online.

FLORIDA INSTITUTE FOR CHILD WELFARE

Discussion and Recommendations The child welfare pre-service training evaluation was designed to: 1) assess the readiness of case managers and CPIs to begin their job responsibilities, 2) determine whether pre-service training is at the level it should be at, and 3) identify both environmental factors and individual coping strategies of workers that facilitate and hinder knowledge acquisition and skill development while in the roles of case managers and child protective investigators. Evaluation activities began at nine sites when a new training cohort came on board in their local area, thus each site progressed through the different data collection milestones at different dates. The nine study sites include six CMOs, two DCF Circuits, and one Sheriff’s Office. Five of the six Regions in Florida are represented. The SW Region was excluded from study due to changes in local leadership. The SE Region includes two sites. This report shared new data from the FFA analysis and training directors focus group, as well as a retrospective analysis of the Knowledge Assessments, the electronic Self-Assessments, both case manager and CPI focus groups and observations, and the organizational culture and climate survey. KNOWLEDGE ASSESSMENT The Knowledge Assessment component was developed to examine transfer of knowledge that occurred as a result of pre-service curriculum training, in addition to knowledge retained over time. This component was designed to speak to the primary research questions regarding whether case managers and CPIs are ready to begin their jobs after going through pre-service training, as well as whether pre-service training is at the level it should be. Findings from knowledge assessments administered to pre-service trainees indicated that knowledge was gained as a result of the CORE curriculum training as well as the Specialty Track training. Although some scores decreased by the 6-month follow-up assessment and again at the 12-month assessment, overall, scores were still greater than those observed at pre-test. It is likely that some items included in the knowledge assessments are more applicable to case managers’ and CPIs roles than others, which may explain the fluctuation over time of correct responses. In addition, as reflected in responses from 6- and 12-month focus groups, supervisor expectations vary considerably, which may contribute to a more nuanced understanding of processes and practices than what can be represented by the knowledge assessments. Further, findings of the Knowledge Assessments were discussed in a focus group for training directors across sites. Focus group participants suggested there is a heavy emphasis on preparing trainees to pass the test. The decrease observed in Knowledge Assessment scores may reflect the emphasis on “book knowledge” that might decrease as skill-based practice becomes more of a priority. Participants also stated there may be a gap between what is learned in pre-service training and what supervisors support and prioritize once workers are in the field. Lastly, when discussing findings of the Knowledge Assessments, training directors suggested that at the 6-month point, in particular, stress levels are typically highest because new CPIs and CMs are coming off of a protected caseload and beginning a full rotation. CASE MANAGER FOCUS GROUPS The purpose of the case manager 6- and 12-month focus groups was to examine how helpful the child welfare pre-service training has been to case managers in the day-to-day work they do. Areas covered during the focus groups at each data point included cultural competency, communication, interviewing techniques 33

and strategies, Family Functioning Assessments, age appropriate assessment of children, strengths-based assessment, case and safety plan implementation, Florida’s Child Welfare Guiding Principles, discussing progress with clients, active and ongoing assessment, modification of safety plans, and suggestions for improvement. Aspects of this component respond in part to each of the primary research questions regarding whether case managers and CPIs are ready to begin their jobs after going through preservice training, whether pre-service training is at the level it should be, and contextual factors which may hinder knowledge acquisition and skill development while in the roles of case managers and child protective investigators. Overall, case managers at 6 months did feel pre-service training was beneficial to developing the necessary skill set for their daily work as child welfare professionals. They indicated that information provided on identifying and understanding maltreatment was adequately covered. Areas of the pre-service training they found most helpful were role playing exercises, shadowing experienced child welfare professionals, and field days. Case managers also identified other experiences as having had a strong influence on their skill set in some areas, more so than the pre-service training. These types of influences were most commonly noted as previous work experience serving children (e.g., previous child welfare experience, teaching, day-care, or serving as a GAL), previous education (e.g., a social work degree program), on the job experience in the last 6 months commonly referred to as “learn as you go,” and help provided by supervisors and coworkers over the past 6 months as needed. One theme that emerged from the 6-month data was a perceived lack of curriculum on specific case manager functions and how to complete them. Case managers felt the CORE training was geared more toward CPIs and proposed several areas they needed more training on, such as navigating all aspects of the judicial process, case and safety plan updates, working in FSFN, aspects of adoption work, and more help with time and stress management. Case managers also asked for more field days scheduled at times when experienced case managers conduct home visits. There was an indication that field days were sometimes scheduled on days when experienced workers were not available to shadow, causing trainees to miss out on valuable in the field experience. Other suggestions included more role playing exercises, jointly carrying an active case during pre-service training (with an experienced child welfare worker), and providing a peer advocate for case managers to go to with questions once in the field, as some expressed a reluctance to ask for help from supervisors who had indicated that pre-service training should have already provided the necessary skill set. Another common theme across focus groups was that the examples presented in training seemed to be best case scenarios and not very representative of the complexity of issues families and child welfare professionals face in real life. Participants recommended again, either more field time and hands-on work to address this, or more complexity added to the examples within existing training curriculum. Overall, case managers at the 12-month mark indicated that they were provided a foundational understanding of their job responsibilities through pre-service training. However, they found it was difficult to apply the knowledge and skills they gained to the field when they returned to their respective units. Specifically, case managers reported their understanding of FFA-Os and safety planning, two critical functions of their daily responsibilities, were not taught and/or practiced at a level that gave them enough knowledge or confidence once they were working directly with children and families. Gaps in knowledge transfer and application in the field could have case decision implications for children and families; implications could include untimely permanency decisions and incomplete or incorrect information gathering around safety and well-being. In addition, first year case manager turnover rates were

FLORIDA INSTITUTE FOR CHILD WELFARE

high. Case managers pointed mainly to the quality and accessibility of supervisors and the support of coworkers as major factors in their desire to remain in their positions. As case managers turn over has high rates, there may be negative impacts to families, such as interference to service provision and permanency processes and greater challenges to decision-making around safety and well-being. Unique to the 12-month focus groups, participants were asked to reflect on why some of their initial training cohort chose to leave their case manager positions, and why others chose to stay at their jobs. Participants named several factors in their decision to stay employed as a case manager, such as loving the work they do, feeling that they made a difference, having a supportive supervisor, having supportive coworkers, and some case management areas perhaps being easier than others to stay in. Alternately, reasons provided as to why peers left their case management position included emotional strain, an unsupportive supervisor, turnover as a domino effect, and low salaries/lack of recognition. CPI FOCUS GROUPS As with the case manager focus groups, the purpose of the 6- and 12-month focus groups with CPIs was to examine how helpful the child welfare pre-service training was to CPIs in their daily work. Aspects of this component respond in part to each of the primary research questions regarding whether case managers and CPIs are ready to begin their jobs after going through pre-service training, whether pre-service training is at the level it should be, and contextual factors which may hinder knowledge acquisition and skill development while in the roles of case managers and child protective investigators. Generally, respondents at the 6- and 12-month points spoke very positively about trainers and their competencies, often referring to them as “amazing” or as deserving of “high praise.” Many CPIs felt that pre-service training provided a strong foundation of knowledge of core principles, especially with regard to understanding and assessing different types of maltreatment. They acknowledged throughout the focus groups that they encounter a wide range of families and scenarios on their jobs, that it is nearly impossible to teach everything they needed to know before starting actual investigations. Respondents were also relatively consistent with key areas they saw as needing improvement, which were to include more opportunities for hands-on learning, including more field days; to provide more practice with assessments and data systems that are required for daily use; and to make the training more comprehensive and realistic in addressing the wide variety of individuals and situations that CPIs can be expected to or encounter in their work. CPIs reported that their understanding of FFA-Is and safety planning, two critical functions of their daily responsibilities, required more opportunities to observe and practice with “real” families in order to fully understand. CPIs relied on their unit supervisors and coworkers to fill in gaps and provide guidance. While relying on unit support is appropriate, CPIs pointed out that expectations and understanding varied from unit to unit even within the same agency. Further complicating this gap in pre-service training is what appears to be a lack of understanding of the holistic approach to the FFA-I. CPIs indicated that once a child is determined to be safe the FFA-I is redundant and unnecessary. Implications to the field could include missing critical information needed to adequately determine child safety and risk by not having an extensive and comprehensive knowledge of the individual and family conditions in the home. In addition, CPIs expressed concerns about turnover rates. With higher turnover rates typically leading to higher caseloads and stress on the system, CPIs may miss critical information needed to determine safety and well-being.

SELF-ASSESSMENT SURVEY Case managers and CPIs completed an electronic, self-assessment survey at baseline (approximately one month), 6 months, and 12 months after their unit start dates. The items on the self-assessment survey are derived from the Florida Child Welfare Competencies and are focused on skills related to family engagement, risk and safety assessment, and identification of strengths and needs. Aspects of this component respond in part to each of the primary research questions regarding whether case managers and CPIs are ready to begin their jobs after going through pre-service training, whether pre-service training is at the level it should be, and contextual factors which may hinder knowledge acquisition and skill development while in the roles of case manager or child protective investigator. Overall, the responses by case managers on the 6-month selfassessment indicated ongoing challenges with key areas in transfer of knowledge that were also reported at the 1-month interval. These included: difficulty obtaining enough hands-on experience in pre-service training, a stressful work environment with unrealistic expectations for caseloads, and some discrepancies between expectations during pre-service training and actual practice, which may, in part, be due to different approaches to the work by superiors. With regard to supports in retaining knowledge and skills, case managers and CPIs consistently pointed to the benefit and importance of internal support by supervisors and other colleagues and mentors. Responses from the 12-month self-assessment indicated increased self-reported competency in completing FFAs over time, and a continuation of frequent skill and competency integration into practices. Though open-ended responses were limited, there were frequent references to the benefits of hands-on learning strategies (e.g., role plays, interviewing skills, field days) and the need to ensure that opportunities for field days are as structured as possible so trainees can benefit from direct observation. It was also apparent that caseload size and high turnover rates continued to have a negative impact on workers who remain in their jobs. In addition, peers and supervisors were identified as key supports in applying skills and knowledge. FAMILY FUNCTIONING ASSESSMENTS This section included a review of FFAs completed by CPIs to make initial case decisions and by case managers to make ongoing case decisions. Action for Child Protection (ACTION) completes quarterly statewide reviews of FFAs for the Department of Children and Families. Therefore, to avoid duplication, USF utilized data collected by ACTION specific to FFAs completed by study participants at the 6 and 12-month study marks. This component was designed to speak to the primary research questions regarding whether case managers and CPIs are ready to begin their jobs after going through pre-service training, as well as whether pre-service training is at the level it should be. Overall results of both reviews revealed an overall lack of understanding of “how to” accurately assess and document decisionmaking regarding child safety and risk. In the 6-month review of FFA-Is, sufficient information was gathered and documented in the majority of cases when assessing Present Danger (73%), Extent of Maltreatment (82%), Surrounding Circumstances of Maltreatment (91%), and Child Functioning (73%). In about half of the cases, CPIs adequately documented Adult Functioning (45%), General Parenting (55%), Discipline or Behavior Management (55%), Caregiver Protective Capacities (45%), Impending Danger (45%), and Safety Decisions (45%). Supervisory involvement is critical to the oversight, guidance, and mentoring of CPIs. During the 6-month review, initial consults occurred 82 percent of the time; however, ongoing supervision dropped significantly to 27 percent of the time. In the most recent review, at 12 months, findings showed a FLORIDA INSTITUTE FOR CHILD WELFARE

significant decline in accuracy of decision making and sufficiency of documentation in all areas. Most significant were in the areas of Extent of Maltreatment (59%), Surrounding Circumstances of Maltreatment (41%), Child Functioning (18%), Adult Functioning (6%), General Parenting (12%), Discipline or Behavior Management (6%), Caregiver Protective Capacities (6%), Impending Danger Decision (18%), and Adequate Safety Plans (25%). When reviewing supervisory oversight, similar to the 6-month review, documentation was seen in 94 percent of the cases for initial consultation and then fell significantly to 28 percent for ongoing supervisory oversight, coaching, and mentoring. Similarly, the differences in the review of FFA-Os from the 6- to 12-month review was significant. FFA-Os at the 6-month review lacked documentation and evidence of sufficient information collection needed to make informed decisions in most areas and domains. Some of the most significant deficits were seen in Family Engagement (29%), Adult Functioning (24%), General Parenting (18%), and Parenting Discipline/Behavior Management (12%), with the review of the case files at 12 months showing that none of the cases reviewed contained sufficient documentation that would support decision-making. The supervisor’s role is to provide ongoing oversight, coaching, and mentoring. However, only 35 percent of cases documented supervisory consultation before case plan approval and 38 percent had evidence of regular consulting between supervisor and case manager. At the 12-month point, further decline in sufficient information and documentation was noted. None of the eight files reviewed reflected that family engagement occurred at the time the case manager began the FFA-O. Significant deficits in information collection and documentation was also noted in Child Functioning (25%), Adult Functioning (13%), General Parenting (13%), and Parenting Discipline/Behavior Management (0%), with the review of the total case file showing that none of the cases reviewed contained sufficient documentation that would support decision-making. Supervisory involvement also declined, with consulting prior to case plan approval and regular consulting both falling to 13 percent. Responses from the electronic self-assessment survey indicated a high level of competency in completing the FFAs, which contrasts greatly with findings from the formal assessment of FFAs conducted by ACTION. At both the 6- and 12-month points, as CPIs and case managers indicated increased competency on the electronic selfassessment survey, ACTION’s review of cases indicated decreased competency. While data collection instruments and methods differed across these two study components, general patterns of self-perceived FFA competency and competency perceived by the external review certainly differed among remaining workers in the study. According to ACTION’s analysis, information collection and documentation within FFA-Is and FFA-Os, as well as supporting documentation within the official record (FSFN), was not sufficient to inform decision-making. Equally as concerning is the lack of supervisory involvement. Although pre-service training appears to teach, within a classroom setting, the purpose of the FFA, assessment, engagement, and information collection and documentation, these skills and knowledge do not appear to translate to the field. Further supporting this assessment were the CPIs and case managers who participated in focus groups and indicated they were not adequately prepared to complete FFAs once in the field. ORGANIZATIONAL CLIMATE AND WORKPLACE CULTURE ASSESSMENT The Organizational Climate and Workplace Culture Assessment was administered to cohorts who completed pre-service training and had been in the field for 3 months and for 12 months in order 35

to examine perceptions of organizational culture and climate among case managers and CPIs, as well as the job turnover intentions. This study component was implemented to respond to the third primary goal of the study, which was to examine both environmental factors and individual coping strategies of workers that facilitate and hinder knowledge acquisition and skill development while in the roles of case managers and CPIs. Although there was a small sample size, some clear trends were evident. First, the take-aways from the 3-month analysis are different from the current 12-month analysis. Lessons learned from 3-month respondents point us in the direction of ways to prevent turnover during the first year, or even the first few months. Data from the 3-month group tell us that it is commonplace to view a new child welfare job as only temporary until something better comes along, with respondents expressing little aspiration for promotion within the field. Eighty-five percent of respondents planned to leave their positions, and 77 percent of respondents were looking to leave child welfare altogether. Insight from data at this 3-month point can be used to make adjustments to hiring and training practices. For instance, knowledge of the reasons for early turnover may help when considering strategies to more effectively “weed out” new hires who are not suited for the job. In addition, sharing information on what types of challenges workers can expect to face in their first year and what impact frequent turnover has on the system may help trainees better determine whether the job is a good fit for them.

and bureaucratic concerns superseded client interests. The 12-month group was also more likely to feel they were making a difference with clients, had a clearer understanding of their work responsibilities, felt that coworkers functioned well together to serve families, that positive staff performances were rewarded by their employer, and that open communication was valued. While emotional strain and burnout was lower in the 12-month group, possibly indicative of a more resilient respondent base, it was still high, with 71 percent of 12-month respondents feeling emotionally strained and overtaxed. Therefore, throughout the first year of employment and on an ongoing basis for child welfare frontline staff, it may be beneficial for agencies to consider what supports can be provided to help staff better cope with emotional strain. This may prove even more important as some workers such as CPIs are temporarily experiencing fewer investigations due to COVID-19. Child welfare staff may return to a surge in child abuse investigations and case rates post pandemic amid other adjustments to pandemic strain. Another option might be to incorporate time management or disaster planning/coping strategies into the pre-service training, or to have more experienced case managers involved in pre-service training to talk about what strategies they developed to cope. The problem of emotional strain and burnout is very clearly not going to disappear, and with the added strain of COVID-19, it is important to carefully think through mental health supports and coping strategies to develop and build a resilient workforce.

Twelve-month organizational culture and climate survey data provide additional insight. For example, at the 12-month point, only 56 percent of respondents believed they would leave child welfare if the opportunity arose. Examination of differences between the two data points in time may provide a better understanding of the characteristics and reasons why staff stay in their positions, as well as what still can be done to retain them as long-term employees. First, those staff that remained in their positions for 12 months or longer were more satisfied overall with their work than the 3-month respondents. Three-month respondents rated their overall job satisfaction at an average score of 2.61, while 12-month respondents rated it at 3.0. While this most likely indicates that staff remaining in their positions at 12 months were generally happier with their jobs than the 3-month group, certainly another possibility is that those who needed to stay in their positions for economic reasons may value their positions more than those is the

In addition, the positive trend seen across time did not hold true for respondent perception of adequate resources and services to meet their clients’ needs. At both 3 and 12 months, the data remained the same—57 percent agreed that there were adequate resources, 28.5 percent disagreed, and 14 percent were neutral. This seems to indicate that while staff who remained employed were more satisfied overall than those staff who left, the system in which CPIs and case managers worked this past year may still present the same challenges regarding resource availability for families. Further study is needed to confirm these patterns and to tease apart factors that may be attributed to individuals (e.g., work ethic, dedication to helping families) and those that are attributed to agencies (e.g., work climate, structural supports) so that this information can be used to improve understanding of where to focus resources to make the most impactful changes.

3-month group. This pairs with the data mentioned above that 12-month respondents were less likely to see their jobs as a brief steppingstone to a different job. Specific job areas that saw the highest gains in satisfaction as time went on were satisfaction with intellectual challenge, organizational climate, and opportunities for advancement. However, just as important to note, satisfaction over time regarding salary and benefits dropped. These two areas come as no surprise but seem to pose critical opportunities for organizations and service systems to retain workers past the oneyear mark.

POLICY RECOMMENDATIONS

Gains across time were seen in the domain areas of flexibility, proficiency, and resistance. Twelve-month respondents were overall more likely to feel they had flexibility to make case-based decisions (rather than employing a cookie cutter approach to case work), their input was more valued by leadership, their agency emphasized quality of service provision over meeting quotas, creativity in service provision was encouraged, and that new ideas were supported. Additional areas showing gains over time were that workers felt more comfortable voicing disparate opinions, they felt their agencies were more willing to adopt new EBPs, and they perceived their agencies as better addressing service delivery barriers. Additional gains across time were seen in the domain areas of job stress, client engagement, functionality, and staff morale. Specifically, 12-month workers were more likely to feel their job expectations were reasonable, and less likely to feel that paperwork FLORIDA INSTITUTE FOR CHILD WELFARE

The following recommendations are made to improve the training experience and first year of employment for new case managers and CPIs: ● Provide additional information to trainees regarding what types of challenges workers can expect to face during their first year (e.g., viewing a day in the life of an experienced case manager/ CPI), so they may better determine whether or not the job is a good fit for them. ● Consider what formal supports, such as field trainer positions, can be provided during a trainees’ first year to help staff better cope with emotional strain. ● Consider additional support positions for supervisors that would assist with observations and feedback once trainees are back in units (e.g., Mentor, Field Training Coach, Advocate). ● Incorporate time management or disaster planning/coping strategies into pre-service training and consider bringing in a more experienced case managers/CPI to a pre-service training day in order to talk about what strategies they developed to cope. ● Continue emphasizing hands-on learning strategies in preservice training such as mock cases and trials, role plays, and FSFN practice. 36

● Consider increasing the number of field days during pre-service training, as well as a more formalized process for assuring that field days are scheduled on days and times that provide new case managers and CPIs with the ability to observe and learn from relevant activities. Participants stressed the need to learn under less scripted conditions than what the current roleplaying exercises offered during pre-service. ● Consider scheduling field days directly after related training content, to better match training content with field experience. For example, if FFA-Is had just been reviewed in class, participants felt it would be helpful, then, to see a case manager complete one in the field. ● Develop strategies across regions to provide realistic opportunities for interacting with families with diverse scenarios during training (e.g., “field house” model or role plays with outside personnel or volunteers). ● Provide clarification during pre-service training on the function and importance of FFAs for CPIs and ensure dedicated inservice training on FFAs is available on a continuous basis. ● Increase training content and duration on interviewing skills, specific to interviewing both parents and young children. ● Increase training content on case management functions. Case manager trainees suggested spending more time on specific aspects of case management rather than the frontend maltreatment that caused the investigation or aspects of CPI work. Participants were interested in more detail on dayto-day activities, processes, and forms case managers need to complete. ● Integrate supervisors into pre-service training processes to create a more fluid transition between knowledge gained during training and expectations in the field. ● Consider ways to decrease the caseload size of new workers during their first year to prevent the cascading effects of turnover.

FLORIDA INSTITUTE FOR CHILD WELFARE

REFERENCES 1

Glisson, C. (2002). The organizational context of children’s mental health services. Clinical child and family psychology review, 5(4), 233-253.

2 Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child abuse & neglect, 22(5), 401-421. 3 Glisson, C., & James, L. R. (2002). The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior, 23(6), 767-794.

FLORIDA INSTITUTE FOR CHILD WELFARE

APPENDIX A: AVERAGE CORRECTLY ANSWERED CORE CURRICULUM KNOWLEDGE ASSESSMENT QUESTIONS Table 16: Average Correctly Answered CORE Curriculum Knowledge Assessment Questions Pre-Test (n = 91)

Post-Test (n = 91)

X (Stand. Dev.)

The Practice Module (4 items)

2.57 (.92)

3.11 (.81)

t (90) = - 4.96 ***

Child Development (3 items)

2.44 (.60)

2.53 (.67)

Trauma and the Child (3 items)

2.10 (.76)

2.23 (.73)

Family Conditions (4 items)

3.42 (.72)

3.52 (.72)

Understanding Child Maltreatment (5 items)

2.91 (1.13)

3.67 (1.10)

t (90) = - 5.38 ***

Assessing and Analyzing Family Functioning (6 items)

4.32 (1.19)

5.20 (.93)

t (90) = - 6.99 ***

Safety and Risk (5 items)

3.53 (1.13)

4.35 (.82)

t (90) = - 6.11 ***

Safety Planning (5 items)

3.41 (1.14)

4.11 (.90)

t (90) = - 5.08 ***

24.69 (3.53)

28.71 (3.53)

t (90) = - 11.48 ***

Overall (35 items)

FLORIDA INSTITUTE FOR CHILD WELFARE

Paired t-tests

APPENDIX B: OBSERVATION CHECKLIST FOR CASE MANAGERS Blind File #:

Case Manager:

Circuit:

Reviewer:

Region:

Review Date:

Transfer Staffing

Home Visit

Permanency Staffing

Other Attendee Roles and Agencies (No names):

INSTRUCTIONS: For each competency area, mark “Y” if the competency or skill was clearly observed and “N” if it was not. If there was no opportunity for the competency or skill to be addressed, mark “N/A” for not applicable. For each competency or skill that was observed, provide one brief case-specific example in the “Example” column. Provide a brief statement about the overall observation, including any situational abnormalities in the “Observer Notes” section at the bottom. Competency or Skill

Assessment (Mark Y, N, or N/A)

Example

Demonstrates sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.). Communicates with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs. Interacts with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a non-confrontational manner. Utilizes interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family. Interprets the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations.

FLORIDA INSTITUTE FOR CHILD WELFARE

Competency or Skill

Assessment (Mark Y, N, or N/A)

Example

Conducts or has conducted on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met. Engages and assesses families from a strengths-based perspective. Develops and implements (or has developed and implemented) a case plan/safety plan based on strengths-based assessment. Identifies and facilitates referrals to appropriate services based on the Family Functioning Assessment. Discusses progress in recommended services. Demonstrates active and ongoing assessment of safety management plan Demonstrates use of safety analysis and planning criteria needed to modify safety plans Demonstrates knowledge and application of Florida’s Child Welfare System Guiding Principals

Observer Notes:

FLORIDA INSTITUTE FOR CHILD WELFARE

APPENDIX C: FIELD OBSERVATION CHECKLIST FOR CPIs

APPENDIX C: FIELD OBSERVATION CHECKLIST FOR CPIs Blind File #:

Case Manager:

Circuit:

Reviewer:

Region:

Review Date:

Transfer Staffing

Home Visit

Permanency Staffing

Other Attendee Roles and Agencies (No names):

INSTRUCTIONS: For each competency area, mark “Y” if the competency or skill was clearly observed and “N” if it was not. If there was no opportunity for the competency or skill to be addressed, mark “N/A” for not applicable. For each competency or skill that was observed, provide one briefcase-specific example in the “Example” column. Provide a brief statement about the overall observation, including any situational abnormalities in the “Observer Notes” section at the bottom.

Competency or Skill

Assessment

Example

(Mark Y, N, or N/A) Demonstrates sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.). Communicates with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs. Interacts with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a nonconfrontational manner. Utilizes interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and conduct ageappropriate interviews with the child, the caregiver, and the family.

Florida Institute for Child Welfare FLORIDA INSTITUTE FOR CHILD WELFARE

76 42

Competency or Skill

Assessment

Example

(Mark Y, N, or N/A) Interprets the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations. Conducts or has conducted on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met. Engages and assesses families from a strengths-based perspective. Develops and implements (or has developed and implemented) a case plan/safety plan based on strengthsbased assessment. Identifies and facilitates referrals to appropriate services based on the Family Functioning Assessment. Discusses progress in recommended services. Demonstrates active and ongoing assessment of safety management plan Demonstrates use of safety analysis and planning criteria needed to modify safety plans Demonstrates knowledge and application of Florida’s Child Welfare System Guiding Principals

Observer Notes:

Florida InstituteFOR forCHILD Child WELFARE Welfare FLORIDA INSTITUTE

APPENDIX D: OBSERVATION CHECKLIST OBJECTIVE AND CORRESPONDING COMPETENCY KEY

Appendix D: Observation Checklist Objective and Corresponding Competency Key Item

Curriculum Component

Demonstrates sensitivity to cultural differences and ethnicity among clients.

Competency 15 Objectives 33, 44, & 53

Communicates with family members and collaterals by asking parents or guardians to elicit, define, and prioritize concerns and needs.

Competency 17

Interacts with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a non-confrontational manner.

Competency 18

Utilizes interviewing techniques and strategies to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family.

Competency 19

Interprets the results of the department’s Family Functioning Assessment to make appropriate child safety determinations.

Competency 39

Identifies, documents, and conducts on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met. Engages and assesses families from a strengths-based perspective to develop and implement a case plan/safety plan based on this assessment. Identifies and makes referrals to appropriate services based on the Family Functioning Assessment. Discusses progress in suggested services.

Demonstrates general knowledge of Florida’s child welfare system.

Florida Institute for Child Welfare

FLORIDA INSTITUTE FOR CHILD WELFARE

Core Labs 1 - 3

Core Lab 1

Core Labs 1 - 4

Objectives 133 - 147 Competency 43 Objectives 133 – 135 Core Lab 5 Competency 64 Objective 141 Core Labs 1 & 5 Competency 81 Objectives 41, 170 & 171 Competency 89 Objectives 170 & 171 Competency 2 Objectives 1-14

APPENDIX E: 6- AND 12-MONTH CASE MANAGER FOCUS GROUP PROTOCOL We are a team of researchers and practitioners from the University of South Florida. Our study is looking at how helpful the child welfare pre-service training is to you in the day to day work that you do. You may have been a part of some knowledge assessment and observation activities as part of our study earlier this year. This is our follow up activity. We’d like to ask you for your opinions and experiences about the impact of the child welfare pre-service training on your work, now that you have been at your job a bit longer. There are no right or wrong answers and we’d like to remind everyone that we’re not evaluating any of you, just the training curriculum. With your permission, we will be recording this session so that we can focus on listening carefully to what you’re saying. We will not use any names or other identifying information when we write our reports, so we hope you’ll feel comfortable sharing your thoughts freely, as your feedback is very valuable. We know that you have many demands on your schedule, so in order to make this the most effective use of your time, we ask that you please give your full attention to the focus group and refrain from taking non-emergency calls or texts so that you’re able to get back to your jobs as soon as possible. Finally, thank you so much for coming today and sharing your thoughts. Do you have any questions before we begin? 1) Thinking back to your child welfare pre-service training, do you feel that it has helped you show sensitivity to cultural and ethnic differences and among your clients? (Probe: Some examples may include respecting a client’s communication style, acknowledging their value system, or discussing culturally specific services, etc.) • A re there other factors that had a greater influence on your sensitivity toward clients? What are they? 2) Again, thinking back to your child welfare pre-service training, do you feel that it has helped you communicate with family members and collaterals in order to discuss their needs and concerns? Please share some examples. • What other factors have had an influence on your ability to communicate with clients? (Probe: other trainings, mentoring, interactions with clients) 3) Reflecting back on your child welfare pre-service training, do you feel it has helped you develop interviewing techniques and strategies that help you conduct age-appropriate interviews with children, caregivers, and families? (Probe: demonstrating respect for clients, active listening, using exploring and focusing skills) • Are there other factors that had a greater influence on your ability to conduct successful interviews with clients? What are they?

5) Again, thinking back to your child welfare pre-service training, do you feel that it has helped you conduct on-going, ageappropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met? • Are there other factors that had a greater influence on your ability to interpret results and make appropriate referrals? If so, what are they? 6) Reflecting back on your child welfare pre-service training, do you feel it has helped you engage and assess families from a strengths-based perspective? Please share examples. • If there are other factors that had a greater influence on your ability to engage and assess families from a strengths-based perspective, what are they? 7) Do you feel that your pre-service training has helped you develop and implement case plans/safety plans based on strengths-based assessment? Please share examples. • Are there other factors that had a greater influence on your ability to develop and implement case plans/safety plans based on strengths-based assessment? What are they? 8) Again, thinking back to your child welfare pre-service training, do you feel that it has helped you demonstrates knowledge and application of Florida’s Child Welfare System Guiding Principles? Why or why not? 9) Reflecting back on your child welfare pre-service training, do you feel it has helped you feel confident in discussing progress with clients in their recommended services? Why or why not? 10) Do you feel that your child welfare pre-service training has helped you be able to demonstrate active and ongoing assessment of your clients’ safety management plans? Why or why not? 11) Again, thinking back to your child welfare pre-service training, do you feel that it has helped you make good use of safety analysis and planning criteria needed to modify safety plans? Why or why not? 12) Next, how would you improve upon the child welfare preservice training? Is there anything that could be included or left out that would make the training experience more helpful to you in your day to day work? 13) Finally, we’d like to learn a bit more about why some case managers in your training cohort have stayed in their current positions and others have left their jobs. What have been your reasons for staying? 14) If possible, can you tell us generally about what you perceive were your co-workers’ reasons for leaving? 15) What effects, if any, has the COVID-19 pandemic had on your job?

4) Do you feel that pre-service training has helped you interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations? • And how about making appropriate referrals? • Are there other factors that had a greater influence on your ability to interpret results and make appropriate referrals? What are they?

FLORIDA INSTITUTE FOR CHILD WELFARE

APPENDIX F: 6- AND 12-MONTH CPI FOCUS GROUP PROTOCOL

We are a team of researchers from the University of South Florida. Our study is looking at how helpful the child welfare pre-service training is to you in the day to day work that you do. You may have been a part of some knowledge assessment and observation activities as part of our study earlier this year. This is our follow up activity. We’d like to ask you for your opinions and experiences about the impact of the child welfare pre-service training on your work, now that you have been at your job a bit longer. There are no right or wrong answers and we’d like to remind everyone that we’re not evaluating any of you, just the training curriculum. Finally, thank you so much for coming today and sharing your thoughts.

6) Reflecting back on your child welfare pre-service training, do you feel it has helped you engage and assess families from a strengths-based perspective? Please share examples.

With your permission, we will be recording this session so that we can focus on listening carefully to what you’re saying. We will not use any names or other identifying information when we write our reports, so we hope you’ll feel comfortable sharing your thoughts freely, as your feedback is very valuable.

7) Thinking back to your child welfare pre-service training, do you feel it has helped you develop and implement case plans/safety plans based on strengths-based assessment? Please share examples.

We know that you have many demands on your schedule, so in order to make this the most effective use of your time, we ask that you please give your full attention to the focus group and refrain from taking non-emergency calls or texts so that you’re able to get back to your jobs as soon as possible. Finally, thank you so much for coming today and sharing your thoughts. Do you have any questions before we begin? 1) Thinking back to your child welfare pre-service training, do you feel that it has helped you show sensitivity to cultural and ethnic differences and among your clients? Please share some examples. • (Probe: respecting a client’s communication style, acknowledging their value system, discussing culturally specific services, etc.). • Are there other factors that had a greater influence on your sensitivity toward clients? What are they? 2) Again, thinking back to your child welfare pre-service training, do you feel that it has helped you communicate with family members and collaterals in order to discuss their needs and concerns? Please share some examples. • Are there other factors that had a greater influence on your ability to communicate with clients? What are they? 3) Reflecting back on your child welfare pre-service training, do you feel it has helped you develop interviewing techniques and strategies that help you conduct age-appropriate interviews with children, caregivers, and families?

• Are there other factors that had a greater influence on your ability to interpret results and make appropriate referrals? What are they?

• Are there other factors that had a greater influence on your ability to engage and assess families from a strengths-based perspective? What are they?

• Are there other factors that had a greater influence on your ability to develop and implement case plans/safety plans based on strengths-based assessment? What are they? 8) Again, thinking back to your child welfare pre-service training, do you feel that it has helped you demonstrates knowledge and application of Florida’s Child Welfare System Guiding Principles? Why or why not? 9) Reflecting back on your child welfare pre-service training, do you feel it has helped you communicate clearly the information collected during an investigation that has been used to determine the need for case management? Why or why not? 10) Next, how would you improve upon the child welfare preservice training? • Is there anything that could be included or left out that would make the training experience more helpful to you in your day to day work? 11) Finally, we’d like to learn a bit more about why some CPIs in your training cohort have stayed in their current positions and others have left their jobs. What have been your reasons for staying? 12) If possible, can you tell us generally about what you perceive were your co-workers’ reasons for leaving? 13) What effects, if any, has the COVID-19 pandemic had on your job?

• (Probe: demonstrating respect for clients, active listening, using exploring and focusing skills) • Are there other factors that had a greater influence on your ability to conduct successful interviews with clients? What are they? 4) Thinking back to your child welfare pre-service training, do you feel it has helped you interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations? And make appropriate referrals? • Are there other factors that had a greater influence on your ability to interpret results and make appropriate referrals? What are they? FLORIDA INSTITUTE FOR CHILD WELFARE

APPENDIX G: 6-MONTH FOCUS GROUP CODING DEFINITIONS – CASE APPENDIX G:MANAGERS 6-MONTH FOCUS GROUP CODING DEFINITIONS – CASE MANAGERS

Assessment: TRNG

Discussion of how pre-service training helped case managers conduct age-appropriate assessment activities that ascertain if a child's physical, mental, social, and educational needs are met.

Assessment: OTHR

Discussion of what other job-related factors outside of preservice training have had a greater influence on helping case managers conduct age-appropriate assessment activities that ascertain if a child's physical, mental, social, and educational needs are met.

Case/Safety Plan: TRNG

Discussion of how pre-service training helped case managers develop and implement case plans or safety plans based on strengths-based assessment.

Case/Safety Plan: OTHR

Discussion of what other job-related factors outside of preservice training have had a greater influence on helping case managers develop and implement case plans or safety plans based on strengths-based assessment.

Communication: TRNG

Discussion of how pre-service training helped case managers communicate with family members and collaterals in order to discuss their needs and concerns.

Communication: OTHR

Discussion of how other factors outside pre-service training that have had a greater influence on case managers’ ability to communicate with family members and collaterals in order to discuss their needs and concerns, including other trainings, mentoring, interactions with clients, etc.

Cultural Competency: TRNG

Discussion of how pre-service training helped case managers show sensitivity to cultural and ethnic differences among clients (e.g. respecting client’s communication style, acknowledging their value system, or discussing culturally specific services).

Cultural Competency: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on case managers’ ability to show sensitivity to cultural and ethnic differences among clients (e.g. respecting client’s communication style, acknowledging their value system, or discussing culturally specific services).

FFA: TRNG

Discussion of how pre-service training helped case managers interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations or referrals.

FFA: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers

Florida Institute for Child Welfare

FLORIDA INSTITUTE FOR CHILD WELFARE

interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations or referrals. Guiding Principles: TRNG

Discussion of how pre-service training helped case managers demonstrate knowledge and application of Florida’s Child Welfare System Guiding Principles.

Guiding Principles: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers demonstrate knowledge and application of Florida’s Child Welfare System Guiding Principles.

Improvements

Discussion of what improvements case managers would like to see to Florida’s child welfare pre-service training, including components that should be added or left out to make the training experience more helpful in their day-to-day work.

Interview Techniques: TRNG

Discussion of how pre-service training helped case managers develop interviewing techniques and strategies that help them conduct age-appropriate interviews with children, caregivers, and families (e.g., demonstrating respect for clients, active listening, use of exploring and focusing skills).

Interview Techniques: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers develop interviewing techniques and strategies that help them conduct age-appropriate interviews with children, caregivers, and families (e.g., demonstrating respect for clients, active listening, use of exploring and focusing skills).

Introduction

Introductory information about the nature of the focus group, including names, agency, interviewer, etc.

Safety Mgmt Assmt: TRNG

Discussion of how pre-service training helped case managers conduct ongoing, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

Safety Mgmt Assmt: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers conduct ongoing, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

Safety Plan Modification: TRNG

Discussion of how pre-service training helped case managers make good use of safety analysis and planning criteria needed to modify safety plans.

Safety Plan Modification: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers make good use of safety analysis and planning criteria needed to modify safety plans.

Florida Institute for Child Welfare FLORIDA INSTITUTE FOR CHILD WELFARE

84 48

Service Progress: TRNG

Discussion of how pre-service training helped case managers feel confident in discussing progress with clients in their recommended services.

Service Progress: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers feel confident in discussing progress with clients in their recommended services.

Strengths-Based: TRNG

Discussion of how pre-service training helped case managers engage and assess families from a strengths-based perspective.

Strengths-Based: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping case managers engage and assess families from a strengths-based perspective.

Florida Institute for Child Welfare FLORIDA INSTITUTE FOR CHILD WELFARE

85 49

APPENDIX H: 6-MONTH FOCUS GROUP CODING DEFINITIONS – CPIs

APPENDIX H: 6-MONTH FOCUS GROUP CODING DEFINITIONS – CPIs Assessment: TRNG

Discussion of how pre-service training helped CPIs conduct appropriate assessment activities that ascertain if a child's physical, mental, social, and educational needs are met.

Assessment: OTHR

Discussion of what other job-related factors outside of preservice training have had a greater influence on helping CPIs conduct appropriate assessment activities that ascertain if a child's physical, mental, social, and educational needs are met.

Case/Safety Plan: TRNG

Discussion of how pre-service training helped CPIs develop and implement case plans or safety plans based on strengthsbased assessment.

Case/Safety Plan: OTHR

Discussion of what other job-related factors outside of preservice training have had a greater influence on helping CPIs develop and implement case plans or safety plans based on strengths-based assessment.

Communication: TRNG

Discussion of how pre-service training helped CPIs communicate with family members and collaterals in order to discuss their needs and concerns.

Communication: OTHR

Discussion of how other factors outside pre-service training that have had a greater influence on CPIs’ ability to communicate with family members and collaterals in order to discuss their needs and concerns, including other trainings, mentoring, interactions with clients, etc.

Cultural Competency: TRNG

Discussion of how pre-service training helped CPIs show sensitivity to cultural and ethnic differences among clients (e.g. respecting client’s communication style, acknowledging their value system, or discussing culturally specific services).

Cultural Competency: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on CPIs’ ability to show sensitivity to cultural and ethnic differences among clients (e.g. respecting client’s communication style, acknowledging their value system, or discussing culturally specific services).

FFA: TRNG

Discussion of how pre-service training helped CPIs interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations or referrals.

FFA: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping CPIs interpret the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations or referrals.

Guiding Principles: TRNG

Discussion of how pre-service training helped CPIs demonstrate knowledge and application of Florida’s Child Welfare System Guiding Principles.

Florida Institute for Child Welfare

FLORIDA INSTITUTE FOR CHILD WELFARE

Guiding Principles: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping CPIs demonstrate knowledge and application of Florida’s Child Welfare System Guiding Principles.

Improvements

Discussion of what improvements CPIs would like to see to Florida’s child welfare pre-service training, including components that should be added or left out to make the training experience more helpful in their day-to-day work.

Interview Techniques: TRNG

Discussion of how pre-service training helped CPIs develop interviewing techniques and strategies that help them conduct age-appropriate interviews with children, caregivers, and families (e.g., demonstrating respect for clients, active listening, use of exploring and focusing skills).

Interview Techniques: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping CPIs develop interviewing techniques and strategies that help them conduct age-appropriate interviews with children, caregivers, and families (e.g., demonstrating respect for clients, active listening, use of exploring and focusing skills).

Introduction

Introductory information about the nature of the focus group, including names, agency, interviewer, etc.

Comm Investigation Info: TRNG

Discussion of how pre-service training helped CPIs communicate clearly the information collected during an investigation that has been used to determine the need for case management.

Comm Investigation Info: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping CPIs communicate clearly the information collected during an investigation that has been used to determine the need for case management.

Strengths-Based: TRNG

Discussion of how pre-service training helped CPIs engage and assess families from a strengths-based perspective.

Strengths-Based: OTHR

Discussion of what other factors outside pre-service training have had a greater influence on helping CPIs engage and assess families from a strengths-based perspective.

Florida Institute for Child Welfare FLORIDA INSTITUTE FOR CHILD WELFARE

87 51

APPENDIX I: CASE MANAGER AND CPI SELF-ASSESSMENT SURVEY

2) I can provide sufficient descriptive information when completing the Family Functioning Assessment:

Strongly Agree

b. Agree

INTRODUCTION The University of South Florida is conducting a study on the impact of pre-service training for newly hired child welfare professionals in Florida. The goal of this survey is to collect responses from new CPIs and Case Managers at different points in time after pre-service training. We are asking you to take part in this research study because you are currently or have recently completed training to be a child protective investigator or case manager. Your responses immediately after training may change as you spend more time in the field and reflect on your experiences. At other times, a response may be so important to you that it stays consistent over time. We will send this survey to you at 3 months, 6 months, 9 months, and 12 months post-training. We are interested in learning from you as you go – whether it is immediately after the training or one-year posttraining. Your participation is this study is voluntary. There are no anticipated risks to taking part in this study, as participation will not affect your job status, and identifying information will be excluded from analyses and reports. You may experience some benefits to participation, including reflecting on the training and skill application process and heightened awareness of the importance of child welfare practice. You may also benefit from the knowledge that your participation will help to improve child welfare training and practice. There is no compensation for participating in this study, and there is no cost. Funding for the study is provided by the Florida Institute for Child Welfare at Florida State University. If you have any questions or concerns about this study, you may contact the Principal Investigator, Amy Vargo, at 813-974-5356 or avargo@usf.edu. Thank you very much for your time!

c. Disagree

• Child Protective Investigator

Strongly Disagree

3) I am confident in my ability to rate the functioning level of children:

Strongly Agree

b. Agree c. Disagree

Strongly Disagree

4) I am confident in my ability to rate the caregiving capacity of adults:

Strongly Agree

b. Agree c. Disagree

Strongly Disagree

5) I have enough time to fully complete the Family Functioning Assessment:

Strongly Agree

b. Agree c. Disagree

Strongly Disagree

6) I f there is anything you would like to elaborate on regarding your experiences with the Family Functioning Assessment, please do so below:

ROLE 1) What is your current role?

PRACTICE COMPETENCY

• Less than 3 months

For the following questions, please indicate whether you feel you demonstrated each skill during your last three months of work by selecting Yes or No. If there was no opportunity to demonstrate a skill, mark N/A. Please provide a briefcase-specific example of how you demonstrated the skill in the Example box (i.e., specific things you said or did during a visit that represents a particular skill).

• Between 3-6 months

• Case Manager • Other (Please Specify) _____________ 2) How long ago did you complete pre-service training?

• Between 6-12 months • More than 12 months SELF-ASSESSMENT – FFA COMPETENCY For the following questions, please select the response you feel best represents your experiences completing the Family Functioning Assessments. 1) I understand all of the components I am required to complete in the Family Functioning Assessment:

Strongly Agree

b. Agree c. Disagree

Strongly Disagree

FLORIDA INSTITUTE FOR CHILD WELFARE

Competency or Skill

Assessment (Mark Y, N, or N/A)

Examples of how YOU have used this skill in your Work

I demonstrated sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.). I communicated with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs. I interacted with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a nonconfrontational manner. I utilized interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family. I interpreted the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations. I conducted or have conducted on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met. I engaged and assessed families from a strengths-based perspective. I developed and implemented (or have developed and implemented) a case plan/safety plan based on strengths-based assessment. I identified and facilitated referrals to appropriate services based on the Family Functioning Assessment. I discussed progress in recommended services. I demonstrated general knowledge of Florida’s child welfare system.

Florida Institute for Child Welfare

FLORIDA INSTITUTE FOR CHILD WELFARE

TRANSFER OF LEARNING (OPEN-ENDED QUESTIONS) 1) At your current point post-training, what do you feel were the most helpful aspects of the pre-service training that helped you prepare for the work you’re doing in the right now in the field? 2) At your current point post-training, what do you feel were the least helpful aspects of the pre-service training that helped you prepare for the work you’re doing in the right now in the field? 3) Please describe any resources or supports given to you given to you during your time as a new case manager that have helped you apply what you learned in the pre-service training to your everyday work. (Some examples may include, but are not limited to supervision, peer support, caseload size, time to complete tasks, or follow-up training) 4) What, if any, are some of the barriers you have experienced as you have tried to implement what you learned during the preservice training to your everyday case work? (Some examples may include, but are not limited to caseload size, turnover, agency policies, etc.). 5) Based on your most recent experiences in the field, what suggestions do you have for improving pre-service training?

FLORIDA INSTITUTE FOR CHILD WELFARE

APPENDIX J: CLIMATE AND CULTURE SURVEY

7) Employees are expected to be responsive to the unique needs of each client.

A team at the University of South Florida is working with the Florida Institute for Child Welfare and the Department of Children and Families to conduct an evaluation of child welfare pre-service training in Florida. As a part of this evaluation, we invite you to participate in the following survey intended to assess the culture and climate of the organization you work for (whether DCF, a lead agency, or a CMO). Case managers and Child Protective Investigators in circuits across Florida have been invited to participate.

8) Employees are expected to have and apply up-to-date knowledge on current best practices.

The following survey is called the Organizational Culture and Climate Assessment. In this survey, you will be asked questions related to your professional work experience, factors related to job satisfaction, and turnover intention. You will also be asked to rate your level of agreement on items related to your work environment. Your participation is valued and important because it will help explain the organizational culture and climate in the child welfare system.

11) Administrators provide visible, ongoing support for new ideas and innovations.

Your participation is completely voluntary—you have the right to stop at any time or decline to answer any question. Your privacy is important to us. Your individual identity will be kept confidential. Shared perceptions of all those who participate in this survey will be aggregated to represent the organizational culture and climate in the child welfare system. It is anticipated that it will take approximately 20 minutes to complete this survey. By completing this survey, you are indicating that you have read the description of this survey and that you agree to participate. The Organizational Culture and Climate Assessment is part of the University of South Florida’s evaluation of child welfare pre-service training in Florida. If you have any questions, please contact Amy Vargo at avargo@usf.edu or (813) 974-5356.

9) There is a stronger emphasis on the quality of services than on the number of clients or adherence to timelines and quotas. 10) Employees are encouraged to come up with better ways to serve their clients.

12) Employees are expected to not make waves. 13) Our organization is quick to adopt new evidence-based and promising practices. 14) There is no real organization effort to address service barriers or develop new ways of providing services. 15) 15. Interests of the client are often replaced by bureaucratic concerns, such as paperwork. 16) I often feel emotionally drained and burnt out from my work. 17) There is not enough time for me to do my work well. 18) There are insufficient services and resources to meet the needs of clients. 19) My workload and job expectations are reasonable. 20) I have a feeling of success and accomplishment in my work.

PART I

21) I care about my clients and treat them with respect.

For the first part of this survey, you will be asked to rate your level of agreement on items related to your work environment. As a reminder, personally identifying questions are not recorded and your individual responses are kept confidential.

22) I work together with my clients to address their needs.

The following scale will be used for this section: Strongly Disagree, Disagree, Somewhat Disagree, Neither Agree nor Disagree, Somewhat Agree, Agree, or Strongly Agree Please indicate the extent to which you agree with each of the following statements about your agency and your job using the scale provided. 1) Employees are given flexibility to make case decisions independently. 2) Employees are supported and encouraged to make decisions that are reflective of the individual strengths and needs of families. 3) Employee performance is primarily evaluated based upon timely completion of required tasks and documentation. 4) Every case must be handled in the same way, regardless of individual family strengths and needs. 5) Employees have opportunities to provide input to leadership concerning ways to improve practice. 6) There is evidence that employee input is considered by leadership in policy and decision making.

FLORIDA INSTITUTE FOR CHILD WELFARE

23) My work offers opportunities to make a difference. 24) Our organization provides opportunities to advance if you work for it. 25) I have a clear understanding of my work responsibilities. 26) My co-workers work together as a team to improve the effectiveness of the services we provide. 27) Outstanding work is recognized and rewarded. 28) I have a good understanding of the organization’s mission, vision, and strategic plan. 29) The work atmosphere encourages open and honest communication. 30) We are given accurate and appropriate feedback about our performance. 31) I am compensated fairly for the work I do. 32) I frequently think about quitting my job. 33) I like my job and the organization I work for. 34) I would leave my job tomorrow if I was offered a different job for the same salary but with less stress.

PART II: The second part of this survey asks you to answer questions on your demographics, education, work experience, importance of job characteristics, job satisfaction, and turnover intentions. Demographics What is your current age? Gender: • Male • Female Race: • Caucasian • African-American/Black • Asian • Native American

Which factor in the question above represents your MOST important reason for working in an area outside of your HIGHEST degree field? Select the most important reason. • Pay, promotion opportunities • Working conditions (hours, equipment, working environment) • Job location • Change in career or professional interests • Family-related reasons • Job in the highest degree field not available Importance of Job Characteristics The following scale will be used for this section: Not Important, Somewhat Unimportant, Somewhat Important, Very Important When thinking about a job, how important is each of the following factors to you?

• Mixed Race

• Salary

• Other

• Benefits

Ethnicity:

• Job security

• Hispanic

• Organizational climate and support

• Non-Hispanic

• Opportunities for advancement

Education What is your highest level of completed education? • High School • Some College • Associate degree • Bachelor degree • Master degree or above What was the primary field of study for your highest degree? • Social Work • Other

• Intellectual challenge • Level of responsibility • Degree of independence • Contribution to society

Job Satisfaction The following scale will be used for this section: Very Dissatisfied, Somewhat Dissatisfied, Somewhat Satisfied, Very Satisfied Thinking about your principal job, please rate your satisfaction with that job’s.... • Salary

Please indicate the “other” primary field of study for your highest degree: _________________________

• Benefits

Thinking about the relationship between your work and your education, to what extent was your work related to your HIGHEST degree field? Was it...

• Organizational climate and support

• Closely related • Somewhat related • Not related If not related, did these factors influence your decision to work in an area outside of your HIGHEST degree field? Mark yes or no for each.

• Job security • Opportunities for advancement • Intellectual challenge • Level of responsibility • Degree of independence • Contribution to society How would you rate your overall satisfaction with your job? • Very satisfied

• Pay, promotion opportunities

• Somewhat satisfied

• Working conditions (hours, equipment, working environment)

• Somewhat dissatisfied

• Job location

• Very dissatisfied

• Change in career or professional interests • Family-related reasons • Job in the highest degree field not available FLORIDA INSTITUTE FOR CHILD WELFARE

Turnover Intentions The following scale will be used for this section: Not likely, Somewhat likely, Very likely During the next year, how likely is it that you will leave this job to: • Accept another job related to child welfare • Accept another job unrelated to child welfare

FLORIDA INSTITUTE FOR CHILD WELFARE