evaluation of department of children and families core and

Florida Institute for Child Welfare 1 | P a g e

Evaluation of Department of Children and Families Core and Specialty Track Pre-

Service Training

conducted by the

Department of Child and Family Studies at the University of South Florida

Research Staff Amy Vargo Pam Menendez Anna Abella Lisa Coy Cassandra Thomas Monica Landers

This report compiled by the Florida Institute for Child Welfare for dates covering

February 1, 2018 – September 20, 2019


Content Executive Summary ....................................................................................................................................... 4

Evaluation Purpose ................................................................................................................................... 4

Evaluation Overview ................................................................................................................................. 4

Knowledge Gain ........................................................................................................................................ 4

Skill Application ......................................................................................................................................... 4

Self-Assessment ........................................................................................................................................ 5

Organizational Climate and Culture Assessment ...................................................................................... 6

Training Initiatives Interviews ................................................................................................................... 7

Summary ................................................................................................................................................... 8

Introduction .................................................................................................................................................. 9

Site Selection ................................................................................................................................................. 9

Pilot Study ................................................................................................................................................... 10

Study Site Progress to Date ......................................................................................................................... 10

Evaluation Activities: Status and Results to Date ....................................................................................... 12

Knowledge Assessment .......................................................................................................................... 12

CORE Knowledge Assessment ............................................................................................................. 12

Specialty Tracks ....................................................................................................................................... 13

Case Manager Specialty Track. ........................................................................................................... 14

CPI Specialty Track. ............................................................................................................................. 15

Observational Checklist and Self-Report Survey..................................................................................... 16

Observation and Self-Assessment Results .......................................................................................... 16

Self-Assessment Results .......................................................................................................................... 19

Family Functioning Assessment .......................................................................................................... 20

Practice Competency: Case Managers ................................................................................................ 21

Transfer of Learning: Case Managers ................................................................................................. 22

Practice Competency: CPIs.................................................................................................................. 23

Transfer of Learning: CPIs ................................................................................................................... 24

Organizational Climate and Workplace Culture Assessment ................................................................. 25

Demographics and Education ............................................................................................................. 25

Importance of Job Characteristics and Job Satisfaction ..................................................................... 26

Turnover Intentions ............................................................................................................................ 27


Organizational Culture and Climate Characteristics ........................................................................... 28

Training Initiative Interviews ...................................................................................................................... 30

Training Structure ................................................................................................................................... 30

First year training initiatives ................................................................................................................... 31

Focus Group Update ................................................................................................................................... 35

Summary and Discussion ............................................................................................................................ 35

Next Steps ................................................................................................................................................... 38

References .................................................................................................................................................. 39

Appendix A: Average Correctly Answered Core Curriculum Knowledge Assessment Questions............... 40

Appendix B: Field Observation Checklist for Case Managers ....................................................................... 0

Appendix C: Field Observation Checklist for CPIs ......................................................................................... 2

Appendix D: Observation Checklist Objective and Competency Key ........................................................... 4

Appendix E: Self-Assessment Survey ............................................................................................................ 5


Executive Summary

Evaluation Purpose The purpose of this study is to provide an evaluation of Florida’s child welfare pre-service training to

determine the training’s impact on job readiness, knowledge gain and application, and work

environment.

Evaluation Overview Multiple methods are being used to conduct this evaluation, including on-site observation, focus groups, electronic surveys, phone interviews, and Family Functioning Assessments (FFA) review. The study is being conducted across nine sites throughout the state, which includes six case management

organizations (CMOs), two DCF circuits, and one sheriff’s office that provides child protective investigations. Five of six regions are represented. Evaluation activities follow cohorts of pre-service trainees from each site from pre-service training through one year on the job. The majority of evaluation activities have been completed for up to six months post-training for all sites, and data from those components are included in this report.

Knowledge Gain Preliminary findings indicate that case managers and child protective investigators (CPIs) across the

state show significant knowledge gain knowledge in after completing the CORE portion of pre-service training in five key areas: 1) The Practice Module; 2) Understanding Child Maltreatment; 3) Assessing and Analyzing Family Functioning; 4) Safety and Risk; and 5) Safety Planning. The average scores from the CORE knowledge assessment increased by 11.5 percent from pre- to post-test. For the Specialty Track components of the pre-service training, significant improvements in scores from pre- to post-tests were seen in five out of eight modules for the Case Manager track and five out of six modules for the CPI track. Overall, the average scores from the Specialty Track component increased by 7.9 percent.

Skill Application Observations were conducted to determine whether newly trained case managers and CPIs demonstrated key competencies learned in pre-service training during interactions with families immediately following their first month on the job. Observations from site visits across the nine study sites suggest that case managers and CPIs demonstrated desired interpersonal skills with clients, including cultural competency, communication, interviewing, engagement, and strengths-based approaches, as well as knowledge of services and the general child welfare system. Case managers demonstrated these skills in 95 percent of observable instances, and CPIs demonstrated these skills in 83 percent of observable instances.

12%

Pre/Post

CORE

Knowledge

Increase

8%

Pre/Post

Specialty

Track

Knowledge

Increase


Self-Assessment This component is conducted via electronic survey approximately one month after the unit start date

and asks case managers and CPIs to assess their practice competency during site visits via skill

assessment mirrored from the observations above. Additionally, case managers and CPIs assessed their

competency in completing Family Functioning Assessments and identifying environmental factors that

support or hinder knowledge transfer from pre-service training.

Practice Competency. Case managers perceived that they were demonstrating most key competency

skills with their families at the one-month assessment point, especially respectful communication. CPI

respondents indicated with high frequency that they demonstrated most of the skills in the assessment

during their last family visit. Skill areas with the highest marks include utilizing interviewing techniques

and strategies, identifying and facilitating referrals to appropriate services, and demonstrating

knowledge and application of Florida’s Child Welfare System Guiding Principles.

Knowledge Transfer. Case managers and CPIs commented that the most valuable aspects of the pre-

service training were the hands-on, practical components, such as interviewing techniques for different

clients, FSFN training, role play, and field observations. For ways that pre-service training could be

improved, case managers suggested they might have

benefited from more FSFN training, a better

understanding of specific processes like making

referrals to specialists and learning more about

available resources and provider services. CPIs gave

feedback that there should be less “book work” and

more hands-on activities.

95%

83%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Case Managers

CPIs

Frequency Key of Competencies Observed During Site Visits

Not knowing exactly what to do, how to

do it, or when to do it can be a barrier.

Having multiple cases with different

requirements can be confusing for

someone just getting started.


Supports and Barriers. Supervisor and peer support

were frequently mentioned as supports to help case

managers and CPIs apply pre-service knowledge to

practice, in addition to support from management,

mentors, team leaders, and career development

coaches (CDCs), as well as trainings, resource

guides, and “how to” guides. When asked about

barriers to practice as a new case manager,

respondents said that having back-to-back cases while still learning the job was very stressful, in

addition to other challenges such as low salary, turnover, fast pace, caseload size, records management,

internal communication gaps, and lack of one-on-one supervision.

Family Functioning Self-Assessment. This section asks respondents to rate their level of ability to

understand and complete the FFA. The majority of respondents reported a high level of competence

with FFA completion. Respondents scored highest on their ability to understand all the FFA components

required for completion and lowest in their assessment of whether they have enough time to fully

complete the FFAs.

Organizational Climate and Culture Assessment This component of the evaluation is designed to examine perceptions of organizational culture and

climate among case managers and CPIs, as well as job turnover intentions. In terms of job satisfaction

and desired job characteristics, respondents ranked organizational climate and support, benefits, salary,

and opportunities for advancement as the most important. Respondents were least satisfied with

opportunities for advancement and salary and gave an overall job satisfaction rating of about 65%.

When asked about turnover intentions within the next year, most respondents (85%) believed it was

somewhat to very likely that they will leave their job to pursue another position within the field of child

welfare.

When aspects such as rigidity, proficiency, resistance, stress, engagement, functionality and morale

were assessed, there were mixed responses among respondents. A majority (79%) of respondents

agreed that they had the flexibility to make case decisions independently, but when asked if staff felt

their input was solicited and valued by leadership, results were split nearly half and half. All

respondents agreed that they are encouraged to be responsive to the unique needs of their clients but

were divided in how they felt about whether or not there was an emphasis on quality over quantity.

Results were also mixed on whether respondents felt they were encouraged to be creative in how to

best serve families, whether or not new ideas and innovation are supported by administrators, whether

staff felt that they were expected not to make waves within their organization, and whether their

organization was quick to adopt new EBPs and promising practices. Approximately 42 percent of

respondents felt that organizations were not addressing service barriers and developing new ways of

providing services. With regard to stress, a significant portion of respondents (85%) felt emotionally

drained with insufficient time to do their work, and 64 percent of respondents felt their workload was

unreasonable. In terms of family engagement, respondents agreed widely that they cared about their

clients and that they were able to make a difference in the lives of children and families through their

work. Regarding functionality, a majority of respondents also felt that they had a clear understanding of

their work responsibilities, that their co-workers collaborated to improve effectiveness of services, and

that they had a clear understanding of their organization’s mission and vision (79% across all domains).

A CBC that I wasn't assigned to help me

to stop over-thinking outcomes for a case

plan. I was really stuck in a rut and she

took the time to help me overcome that

hurdle. I don't have that problem

anymore with outcomes.


Only 57 percent of respondents felt that open communication was encouraged in their workplace.

Finally, responses were mixed on staff morale, with less than half of respondents agreeing that they

received fair compensation (43%), though a majority (79%) indicated that they liked their job and the

agency they worked for.

Training Initiatives Interviews To gain a better understanding of how differences in training initiatives across regions may contribute to

different patterns of knowledge transfer, retention, or employee support, the evaluation team

conducted semi-structured interviews with training staff from each region.

Benefits and Challenges. Respondents

reported that the major benefits to pre-

service training are that it gives a

standardized introduction to child welfare

to all new hires, it challenges pre-

conceived notions about issues related to

families in the system, and it combines

both knowledge and hands-on training.

Organizational Culture and

Climate

85% feel emotionally drained

71% believe oportunities for

advancement exist

57% perceive open communication in

workplace

85% intend to seek other positions in

child welfare in next year

79% perceive clear work responsiblities

42% feel barriers aren't being addressed

The theory of field days is good, that you should

learn and then practice. But reality is that it

doesn’t work. If you’re teaching them something

about family engagement and the next field day,

they’re transporting kids or going to court, it

doesn’t work the way they envisioned it to work.

There is a lot of lost opportunity.


The development of engagement skills was seen as an essential component of the training for both case

management and child protective investigation. With regard to challenges of pre-service training, one

consistent area of concern was that field days are very difficult to coordinate and that trainees often do

not observe in the field what they are learning in the classroom because the person they are shadowing

does not have the particular experience to offer them at the time of observation. There were general

concerns of translating knowledge from the classroom to the field. Other challenges included

redundancy of content, illogical flow of curriculum in some areas, and the contradiction of having a

substantial amount of content in some areas but lacking enough content on particular subjects like

FSFN.

Strengths and Gaps. One strength that was widely commented on was that the specialty tracks greatly

enhanced the curriculum, especially for case managers. Another strength consistently noted was that

the labs and field days really help give trainees hands-on practice that is difficult to obtain in the

classroom. With regard to curriculum gaps, there were differing opinions about whether the content of

the curriculum is appropriate for case managers’ needs; some suggested that topics should be more

comprehensive on safety assessments. Additionally, numerous respondents mentioned that the

curriculum is outdated in several areas, such as visuals and videos that do not work or can no longer be

accessed.

Ongoing Needs. Respondents recommended having a longer period of protected caseload, providing

consistent access to support staff dedicated to helping new employees, and more training on skill

development and professionalism. Training staff noted that the constant modification of policies and

procedures inhibited a streamlined training process and wished to see less change in this regard.

Feedback also indicated a need for different components of pre-service training to be better connected

and more support or training around documentation processes.

Summary Findings from this evaluation period highlight the following outcomes:

Knowledge increases are evident from CORE and Specialty Track components immediately after pre-service training.

Key engagement skills are observed with high frequency one month after unit start dates.

Self-assessments indicate high levels of competence with key competencies and FFA completion.

Case managers and CPIs report having many supports after training but wish to have had more hands-on, practical training during pre-service.

Respondents report high stress levels and a high rate of turnover intention as well as a high level of engagement with families and commitment to the child welfare field.

Training interviews indicate a need for more curriculum alignment with field days, more connection across different modules, and a longer period of protected caseloads.


Introduction

The current report provides an update on study activities conducted during the period of

February 1, 2018 through September 20, 2019. This study is a direct response to Florida Senate Bill 1666,

which has prioritized the evaluation of pre-service training for child protective investigators (CPIs) and

case managers. To accomplish this, the Florida Institute for Child Welfare (FICW) at Florida State

University has contracted with the Department of Child and Family Studies at the University of South

Florida (USF) to conduct a study examining the impact of child welfare pre-service training for newly

hired child welfare professionals in Florida. This study is designed to: 1) Assess the readiness of case

managers (CMs) and child protective investigators (CPIs) to begin their job responsibilities, 2) Determine

whether pre-service training is at the level it should be at, and 3) Identify both environmental factors

and individual coping strategies of workers that facilitate and hinder knowledge acquisition and skill

development while in the roles of case managers and child protective investigators. Thorough

evaluation of these efforts, as outlined in the evaluation plan, will help to highlight areas of strength as

well as opportunities for improvement, and it is also likely that findings will shed light on the

relationship between pre-service training and job readiness, performance, and satisfaction.

Site Selection

To determine the circuit of focus within each of the six regions, team members reviewed several

variables, including population size, population demographics, number of cases, retention and turnover

rates, and any factors that may lead to significant difficulty in obtaining data. Radey and Schelbe’s

(2017) study on job preparedness among Florida child welfare workers supports the idea that pre-

service training is an influential component of job satisfaction and may be a significant contributor to

high turnover rates. Because the intention of this study is to analyze the relationship between pre-

service training and experiences of new workers, the team took a purposive approach to sampling based

primarily on variances in turnover rates as reported in the Child Protective Investigator and Child

Protective Investigator Supervisor Educational Qualifications, Turnover, and Working Conditions Status

Report (DCF 2016) that analyzed retention and turnover rates among sixteen of the twenty circuits in

the state. The team organized the sample from highest turnover to lowest and divided at the median

into two groups. The two circuits with the highest and lowest turnover rates were chosen, and then two

additional circuits from each group of eight was chosen, giving consideration to the variables mentioned

above in order to ensure an inclusive and representative sample. For the pilot site, the evaluation team

selected a circuit based on ease of access and strength of working relationship with leadership within

the circuit.

Study Sites Description

There are nine study sites (the pilot site is not included in the nine study sites). These include six case

management organizations (CMOs), two DCF Circuits, and one Sheriff’s Office providing child protective

investigations. Five of six Regions are represented. The SW Region was excluded from study due to

changes in local leadership. The SE Region includes two sites. Two of the sites have combined CORE

training for case management and child protective investigation. The date range for study

implementation was 10/17/18 – 2/18/19, meaning all sites took part in their first evaluation activity

within this period. The class sizes for each site range from 6 to 14, considering that some classes have

case managers and CPIs combined for CORE. The evaluation team identified a key representative from


each site for overall study communications, as well as training representatives and supervisors for

specific communications regarding evaluation activities.

Pilot Study

The study team selected a CBC lead agency within the Central region for its pilot testing. This pilot site

was selected because of the agency’s relative stability, proximity to the team, and existing cooperative

relationships between the CBC and USF. The pilot concluded in January 2019 and the research team

modified evaluation activities based on lessons learned from the pilot.

Study Site Progress to Date

Evaluation activities began at each site when a new training cohort came on board in their local area,

thus each site is at a varying stage of the evaluation (see Table 1). All sites are on a different timeline for

the evaluation based on when they had a new cohort of staff to train. The evaluation team is monitoring

activities for each cohort over the course of one year. All study sites are in the process of completing

the six-month activities, which include an on-site focus group, a follow-up self-assessment, and a follow-

up specialty track knowledge assessment (two sites have already completed these activities). For an

overview of which evaluation activities have been completed by each site, see Tables 1 and 2. All study

sites have completed the pre- and post-test for the CORE knowledge assessment, specialty track

curricula pre- and post-tests, on-site observations, baseline self-assessments, and the workplace climate

and organizational culture survey.

Results from each of the study components completed to date are presented in this report.

Table 1: Evaluation Plan for Observation Activities and Self Assessments

Evaluation Component Timeframe Method

Evaluation introduction First day of CORE Web conference

CORE pre-assessment First day of CORE Electronic survey

CORE post-assessment Last day of CORE Electronic survey

Specialty track pre-assessment

First day of Specialty Track Electronic survey

Specialty track post-assessment

Last day of Specialty Track, and 3, 6, 9, and 12 months post-Specialty Track

Electronic survey

Observations 1 month after unit start date In-person

CM and CPI focus groups 6 and 12 months after unit start In-person

Self-assessments 1, 6, and 12 months after unit start date

Electronic survey

Culture/climate survey 3 and 12 months after unit start date Electronic survey

Training initiatives interview Within 1 month of evaluation start date

Phone conference


Table 2. Completed evaluation activities across sites: Months 1 - 3

Region Site Training Initiatives Interview

CORE

Pre

CORE

Post

Specialty Track Pre

Specialty Track Post

1-Month Observation

1-Monthv Self- Assessment

Culture/ Climate Baseline

C Pilot

NE

CBC

DCF

SC CBC

C CBC

NW CBC

SE 1

CBC

DCF

SE 2

CBC

Sheriff’s Office

Table 3. Completed evaluation activities across sites: Months 4 - 12

Region Site 6-Month

Focus Group

6-Month

Self-Assessment

6-Month

Specialty Track Post

12-Month Focus Group

12-Month

Self-Assessment

12-Month Specialty

Track Post

12-Month Culture/ Climate

C Pilot N/A N/A N/A N/A N/A N/A N/A

NE

CBC

DCF

SC CBC

C CBC

NW CBC

SE 1

CBC

DCF

SE 2

CBC

Sheriff’s Office


Evaluation Activities: Status and Results to Date

Knowledge Assessment Knowledge assessments were developed to examine the transfer of knowledge that transpired as a

result of the pre-service training curriculum. Content addressed in the knowledge assessments derived

from the CORE pre-service curriculum as well as the training curriculum for the two specialty tracks for

case managers and child protection investigators. Corresponding knowledge assessments were

administered immediately prior to participation in the respective training curriculum and following the

training.

CORE Knowledge Assessment In order to examine the transfer of knowledge that transpires as a result of the pre-service training

curriculum, a Knowledge Assessment was developed based on the content addressed in the CORE

component of the pre-service curriculum. Supplementary assessments were also developed to measure

knowledge gained from trainees’ participation in the specialty track curriculum that follows the CORE

curriculum, and these results will be reported on once all sites have completed the first round of

assessments. The Knowledge Assessment developed for the CORE curriculum was informed by the

following modules: The Practice Module; Child Development; Trauma and the Child; Family Conditions;

Understanding Child Maltreatment; Assessing and Analyzing Family Functioning; Safety and Risk; Safety

Planning; and the CORE Readiness Assessment. Although the curriculum includes two other modules on

Florida’s Child Protection System and another that assesses CORE readiness, these were not included in

constructing the Knowledge Assessment.

Pre-service trainees completed this assessment immediately prior to participation in the CORE training

curriculum and following the commencement of the CORE curriculum. At the time of this report, each of

the sites being evaluated completed the pre- and post-Knowledge Assessment for the pre-service CORE

training curriculum.

Results of the Knowledge Assessment for the CORE curriculum are organized according to the modules

assessed. Findings include data from 91 pre-service trainees across sites that completed the pre- and

post-assessment. Paired t-test analyses for each module are shown in Appendix A. However, for easier

interpretation across modules, Table 4 presents the percentage of questions trainees answered

correctly at pre-test and post-test for each module. Overall, there was significant improvement in items

correctly answered at the post-training assessment compared to the pre-test. Trainees answered 70.5

percent of questions correctly at pre-test and 82 percent of questions correctly at post-test. Further, the

percentage of questions answered correctly at post-test was greater than the pre-test for each module

assessed. Significant increases in correct responses were observed with five modules:

1. The Practice Module

2. Understanding Child Maltreatment

3. Assessing and Analyzing Family Functioning

4. Safety and Risk

5. Safety Planning

The greatest proportion of correct responses following the CORE pre-service training was observed for

Understanding Child Maltreatment (88%), Safety and Risk (87%), and Assessing and Analyzing Family


Functioning (86.6%). The increase from pre-test to post-test was greatest for the Safety and Risk module

and the Assessing and Analyzing Family Functioning module.

Table 4. Proportion of Correctly Answered Core Curriculum Knowledge Assessment Questions Pre-Test

(n = 91)

Post-Test (n = 91) Paired t-tests

The Practice Module (4 items) 64.3% 77.8% ***

Child Development (3 items) 81.3% 84.3% ns

Trauma and the Child (3 items) 70.0% 74.3% ns

Family Conditions (4 items) 85.5% 88.0% ns

Understanding Child Maltreatment (5 items)

58.2% 73.4% ***

Assessing and Analyzing Family Functioning (6 items)

72.0% 86.7% ***

Safety and Risk (5 items) 70.6% 87.0% ***

Safety Planning (5 items) 68.2% 82.2% ***

Overall (35 items) 70.5% 82.0% ***

Taken together, findings from the Knowledge Assessment provides some evidence that knowledge was

gained as a result of the CORE curriculum training. Trainees correctly answered 70.5 percent of

questions at pre-test compared to 82 percent of questions answered correctly at post-test. Similarly, a

greater percentage of items were correctly answered at post-test than at pre-test. Data are also being

collected on knowledge gained as a result of the specialty track pre-service trainings for CPIs and case

managers. Future analysis of these Knowledge Assessments will help provide a more complete

examination of the extent to which the pre-service training curriculum has contributed to the transfer of

knowledge and skills among this group.

Specialty Tracks In addition to the Knowledge Assessment specific to the CORE training curriculum, two additional

assessments were developed specific to the Case Management (CM) Specialty Track and the Child

Protection Investigator (CPI) Specialty Track. Table 5 outlines modules from the specialty track curricula

that were included in the Knowledge Assessments.

Table 5. Specialty Track modules assessed

CM Specialty Track Modules CPI Specialty Track Modules

Introduction to Case Management Introduction to Child Protection Investigations

Case Transfer Assessment of Hotline to Assignments

Safety Management Commencement of the Investigation: Initial Contact and Present Danger

Out-Of-Home Care Present Danger Assessment


CM Specialty Track Modules CPI Specialty Track Modules

Family Engagement Standards: Preparation and Introduction

The FFA- Investigation and Safety Planning

Family Engagement Standards: Exploration Developing In-Home or Out-Of-Home Safety Plan

Family Engagement Standards: Case Planning

Evaluating Family Progress

Results of the Knowledge Assessments for the specialty track curricula are organized according to the

modules assessed. Findings include data from 53 CM trainees and 34 CPI trainees across sites that

completed the pre- and post-assessment. Paired t-test analyses for each module were used to assess

for change in pre- and post-assessment scores for each trainee. However, for easier interpretation

across modules, reported findings detail the percentage of questions trainees answered correctly at pre-

test and post-test for each module.

Case Manager Specialty Track. Of the 59 trainees who completed the CM Specialty Track training pre-

Knowledge Assessment, 89 percent (n = 53) also completed the post-Knowledge Assessment

(see Table 6). Overall, there was significant improvement in items correctly answered at the post-

training CM Specialty Track assessment compared to the pre-test. Trainees answered 74.7 percent of

questions correctly at pre-test and 82.6 percent of questions correctly at post-test. There was an

increase in correct responses for each module assessed. Significant increases, however, were observed

with four of the eight modules: Case Transfer, Safety Management, Family Engagement Standards:

Preparation and Introduction, and Family Engagement Standards: Case Planning. The greatest

proportion of correct responses following the CM Specialty Track training was observed for Evaluating

Family Progress (88.1%), Family Engagement Standards: Exploration (87.6%), and Case Transfer (84.9%).

From pre-test to post-test, the greatest increase was observed for two of the Family Engagement

Standards modules, specifically, Case Planning, and Preparation and Introduction.

Table 6. Average module scores for the CM Specialty Track Knowledge Assessment

Pre-Test (n = 53)

Post-Test (n = 53)

Paired t-tests

Introduction to Case Management (1 item) 67.9% 75.5% ns

Case Transfer (5 items) 75.9% 84.9% *

Safety Management (6 items) 77.7% 83.3% *

Out-Of-Home Care (7 items) 72.0% 75.7% ns

Family Engagement Standards: Preparation and Introduction (5 items)

69.1% 79.6% **

Family Engagement Standards: Exploration (3 items) 73.6% 75.5% ns

Family Engagement Standards: Case Planning

(7 items)

72.0% 87.6% ***

Evaluating Family Progress (7 items) 82.2% 88.1% ns

Overall (41 items) 74.7% 82.6% *** Note: *p < .05; **p < .01 ***p < .001; ns = not significant


CPI Specialty Track. Forty trainees completed the CPI Specialty Track training and 85 percent of

trainees (n = 34) completed both the pre- and post- Knowledge Assessments. Table 7 shows the

proportion of correct responses for modules examined from this curriculum. Similar to results of the

Case Manager Specialty Track assessments, overall, there was significant improvement in items correctly

answered at the post-training CPI Specialty Track assessment compared to the pre-test. Trainees

answered 74.8 percent of questions correctly at pre-test and 86.9 percent of questions correctly at post-

test. A greater number of items were answered correctly following the training for each module

assessed. Significant increases were observed for all modules with the exception of the Assessment of

Hotline to Assignments module. Although a significant increase was not observed with this module,

almost 80 percent of trainees correctly answered these questions at pre-test. Following the CPI

Specialty Track training, modules with the greatest percentage of correct responses was observed for

Commencement of the Investigation: Initial Contact and Present Danger (100%), Introduction to Child

Protection Investigations (94.1%), and Present Danger Assessment (93.1%). From pre-test to post-test,

the greatest increase was observed for material assessed in the Introduction to Child Protection

Investigations modules and the Present Danger Assessment module.

Table 7. Average module scores for the CPI Specialty Track Knowledge Assessment

Pre-Test (n = 34)

Post-Test (n = 34)

Paired t-tests

Introduction to Child Protection Investigations (1 item) 73.5% 94.1% *

Assessment of Hotline to Assignments (2 items) 79.4% 85.3% ns

Commencement of the Investigation: Initial Contact and Present Danger (3 items)

95.1% 100% *

Present Danger Assessment (6 items) 76.0% 93.1% ***

The FFA- Investigation and Safety Planning (5 items) 82.9% 88.2% *

Developing In-Home or Out-Of-Home Safety Plan (7 items) 58.0% 74.4% ***

Overall (24 items) 74.8% 86.9% ***

Note: *p < .05; **p < .01 ***p < .001; ns = not significant

Taken together, findings from knowledge assessments administered to pre-service trainees provide

evidence that knowledge was gained as a result of the CORE curriculum training as well as the Specialty

Track training. Overall, trainees correctly answered significantly more questions at post-test compared

to the pre-test for each of the three assessments. Further, a greater percentage of items were correctly

answered at post-test than at pre-test for each module examined within curricula.

A final strategy to comprehensively examine the transfer of knowledge as a result of the pre-service

training curriculum is to re-administer the specialty track assessments after trainees have been in the

field for at least six and at least 12 months, respectively. This will provide evidence of the extent to

which material learned in pre-service training was retained by case managers and child protection

investigators.


Observational Checklist and Self-Report Survey The purpose of the observation component is to assess the ability of front-line staff to implement in

practice the critical skills taught in pre-service. An observation tool was developed using the Florida

Child Welfare Competencies as a guide, focusing most concretely on skills related to family engagement,

risk and safety assessment, and identification of strengths and needs (see Appendices B and C for the

observation checklists). The observation checklist includes 13 items for case managers and 11 for CPIs.

Each item corresponds with a particular competency and one or more objectives from the CORE pre-

service curriculum (see Appendix D for a key matching these items). Instructions for the observer are to

indicate whether a skill was demonstrated by marking “Yes” or “No,” or to mark “N/A” if there was not

an opportunity for the skill to be observed. The observer was asked to include an example for each of

the skills demonstrated and to remark on any other factors that were important to note for the

observation.

The areas of assessment focus largely on interpersonal skills with clients, including cultural competency,

communication, interviewing, engagement, and strengths-based approaches, as well as knowledge of

services and the general child welfare system. To determine these areas, the child welfare competencies

were reviewed for the following criteria: 1) The skill or competency must have been learned during pre-

service training, 2) The skill or competency must be clearly observable during a home visit or discussion

in a staffing, and 3) The skill or competency must be of importance to stakeholders (e.g., families

involved in the child welfare system, DCF, FICW). The team’s knowledge of these criteria is based on

previous experience working in child welfare, partnering with DCF and FICW, and a current literature

review. For instance, several studies emphasize parent and caregiver engagement as key to successful

outcomes (Altman 2008; Chapman, et al., 2003; Kemp, et al., 2009; Ortega and Faller 2011), and

therefore, components such as interviewing and communication skills were emphasized in the checklist.

The self-assessment survey is an electronic survey that will be sent to designated workers at five

intervals, including baseline (within one month of training completion, 3 months post-training, 6 months

post-training, 9 months post-training, and 12 months post-training (see Appendix E). This strategy will

allow the evaluation team to observe changes in caseworker skills over time. The survey includes three

primary domains: 1) A section on Family Functioning Assessment (FFA) competency which asks the

caseworker to rate their ability to understand and complete the FFA; 2) A practice assessment that is

modeled after the observation checklist described above; and 3) A transfer-of-learning section that

includes open-ended questions to better understand caseworkers’ experiences with pre-service training,

on-the-job training, and other supports or barriers that have helped or hindered their ability to

implement skills.

Observation and Self-Assessment Results A member of the evaluation team coordinated with a point of contact at each site to schedule

observations with as many members of the training cohort as possible. Observations took place with a

total of 37 participants across the five study site regions, which included six CMOs, two DCF circuits, and

one Sheriff’s Office. On-site observations were conducted between January and June 2019 at

approximately one month after case managers and CPIs were assigned to their units. Multiple types of

interactions were observed, including home visits, court hearings, and staffing meetings (see Table 8).

For case managers, the evaluation team observed 22 home visits, two court hearings, and one

permanency staffing. A focus group was also conducted with case managers during one site visit when

scheduling individual observations when local management did not feel that parents would be


comfortable with an additional person at the home visit. For CPIs, the evaluation team observed six

home visits and one transfer staffing.

Table 8. Number of observations by type and position

Site Visit Type Case Managers CPIs Total

Home Visit 22 6 28

Court Hearing 2 -- 2

Permanency Staffing 1 -- 1

Transfer Staffing -- 1 1

Total 25 7 32

Case Manager Observations

Results from the case manager observations are provided in Table 9 below. Scores indicate that case

managers were demonstrating desired skills the vast majority of the time (55%) and were not

demonstrating skills in only nine instances, or 2 percent of the time. However, it is important to note

that interactions were not applicable or observable 42 percent of the time. Most often, this was

because the purpose of the home visit or other interaction did not warrant demonstration of the

particular skill. The most frequently demonstrated skills were related to utilizing interviewing

techniques and strategies, engaging and assessing families from a strengths-based perspective, and

discussing progress in recommended services. Examples of these skills include the following: “Case

manager asked open-ended questions, repeated information back, and had a conversation – not just

note taking,” and “Case manager was strengths-based with the youth, very encouraging and pointed out

many areas where the youth is doing well.” Examples of instances in which case managers

demonstrated lack of skills include not being very engaged with the caregiver, only being strengths-

based with some family members or caregivers, and not discussing the safety plan during the visit.

Table 9. Case manager observation skill counts

Item Yes No N/A

Demonstrates sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.).

12 0 13

Communicates with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs.

20 1 4

Interacts with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a non-confrontational manner.

6 0 19

Utilizes interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family.

22 2 1


Item Yes No N/A

Interprets the results of the Department’s Family Functioning Assessment to make appropriate child safety determinations.

1 2 22

Conducts or has conducted for this case on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

14 0 11

Engages and assesses families from a strengths-based perspective. 22 0 3

Develops and implements (or has developed and implemented) a case plan/safety plan based on strengths-based assessment.

11 1 13

Identifies and facilitates referrals to appropriate services based on the Family Functioning Assessment.

17 0 8

Discusses progress in recommended services. 24 1 0

Demonstrates active and ongoing assessment of the safety management plan. 6 2 17

Demonstrates use of safety analysis and planning criteria needed to modify safety plans. 5 0 19

Demonstrates knowledge and application of Florida's Child Welfare System Guiding Principles.

19 0 6

Total 179 9 136

% 55 3 42

CPI Observations.

Scores from the CPI Observations are provided in Table 10. Skills were observable 61 percent of the

time. Out of observable skills, CPIs demonstrated proficiency in desired skills 83 percent of the time and

showed lack of skill demonstration 17 percent of the time. The areas where skills were most frequently

demonstrated include engaging and assessing families from a strengths-based perspective, identifying

and facilitating referrals to appropriate services based on the FFA, and demonstrating knowledge and

application of Florida’s Child Welfare System Guiding Principles. Examples of these skills include: “CPI

was very strengths-based and acknowledged positive steps Mom has taken already,” “Discussed

appropriate referrals and available services,” and “Discussed child safety and goal to keep family

together.” For instances in which observers did not see sufficient skill being demonstrated, some of the

reasons were that the CPI was focused on completing a form and missed some opportunities to delve

deeper, the CPI used a document to guide the interview and did not ask a lot of probing questions, or

not accommodating a family that spoke another language, relying instead on the family member being

investigated for translation.


Table 10. CPI observation skill counts

Item Yes No N/A


4 1 2


3 1 3


4 1 2


3 0 4


2 0 5

Conducts or has conducted for this case on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

2 1 4

Engages and assesses families from a strengths-based perspective. 5 1 1


3 1 3


5 1 1

Demonstrates knowledge and application of Florida's Child Welfare System Guiding Principles.

6 1 0

Communicates clearly the information collected during the investigation that has been used to determine the need for case management.

2 0 5

Total 39 8 30

% 51 10 39

Self-Assessment Results After evaluation team members conducted observations, all case managers and CPIs from the study

sites received an electronic link to a self-assessment survey, even if they did not participate in an

observation. They were asked to complete the survey within three weeks, with two one-week reminder

emails. The items on the self-assessment survey mirror the items on the observation checklist, with the

addition of one section on Family Functioning Assessment competency that includes five Likert-scale

questions, and one section on Transfer of Learning, which includes five open-ended questions that

assess supports, barriers, and other feedback on how knowledge and skills are applied after training


(see Appendix E). Case managers and CPIs were asked to reflect on their most recent family visit when

answering questions. The survey was sent to 62 participants in the study, and a total of 21 respondents

completed the survey (15 case managers and 6 CPIs), resulting in a completion rate of 34 percent.

Results are reported by each section below.

Family Functioning Assessment The FFA-Investigations is the process by which investigators apply critical thinking skills to guide

decision-making regarding child safety and risk, based upon having an extensive and comprehensive

knowledge of the individual and family conditions in the home. This information is summarized in six

information domains and is essential to the investigator being able to accurately identify impending

danger threats, assess the sufficiency of caregiver protective capacities, complete a safety analysis and

implement a safety plan (as appropriate) and determine the risk for future maltreatment to the

child(ren). At the time of case transfer for services, the case manager is responsible for completing the

FFA-Ongoing. The FFA-Ongoing captures an understanding of the family over the life of the child

welfare case to support accurate analysis and decision making.

The purpose for reviewing completed FFA’s, both Investigation and Ongoing, is to evaluate knowledge

transfer of assessment and documentation skills needed to complete the six information domains of the

FFA. USF initially created FFA review protocols, however during conversations with DCF, it was revealed

that Action for Child Protection (ACTION) is currently conducting comprehensive reviews of FFA’s

statewide and reporting on data collected. ACTION has shared with USF their current activities as well

as the FFA tool they are using to collect by quantitative and qualitative data. USF has begun a review of

ACTION’s tool to determine which data points align with USF’s evaluation plan. The study team will have

follow-up discussions with ACTION to determine how ACTION can integrate a random sample of cases of

newly trained investigators and case managers within the selected circuits. USF will work with ACTION

to utilize their data and minimize the impact on the child welfare workforce.

Results

The Family Functioning Assessment (FFA) section asks respondents to rate their level of ability to

understand and complete the FFA based on a four-point Likert scale (where 4 = Strongly Agree and

1 = Strongly Disagree). A summary of mean scores for all respondents is in Table 11 below.

Table 11. Mean scores for Family Functioning Assessment section

Item Mean Score

I understand all of the components I am required to complete in the Family Functioning Assessment.

3.24

I can provide sufficient descriptive information when completing the Family Functioning Assessment.

3.33

I am confident in my ability to rate the functioning level of children. 3.10

I am confident in my ability to rate the caregiving capacity of adults. 3.10

I have enough time to fully complete the Family Functioning Assessment. 2.81


The majority of respondents agreed or strongly agreed with each item, demonstrating a high level of

self-reported competence with FFA completion. Respondents scored highest on their ability to

understand all of the FFA components required for completion and lowest in their assessment of

whether they have enough time to fully complete the FFAs; 36 percent of respondents disagreed or

strongly disagreed with this statement.

Practice Competency: Case Managers

The Practice Competency section mirrors the domains assessed during observations. The practice areas

included in this section are derived from the pre-service curriculum and the Florida Practice Model.

Case managers were asked to mark whether they performed each skill at their most recent family visit.

Table 12 below shows the frequency with which each response was marked.

Table 12. Self-Assessment skill response counts for case managers

Item Yes No N/A

I demonstrated sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.).

11 0 4

I communicated with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs.

15 0 0

I interacted with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a non-confrontational manner.

15 0 0

I utilized interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family.

15 0 0

I interpreted the results of the Department’s

Family Functioning Assessment to make appropriate child safety determinations. 13 0 2

I conducted or have conducted for this case on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

10 2 3

I engaged and assessed families from a strengths-based perspective. 14 0 1

I developed and implemented (or have developed and implemented) a case plan/safety plan based on strengths-based assessment.

9 4 2

I identified and facilitated referrals to appropriate services based on the Family Functioning Assessment.

14 1 0

I discussed progress in recommended services. 12 2 1

I demonstrated active and ongoing assessment of the safety management plan. 13 0 2

I demonstrated use of safety analysis and planning criteria needed to modify safety plans. 11 0 4


Item Yes No N/A

I demonstrated knowledge and application of Florida's Child Welfare System Guiding Principles.

13 1 1

Total 165 10 20

% 85 5 10

The responses indicate that case managers perceived that they were demonstrating the majority of skills

listed during their most recent family visit, particularly with regard to respectful communication.

Several respondents provided examples of ways they demonstrated these skills. For instance, when

describing how they interacted in a non-confrontational manner, one case manager said, “When

addressing the allegations with the mother, I remained mindful of my facial expressions, tone, and body

language. I also allowed for her to explain her side before asking questions.” Another respondent noted

ways they engage with and assess families from a strengths-based perspective: “I always ask the clients

what do they enjoy doing and what are their strengths. If they struggle, I point out accomplishments

such as keeping a job, maintaining a clean home, etc.” For instances in which case managers felt the skill

could not be assessed, they marked “N/A,” which accounted for 10 percent of the responses. The most

common item for which respondents said they did not demonstrate the skill was on developing and

implementing a case plan/safety plan based on a strengths-based assessment. There were no

comments to explain the responses. The higher number of “Yes” responses overall compared to the

observers’ marks is likely due to the greater knowledge case managers have of their own cases than

what evaluation team members were able to observe.

Transfer of Learning: Case Managers This section includes five open-ended questions that gauge the transfer of learning from pre-service to

practice. When asked what areas of pre-service training were most helpful in preparing them for work,

respondents gave the following responses: techniques for interviewing and relating to different clients,

discussions on policy and “real-life” cases, FSFN training, explanations of explanation the process from

FFI to FFA to case plan to judicial review to TPR/ Permanency, and shadowing and field observations.

These responses highlight the value of hands-on components of pre-service training, which is consistent

with the one-month self-assessments. When asked about the least helpful aspects of pre-service

training, some respondents indicated that more time should be spent on FSFN training (at least five

days), car seat practice, and the specifics of case management; some said that more training on specific

forms or processes was needed, like what to include on the POS form or how to make referrals to

specialists; and one suggested that trainees need more information on all of the services offered

throughout their county.

Respondents were asked about the types of resources and supports available to them to help them

apply pre-service learning to everyday work. Supervisor and peer support were frequently mentioned,

as well as support from other staff, such as management, mentors, team leaders, and career

development coaches (CDCs). One respondent noted, “A CDC that I wasn't assigned to help me to stop

over-thinking outcomes for a case plan. I was really stuck in a rut and she took the time to help me

overcome that hurdle. I don't have that problem anymore with outcomes.” Other responses pointed to

ongoing trainings and resource guides as supports during the first year on the job.


When asked about barriers to practice as a new case manager, respondents said that having back to

back cases while still learning the job was very stressful, in addition to widespread challenges such as

low salary, turnover, caseload size, records management, internal communication gaps, and lack of one-

on-one supervision. One respondent described ongoing challenges with learning processes: “Not

knowing exactly what to do, how to do it, or when to do it can be a barrier. Having multiple cases with

different requirements can be confusing for someone just getting started.” And finally, some

respondents mentioned “pushback” or lack of cooperation from families as a barrier.

Practice Competency: CPIs Response counts for the practice competency section from the seven CPIs who completed the survey

are in Table 13.

Table 13. Self-Assessment skill response counts for CPIs

Item Yes No N/A

I demonstrated sensitivity to cultural differences and ethnicity among clients (e.g., respecting communication style, acknowledging value system, discussing culturally specific services, etc.).

4 0 2

I communicated with family members and collaterals by asking parents or guardians to identify, discuss, and prioritize concerns and needs.

5 0 0

I interacted with persons allegedly responsible for maltreatment in a way that allows information to be communicated in a non-confrontational manner.

5 0 1

I utilized interviewing techniques and strategies (e.g. demonstrating respect, active listening, using exploring and focusing skills, etc.) to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family.

6 0 0

I interpreted the results of the Department’s

Family Functioning Assessment to make appropriate child safety determinations. 5 0 1

I conducted or have conducted for this case on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

3 1 2

I engaged and assessed families from a strengths-based perspective. 5 0 1

I developed and implemented (or have developed and implemented) a case plan/safety plan based on strengths-based assessment.

3 1 2

I identified and facilitated referrals to appropriate services based on the Family Functioning Assessment.

6 0 0

I discussed progress in recommended services. 4 1 1

I demonstrated active and ongoing assessment of the safety management plan. 3 1 2

I demonstrated use of safety analysis and planning criteria needed to modify safety plans. 3 1 2


Item Yes No N/A

I demonstrated knowledge and application of Florida's Child Welfare System Guiding Principles.

6 0 0

Total 58 5 14

% 75 6 18

CPI respondents indicated with high frequency that they demonstrated most of the skills in the

assessment during their last family visit. Skill areas with the highest marks include utilizing interviewing

techniques and strategies, identifying and facilitating referrals to appropriate services, and

demonstrating knowledge and application of Florida’s Child Welfare System Guiding Principles. Some

examples were provided for context. For demonstrating cultural competency, one respondent noted,

“Being an African American CPI working with clients from a diverse population, I look at parenting

differently from how I was raised than some of the clients I work with without judgement.” For

communicating with family members to identify, discuss, and prioritize concerns and needs, one

respondent did this by “asking parents what do they love about being a parent and what are their

struggles as a parent,” and another by “asking the family what is it that I can do to better assist the

family.” Respondents felt that the skill area was not applicable to their interactions 18 percent of the

time. All but one of the “No” responses came from one individual, indicating that the majority of CPIs

who took part in the survey are confident in their abilities to carry out necessary skills.

Transfer of Learning: CPIs Most responses indicated that pre-service training was helpful to the CPI’s job, particularly elements

such as role play, the breakdown of Department policies, and creative ways of quizzing. However, one

respondent disagreed, commenting, “I do not feel as if the pre-service helped me for this current

position. Once on the floor, everything was different.” In terms of the least-helpful aspects of pre-

service training, some respondents agreed that there should be less “book work” and more hands-on

activities and field experiences. One CPI said that the time spent learning how to write FFAs and PDAs

was ineffective because there was no feedback given on them, and because the processes were

different once in the field.

In terms of supports on the job, peer and supervisor support were indicated as helpful during the

transition from trainee to CPI, in addition to receiving files with sample “how-to” forms to use as a

guide. Barriers discussed included being given large case load sizes at a fast pace. One respondent

commented, “Every case is different, some cases need more work than others. There is so much work to

be done, not enough time to complete.” In addition to suggestions previously mentioned, CPI

respondents said that improvements to pre-service training could be made by doing more interviewing

practice, having more hands-on field experience, and communicating better with agencies on what new

employees will be doing once they start.

All sites have continued to receive six-month electronic self-assessments, as initially outlined in the

evaluation plan. This self-assessment is designed to capture case manager and CPI feedback on their

perceptions of how well they have mastered skill sets related to key competencies based largely on

client engagement. They are also asked to give feedback on their competency in conducting Family


Functioning Assessments and whether there are other trainings or supports that have helped them or

would help in mastering the skills being measured and feeling confident in their roles. Four sites have

completed the self-assessments, and the remaining five will be completed by the beginning of

November 2019.

Organizational Climate and Workplace Culture Assessment One of the goals of this evaluation is to identify factors that may impact turnover among case managers

and CPIs. Research suggests that organizational culture and climate is associated with employee’s work

attitudes and have found that these attitudes impact job turnover intentions (Saeed, Waseem, Sikander,

& Rizwan, 2014). As such, this evaluation component is designed to examine perceptions of

organizational culture and climate among case managers and CPIs, and whether factors related to the

workplace environment affect job turnover intentions. Domains pertaining to organizational culture

include, for example, role conflict, role overload, depersonalization, and emotional exhaustion.

Organization climate includes domains such as rigidity, proficiency, engagement, and functionality. The

survey is sent via Qualtrics, a web-based survey platform, at approximately 3 and 12 months post pre-

service training.

This aspect of the child welfare pre-service evaluation is designed to examine perceptions of

organizational culture and climate among case managers and CPIs, as well as the job turnover intentions

of case managers and CPIs. A survey developed by Glisson and colleagues was adapted for use in this

project in order to assess the climate and culture of each localized child welfare system (Glisson, 2002;

Glisson & Hemmelgarn, 1998; Glisson & James 2002). Specifically, subscales of the Children’s Services

Survey measuring organizational climate and culture were adapted for use herein. Domains pertaining

to organizational culture include, for example, role conflict, role overload, depersonalization, and

emotional exhaustion. Domains related to organizational climate include rigidity, proficiency,

engagement, and functionality. Domain labels were omitted from the survey so that the negative

wording (e.g., work overload and rigidity) did not impact participants responses. These scales have

sound psychometric properties and a number of studies in varied workplace environments found the

scales to be reliable.

The Organizational Climate and Culture Assessment was administered to cohorts of case managers and

child protective investigators who had completed pre-service training and been in the field for 3 months.

The survey will be administered again as workers reach 12 months in the field. The survey was sent to

each participant via Qualtrics, a web-based survey platform. The current report includes the 3 months

post-training data collection point for all sites. Although the survey was sent to participants at all sites, a

somewhat small sample size (14) occurred due to worker turnover (approximately half of newly trained

workers across all sites were reported to have left their positions by the three-month mark) and likely

the demands on the remaining workers. While repeated attempts were made to send reminders to

complete the survey, newer workers may have generally prioritized their case work before a survey,

which is understandable given their high workloads.

Demographics and Education All but one of the survey respondents were female. Respondents ranged in age from 22 to 51 years.

The average age of respondents was 31 years. As can be seen in Figure One, forty three percent of

respondents identified themselves as Caucasian and half of Caucasian respondents identified as having

Hispanic ethnicity.


Figure 1. Race and ethnicity of organizational culture and climate survey respondents

Another 43 percent of respondents identified themselves as African American and 17 percent of African

American respondents identified themselves as having Hispanic ethnicity. Seven percent of respondents

identified as Asian and the remaining seven percent marked Other for Race and identified themselves as

having Hispanic ethnicity. The majority of survey respondents had a bachelor’s degree, with 21 percent

reporting a master’s degree or higher. Fields of study varied within the social sciences but generally

included social work, psychology and criminal justice. Sixty-four percent of respondents felt that their

current work was closely related to their highest degree, with another 21 percent answering that it was

somewhat related. Of the two respondents who did not feel their current work was related to their

degree, one was outside the area of social science and one had graduated from a social work program.

Importance of Job Characteristics and Job Satisfaction Respondents were asked when thinking about a job, how important several factors were to them. On a

scale of one to four points, the most important job factor listed was organizational climate and support

(3.92), followed by benefits (3.84), salary (3.76) and opportunities for advancement (3.76). Job security

(3.61) and contribution to society (3.61) were also listed as important. Level of responsibility (3.46) and

degree of independence (3.46) were still important to respondents but less so than other characteristics

and intellectual challenge was rated lowest (3.23).

Next, respondents were asked to rate how satisfied they were with the above characteristics specific to

their current job. In Figure 2, these satisfaction levels are compared to responses on the importance of

job characteristics described above. Survey respondents were most satisfied with the contribution their

work was making to society (3.38) and the degree of independence they had in their daily work (3.31),

followed by the intellectual challenge of the position (3.15) and the level of responsibility they had been

Caucasian -Not Hispanic

21.5%

Caucasian -Hispanic

21.5%

African American - Not Hispanic

26%

African American - Hispanic

17%

Asian7%

Other - Hispanic7%


given (3.08). Respondents were somewhat satisfied with their level of job security (3.00) while

respondents were somewhat dissatisfied with benefits (2.85) and organizational climate and supports

(2.85). Respondents were least satisfied with opportunities for advancement (2.77) and salary (2.23).

Interestingly, respondents rated their overall job satisfaction at an average score of 2.61, indicating that,

when taken as a whole, income level and opportunities for promotion may influence a worker’s overall

perception of or value in the work they do. Additionally, Figure 2 shows that salary, organizational

culture/climate and benefits were most disproportionate in perceived importance and satisfaction.

Figure 2. Perceived Importance Versus Satisfaction with Job Characteristics

Turnover Intentions Respondents were asked in the next year how likely it is that they will leave their job to pursue another

position within the field of child welfare. The majority of respondents (85%) believed it was somewhat

to very likely. Respondents were also asked how likely it was they would leave their current jobs to

accept work in another field other than child welfare. In this case, 77 percent of respondents believed it

was somewhat to very likely they would leave child welfare to pursue another vocation. Although these

findings should be interpreted within the context of a somewhat small sample size, they do show that

most individuals in the position of a case manager or CPI desired to leave their position for career

advancement, within or outside of child welfare. This may be reflective of the challenges front line

workers face and/or the perception that front-line work is not a long-term job. It is also interesting to

note that a limitation of the survey sample is that those having already left their positions were not

contacted to complete the survey.

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Importance Satisfaction


Organizational Culture and Climate Characteristics The specific survey domains we used to assess organizational culture and climate were rigidity,

proficiency, resistance, stress, engagement functionality and morale. This section explains each domain

and includes any trends that were apparent from the data.

Job Flexibility. Rigidity refers to things like how much flexibility and discretion an employee feels

in their work, whether or not they feel they have input into the decision-making processes in their work

environment, and the extent to which rules and bureaucratic processes govern an organization versus

being able to tailor case decisions to individual family need. This last trait is often talked about in terms

of centralization versus formalization within a work environment. In a highly centralized environment,

decision making authority would be concentrated in the hands of a few people versus diffused authority

at different levels of an organization. Formalization has to do with how many policies and procedures

govern employee behavior. First, the majority (79%) of respondents agreed that they had the flexibility

to make case decisions independently, but the intensity of that agreement varied (Somewhat Agree

36%; Agree 36%; Strongly Agree 7%). When asked if they felt they could make case specific decisions or

if all cases had to be handled in the same way, there was a trend toward making case-based decisions

that matched family need represented by a somewhat to strong agreement rating for 71 percent of

respondents. When asked if staff felt their input was solicited and valued by leadership, results were

split down the middle with 43 percent agreeing their input was valued, 43 percent feeling their input

was not valued, and 14 percent not yet sure how they felt.

Proficiency. The second domain included in the organizational culture and climate survey is

proficiency. Proficiency within an organization encourages norms and behaviors that place families and

quality of care first above timelines and numbers served. In child serving organizations it is often

associated with higher levels of professionalism, being responsive to clients, implementation of EBPs,

maintenance of updated treatment models, and improved outcomes for youth served. First, all

respondents agreed that they are encouraged to be responsive to the unique needs of their clients (36%

Strongly Agree; 50% Agree; 14% Somewhat Agree). Respondent agreement was also strong when asked

if they were expected to apply up to date knowledge of EBPs and best practices (36% Strongly Agreed;

57% Agreed; 7% Somewhat Agreed). Respondents were also asked whether they felt that emphasis was

placed on quality of the services they provided versus meeting certain quotas or timelines. Respondents

were split in how they felt about whether or not there was an emphasis on quality over quantity.

Specifically, 43 percent agreed that there was a focus on quality, while 43 percent disagreed that quality

was valued over quotas, and 14 percent weren’t sure how they felt yet. Finally, respondents were also

asked about whether they were encouraged to be creative in how to best serve families. Results were

again mixed; 64 percent agreed, 21 percent disagreed, and 14 percent were not sure yet whether or not

creativity in service provision was encouraged.

Resistance. The third domain included in the organizational culture and climate survey is

resistance. Organizations that score higher on resistance are those that are not as open to changing

their service delivery or case practice models. Resistant organizations often encourage apathy or

passivity toward change and suppress behaviors to introduce new practices and innovative service

approaches. Behaviors might range from a simple inactivity toward change to direct opposition and

criticism of individuals promoting change. Specific to the survey, respondents were mixed regarding

whether or not new ideas and innovation are supported by administrators (50% Agreed; 43% Disagreed;

7% Neutral). Results were also mixed regarding whether staff felt that they were expected not to make


waves within their organization (28% Agreed; 21% Disagreed; 51% Neutral). Results were again mixed

when staff were asked whether their organization was quick to adopt new EBPs and promising practices

(35% Agreed; 35% Disagreed; 29% Neutral). Finally, results were somewhat negative regarding whether

or not organizations were perceived not to be addressing service barriers and developing new ways of

providing services (42% Agreed; 28% Disagreed; 29% were Neutral).

Job Stress. The fourth domain included in the organizational culture and climate survey is

stress. A stressful work climate is one in which workers feel emotional stress and burnout either due to

role conflict, competing demands, insufficient resources, and/or not enough time to get their assigned

work done. Responses were mixed in this domain. The majority of respondents agreed (79%) that

paperwork and other bureaucratic concerns do tend to supersede client interests (the remainder of this

response was 21% Neutral). Another clear finding was that the majority of respondents (86%) felt

emotionally drained and burned out without enough time to do their work, with 7 percent neutral and 7

percent not reporting such stress and time shortages. Likewise, the majority (64%) of respondents felt

that their workload and job expectations weren’t reasonable (the remainder of this response was 29%

Neutral and 7% Disagree). Respondents were mixed regarding whether they felt there were enough

resources and services to meet their client’s needs (57% Agreed; 29% Disagreed; 14% Neutral).

Child and Family Engagement. The fifth domain included in the organizational culture and climate survey is engagement. Engaged environments contain workers who perceive themselves to be doing something worthwhile and to feel a personal connection to the work they do with families. Engagement within an organizational climate typically includes personalization, the extent to which staff feels involved with families, and personal accomplishment, the level of success staff feel they have in their work with families. There was a moderate trend across the engagement domain that respondents felt connected to the families they worked with and perceived themselves as making a difference in their lives. Fifty-seven percent of respondents agreed in contrast to only 29 percent who disagreed and 14 percent who remained neutral. All respondents agreed that they care about their clients and treat them with respect (79% Strongly Agreed; 21% Agreed). All respondents also agreed to some extent that they worked with clients to address their family’s needs (71% Strongly Agreed; 29% Agreed). Finally, the majority of respondents agreed to some extent that their work offered them opportunities to make a difference in the lives of children and families (93% Agreed; 7% Disagreed).

Functionality. The sixth domain included in the organizational culture and climate survey is

functionality. Functional work climates are those where employees feel they understand the vision,

mission and goals of an organization, understand their place and roles within the organization, and feel

a sense of cooperation from co-workers and administrators in supporting their organization’s work

toward such goals. Job performance standards are clear, appropriate feedback is given to employees,

and opportunities for personal growth and development exist and are clearly presented to staff. First,

the majority of respondents (71%) agreed to some extent that their organization provided advancement

opportunities if they worked toward them (the remainder of responses for this item were 21% Neutral

and 7% Disagree). Additionally, the majority of respondents (79%) felt that they had a clear

understanding of their work responsibilities (21% Disagreed). In terms of how well staff perceived that

co-workers functioned together to serve families, a clear positive trend was seen in that the majority

(79%) of respondents agreed to some extent that their co-workers collaborated to improve the

effectiveness of services delivered to families (14% Disagreed; 7% Neutral). A more moderate positive

trend was seen when staff were asked if this type of positive performance was recognized by their

organization and rewarded. Half of respondents felt that due recognition was given, 29% did not and 21


percent were neutral. Results indicated that most staff (79%) do have a basically clear understanding of

their organization’s mission, vision and strategic plan, but results were very mixed regarding whether

the work atmosphere encouraged open communication. Fifty-seven percent of respondents agreed

that it did, and 43 percent had not encountered a work environment where open communication was

encouraged. The majority of respondents also agreed to some extent (71%) that accurate and timely

feedback was given to staff.

Staff Morale. The seventh domain included in the organizational culture and climate survey is

morale. Morale relates to employees’ attitudes about their job and career path, their level of job

satisfaction, and their level of commitment to an organization and its mission. Participants were asked

how they felt about their salary. Results were mixed regarding whether respondents felt that they were

compensated fairly. Half of respondents did not feel like they were compensated fairly, 43 percent did,

and 7 percent were neutral. In a clear positive trend, 79 percent of respondents agreed at least

somewhat with the statement that they liked their job and the agency they worked for. However, 57

percent of respondents indicated that they agreed to some extent with the statement that they often

thought about quitting their job, and respondents generally agreed (86%) they would leave their current

position if they were offered a different job that paid the same salary.

Training Initiative Interviews

To gain a better understanding of how differences in training initiatives across regions may contribute to

different patterns of knowledge transfer, retention, or employee support, the evaluation team

conducted semi-structured interviews with training staff from each region. The interview protocol

included 12 questions aimed at understanding what types of formal and informal trainings and supports

are offered to new case managers and investigators during their first year, what kinds of benefits,

challenges, and gaps exist with the pre-service and other training initiatives, how training outcomes are

analyzed, and what improvements are needed to make these early training initiatives most effective

(see Appendix E for interview protocol). Representatives from all nine of the study sites took part in the

interviews, but because some Community-Base Care agencies (CBCs) trained both case managers and

CPIs, seven interviews were conducted. These interviews included five CBCs, one Department of Child

and Family Services Circuit, and one Sheriff’s Office. The evaluation team conducted interviews by

phone, lasting approximately 30 to 60 minutes. Responses from all interviews were then organized by

domain using the interview questions as a guide. Findings by theme are discussed below.

Training Structure

Interviewees were asked to describe the structure of both pre-service and other trainings during the

first year. Most study sites indicated that training personnel includes one administrative position (e.g.,

Director) that oversees several trainers who specialize in different areas or are assigned to different

units, such as pre-service training, in-service training, case manager training, child protective

investigator (CPI) training, field training, or certification support. Some CBCs train both case managers

and CPIs, and some train only case managers. Responses indicated that there are usually between four

and six cycles of pre-service training per year. The total time spent in pre-service training is typically 10

to 12 weeks, with CORE training lasting three to five weeks with lab days and field days woven

throughout, and Specialty Track training lasting between three to five weeks. Specialty trainings on

issues like human trafficking, mental health, sexual abuse, trauma, water safety, or car seat safety are

often included after CORE training, and most respondents referenced specific practical training for the


FSFN system, which is usually given all at once at the end of pre-service training, though one site

mentioned dividing the FSFN portion up throughout pre-service. The structuring of field days varies

across sites. The number of structured field days across all study sites ranges from five to 17, and the

field days are interspersed at various points throughout the pre-service training. Some training staff

shared examples of strategies they’ve used to strengthen the knowledge gained during field days, such

as creating a one-page double sided checklist that trainees could use as a guideline in their observations.

This was helpful in ensuring that the training staff’s observations were as effective as possible and

included feedback and debriefing. It was suggested that this is something that could be done within the

current training structure, as one respondent noted that a barrier to training is that training directors

are limited in what they can do to make improvements when they notice challenges.

Across study sites, there are different strategies in place for increasing trainees’ awareness of internal

and external partners. For instance, some sites incorporate presentations from numerous community

partners (e.g., substance abuse and mental health agencies, legal partners, law enforcement, and

former clients) so that trainees have a stronger understanding of which agencies they’ll work with and

what processes they’ll use when they begin practicing independently. Alternatively, some sites

incorporate field trip days, where trainers bring their classes to some of these agencies so they can

become familiar with them. Respondents also discussed bringing in internal staff to speak at trainings,

such as clinical and kinship workers and DCF or CMO leadership.

First year training initiatives In addition to pre-service training, many sites have required in-service trainings for new employees

during their first year, which may include “onboarding” trainings to orient new employees to their

agency as well as topical trainings relevant to their practice. Some of the required trainings are

determined by DCF, and some are offered on an “as needed” basis depending on staff needs or issues

discovered during Quality Assurance (QA) reviews. The timeframe for in-service trainings ranges from

less than one hour to three hours, and occasionally one half or full day, though all respondents

emphasized the importance of keeping training requirements low because of the high caseloads many

caseworkers carry. Additionally, the training structure varies by site in terms of which agency is

responsible for training; some trainings are conducted in-house and some are provided by local,

contracted agencies or e-learning platforms. A listing of the various training topics discussed by

respondents is below:

Human Trafficking

Sexual Abuse Protocol

Medical Neglect

Caregiving Capacity

Psychotropic Medications

Self-Care/Compassion Fatigue

Quality Contacts & Engagement

Time Management

Child Placement Agreement

Missing Children

Mental Health First Aid

Trauma-Informed Care

Safety Planning

Conditions for Return

Organizational Skills

Documentation

Critical Thinking

Practice Model

Caregiver Protective Capacities

Domestic Violence


Several sites mentioned that new trainings are also required when updates to policies and procedures

take place, which is a frequent occurrence. Examples of informal or unstructured training include

problem-solving of case complications, frequent one-on-one supervision meetings early on, modeling in

the field during the initial months, technical support from various staff (e.g., supervisors, mentors,

trainers, lead child advocates, or job coaches), “lunch and learn” sessions on topics in demand, and unit

meetings. Respondents also discussed the creation of agency-specific documents and tools such as “how

to” guides and tip sheets.

Many sites have fluctuated between different pre-service training models over the past five years, and

some are in the process of making changes. At one site, new employees are put into a structured in-

service for certification for four weeks. They are assigned a career development counselor for 10-12

weeks, whose responsibilities include accompanying the new employee on court hearings and home

visits in addition to conducting workshops, running mock hearings, and ensuring that staff are retaining

instruction. It was reported that there has been an increase in retention since this model was

implemented within the last few years. Another site has implemented recent changes to their training

model by having a trainer meet with new employees at regular intervals to determine if transfer of

learning is happening and to assist with areas where employees need additional development.

Respondents at this site reported that this method increases communication between new hires and

other staff and allows the new employee to feel more supported and more comfortable voicing

concerns. A site that trains CPIs discussed their model of having a “training squad or unit” that new

hires are assigned to after completing pre-service training, in which a trainer works closely with them for

eight weeks on a reduced caseload, providing feedback and reinforcing expectations throughout that

timeframe. Another site previously used this model for case managers, but has since switched to have

new employees start directly with a regular unit. Some sites discussed changes in training topics over

the past five years, with active shooter training, mental health first aid, trauma informed care, drug

identification, and human trafficking being some of the more recent additions. It was evident that sites

were responsive to a multitude of factors in the structuring of trainings, including staff needs, agency

and policy changes, and changing community patterns.

Benefits. When asked about the major benefits of pre-service training, nearly all respondents

spoke about the standardized foundation that it gives new hires on child-welfare practice and on

understanding provider agencies. This is especially true for trainees outside the social work field, for

whom some of the basic concepts of child welfare may be new. And for all trainees, the pre-service

training was seen as helpful in dispelling myths or challenging pre-conceived notions about issues

related to families in the child welfare system. Respondents also pointed to the benefit of the format

that combines both knowledge and hands-on training through labs and field experience. The

development of engagement skills was seen as an essential component of the training that is necessary

for both case management and child protective investigation. One respondent pointed out the benefits

of training case managers and CPIs together, such as the opportunity for trainees to build relationships

and understand the system at a deeper level by learning about each other’s roles and responsibilities.

On this point, some respondents suggested that the CORE component of pre-service is heavily focused

on child protection, so case managers may naturally learn a little more about CPIs’ roles than CPIs’ learn

about case management.


Challenges. With regard to challenges of pre-service training, several themes emerged. One consistent

area of concern is that field days are very difficult to coordinate and that trainees often do not observe

in the field what they are learning in the classroom because the person they are shadowing doesn’t have

the particular experience to offer them at the time of observation, and therefore trainees can’t relate

the experience back to the content. One interviewee explains:

The theory of field days is good, that you should learn and then practice. But reality is that it doesn’t work.

If you’re teaching them something about family engagement and the next field day they’re transporting kids

or going to court, it doesn’t work the way they envisioned it to work. There is a lot of lost opportunity. There

is chunky paperwork to do observations, and a lot of times trainees come back not feeling good about the

field days because they didn’t get to do what they wanted or expected.

Other respondents echoed this concern of the difficulties of translating knowledge from the classroom,

in that the way trainees often learn methodologies in training is different from what they learn in their

field observations, which raises a lot of questions for them.

Other themes related to challenges of pre-service training include redundancy of content, illogical flow

of curriculum in some areas, and the contradiction of having a substantial amount of content in some

areas but lacking enough content on particular subjects like FSFN. For instance, some respondents

described the pre-service curriculum as repetitive, especially if there are multiple trainers who may not

be aware of sections that other trainers have already covered. Some interviewees spoke of efforts to

rearrange or bolster the curriculum (within guidelines) in order to make it more logical, such as

introducing the practice model at a different point, or coming up with a site-specific field guide that

both supports the curriculum and adds relevant information about the child welfare system. With

regard to length, the volume of information included in the pre-service curriculum in a relatively short

period of time was described as overwhelming to many trainees and difficult to retain. This was

compounded by the changes to the curriculum that have taken place over the past several years, which

was perceived as adding a significant burden to trainers. And for FSFN concerns, many respondents

indicated that there is not enough time in the current training model for trainees to get enough practical

experience to feel competent with the system, and though many sites continue their own form of

informal FSFN training after employees complete pre-service training, it was suggested that the FSFN

portion be longer and more fully integrated into the curriculum, perhaps by assigning trainees a case to

complete while they are in training and have direct support. This was reflected by case managers and

CPIs during their observations and in their self-assessment responses as well. Finally, one veteran

trainer expressed that it is unrealistic to expect pre-service training to prepare case managers for what is

demanded of them today based on the current practice model, and that the limited funding CBCs

receive makes it difficult to find solutions to the challenges case managers face.

Strengths. Interviewees also identified several strengths of the pre-service curriculum. Several

respondents indicated that the addition of specialty tracks is a significant enhancement to the

curriculum, especially for case managers, and that trainees feel that these components offer tools that

pertain specifically to their jobs. It was also widely noted that the CORE component of the curriculum is

fairly comprehensive as a foundational element and that it allows all trainees to have the same basis for

practice. Another strength consistently noted was that the labs and field days really help give trainees

hands-on practice that is difficult to obtain in the classroom. Other respondents commented on specific

aspects of the curriculum they felt were strong, such as the focus on trauma and the quality of content

on maltreatment, and that it is a good reflection of the social work field and human services in general.


Gaps. Interviewees were also asked about perceived gaps in the pre-service curriculum. There

were differing opinions about whether the content of the curriculum is appropriate for case managers’

needs; some suggested that topics should go more in depth. One area that was given as an example

where training staff wished for more in-depth information, was understanding more about how bruises

and burns are related to safety assessment. An area that training staff gave an example of the

curriculum being too in-depth was the information about child development and neurological bases for

trauma. Respondents from one site noted that there is a gap in discussions of partnerships between

foster parents and other system partners, and that understandings of system partnerships are lacking in

general in the curriculum, though they are an important part of the practice model. Several

respondents mentioned that the curriculum is outdated in several areas, such as visuals and videos that

don’t work or can no longer be accessed.

Evaluation. Respondents were asked about what steps are taken to evaluate training outcomes.

Strategies for evaluating training outcomes varied across sites, but most indicated use of electronic or

paper survey (or both) after various pre-service and in-service trainings and workshops to gather trainee

or employee feedback. In addition to these formal methods, some sites also gather informal, open

feedback from trainees and employees after trainings. Numerous sources of data are also analyzed by

CBCs, CMOs, and DCF to determine the effectiveness of training, such as pre/post scores and

certification passing rates. Data from exit interviews and rates of turnover and retention rates are also

analyzed in order to try to determine whether there is any relationship between training and turnover,

and whether changes in training can help increase retention. Feedback from quality assurance reviews

and continuous quality insurance (CQI) meetings were also mentioned as sources of input on evaluation

methods for training. Other ways that respondents said they measured training impact were through

general observation of trainees’ and new employees’ skills in order to understand whether content is

being absorbed. However, many respondents pointed out that measuring the impact of training is

complex, and there are many confounding factors that make it difficult to know whether problems are

related to deficiencies in training, or other issues like the demanding nature of the job or being the

wrong career choice for some people.

Ongoing Needs. Respondents provided many examples of needs that still exist for employee

support during and after pre-service training. There was wide agreement that during the first year—and

especially the first three months—there needs to be greater support for new hires, and not just in the

form of training, which was sometimes seen as interfering with opportunities for practice. Respondents

recommended having a longer period of protected caseload that is supported by administration so that

new employees can take the time necessary to learn how to independently handle their cases. Some

respondents expressed frustration with the amount of changes to policy and practices that frequently

occur, commenting that the new trainings and procedures that are rolled out as a result of these

changes are time consuming and overwhelming and cause employees to spend more time on

assessments and less time with families. It was also recommended that new case managers and CPIs

should have consistent access to support staff dedicated to helping new employees, such as a job coach

or field trainer that works directly with them for several weeks or months to ensure they can become

adequately familiar with processes and refine their skills. Some respondents expressed the need for

more skill development through practical components, suggesting that “trying to train that into people

is challenging.” Similarly, one interviewee suggested that there needs to be more training around


professionalism, such as basic customer service skills and how to conduct oneself in a meeting, since

many new hires have never had a job prior to being a case manager or CPI.

Some feedback indicated the need for different components of pre-service training to be better

connected. Skill development could be tied into practice better, for instance, by showing how

interviewing skills help with assessments, rather than teaching each component separately. Another

suggestion was to connect the content more to legal processes, such as helping trainees understand the

relevance of Conditions for Return to court orders. One respondent felt it would be helpful to

incorporate supervisors more regularly into training so that the supervisors can better assist with

trainees’ transfer of learning once they are on the job. It was suggested that “showing a united front”

among trainers and supervisors would also enhance the consistency between what trainees learn in pre-

service and what they learn in their units. It was also suggested that there be more support or training

around documentation processes, such as quality information collection through Family Functioning

Assessments - Investigation (FFAIs) or Family Functioning Assessments – Ongoing (FFAOs), and that

trainees have more opportunities to practice these aspects of casework.

Focus Group Update

Four sites have taken part in the six-month focus groups, which have replaced the six-month

observations originally outlined in the evaluation plan. The purpose of the focus groups is to gather

feedback from workers who completed pre-service training together on the same components outlined

in the on-site observations in order to determine to what extent case managers and CPIs feel that pre-

service or other training has helped them develop skills related to core competencies. A member of the

research team works with site contacts to schedule a focus group that includes as many case managers

and CPIs as possible. The remaining four sites will participate in focus groups over the next month, and

results from this component will be included in the next report.

Summary and Discussion

In conclusion, findings from the Knowledge Assessment pertaining to the CORE training curriculum

provided some evidence that knowledge was gained as a result of the training. Overall, there was

significant improvement in items correctly answered at the post-training for both the CPI and the Case

Manager Specialty Track Assessments compared to the pre-test. There was also an increase in correct

responses for each module assessed. Taken together, findings from Knowledge Assessments

administered to pre-service trainees provide evidence that knowledge was gained as a result of the

CORE curriculum training as well as the Specialty Track training.

The purpose of the observation component is to assess whether or not this knowledge gained is

transferable into practice. The evaluation team completed a skill checklist while observing new case

managers and CPIs interact with families, primarily on home visits. Scores on the observation checklist

indicate that case managers were demonstrating desired skills the vast majority of the time. The most

frequently demonstrated skills were related to utilizing interviewing techniques and strategies, engaging

and assessing families from a strengths-based perspective, and discussing progress in recommended

services. In less frequent instances case managers struggled with engaging caregivers, being strengths-

based with each family member, and not referring back to or discussing the safety plan during the home

visit. Scores from the CPI observations indicate that CPIs were also demonstrating desired skills the

majority of the time. The areas where skills were most frequently demonstrated include engaging and

assessing families from a strengths-based perspective, identifying and facilitating referrals to


appropriate services based on the FFA, and demonstrating knowledge and application of Florida’s Child

Welfare System Guiding Principles. CPIs that struggled in some skill areas were at times more focused

on policy and forms than beneficial interaction with family members, relied more heavily on a form to

complete the interview rather than asking follow up questions, and did not attempt to or did not have

the resources within the time allowed to provide non-English speaking families with a translator.

All case managers and CPIs from the study sites received an electronic link to a Self-Assessment Survey.

The majority of case managers self-reported a high level of competence with FFA completion.

Respondents scored highest on their ability to understand all the FFA components required for

completion and lowest in their assessment of whether they have enough time to fully complete the

FFAs. Second, self-reported practice competency was included in the survey for both case managers

and CPIs. Responses indicate that case managers perceived that they were demonstrating the majority

of skills listed during their most recent family visit, particularly with regard to respectful communication.

CPI respondents indicated with high frequency that they demonstrated most of the skills during their

last family visit. Skill areas with the highest marks on the CPI surveys include utilizing interviewing

techniques and strategies, identifying and facilitating referrals to appropriate services, and

demonstrating knowledge and application of Florida’s Child Welfare System Guiding Principles.

Third, a transfer of learning section on both case manager and CPI Self-Assessments gauged the

perceived transfer of learning from pre-service to practice. Case managers felt that the pre-service

training had helped them prepare for their jobs by teaching them techniques for interviewing and

relating to different clients, discussing policy and “real-life” cases, providing an overview of FSFN, and

explaining the processes involved in moving a case through the system from the assessment phase to a

case plan, to judicial review, and to TPR (Termination of Parental Rights)/Permanency. Shadowing and

field observations were also mentioned as being helpful. Most CPI Self-Assessments indicated that pre-

service training was helpful to the CPI’s job, particularly elements such as role play, discussion of policies

and procedures, and creative ways of quizzing. However, an overall sentiment expressed by many

respondents was that there should be less “book work” and more hands-on activities and field

experiences that would better prepare trainees for real world settings.

To gain a better understanding of how differences in training initiatives across regions may contribute

to different patterns of knowledge transfer, retention, or employee support, the evaluation team

conducted semi-structured interviews with training staff from each study site region. One primary

difference with the pre-service training is that some lead agencies train both case managers and CPIs,

and some train only case managers. Training staff indicated that typically, there are between four and

six cycles of pre-service training per year. Total time spent in pre-service training is typically 10 to 12

weeks, with CORE training lasting three to five weeks inclusive of lab days and field days woven

throughout. Specialty Track training typically lasted between three to five weeks. Specialty trainings on

issues like human trafficking, mental health, and trauma-informed practice are commonly included after

CORE training, and most respondents referenced specific practical training for on FSFN. The structuring

of field days varied across sites. The number of structured field days across all study sites ranged from

five to 17. In addition to pre-service training, many sites have required in-service trainings for new

employees during their first year, which may include trainings on agency policy and procedure in

addition to topical trainings relevant to their practice.

When asked about the major benefits of pre-service training, nearly all respondents felt that it does

provide new hires with a standardized foundation on child-welfare practice and on understanding


provider agencies. It was noted that this is especially helpful for trainees who have a degree other than

social work. With regard to challenges of pre-service training, one consistent area of concern was a

disconnect between what is taught in pre-service versus what trainees encounter in the field, for

example the impact of poverty and language barriers. Though respondents were very favorable about

the inclusion of field days in pre-service training, there was wide agreement that they need to be

restructured in a way that better ensures alignment with skills that trainees are being taught in training.

Other themes related to challenges of pre-service training include some redundancy of content, illogical

flow of curriculum in some areas, and the contradiction of having a substantial amount of content in

some areas but lacking enough content on particular subjects like FSFN. Respondents also provided

examples of needs that still exist for employee support during and after pre-service training. There was

wide agreement that during the first year—and especially the first three months—there needs to be

greater support for new hires. Respondents recommended having a longer period of protected

caseload that is supported by administration so that new employees can take the time necessary to

learn how to independently handle their cases.

Although there was a small sample size for the organizational culture and climate survey, some clear

trends were evident. One staggering trend was that most staff saw their current positions as case

managers and CPIs as steppingstones to something else. Eighty-five percent of respondents planned to

leave their positions, and 77 percent of respondents were looking to leave child welfare all together.

While there were clear positive trends in the data such as staff feeling positive about making a positive

contribution to society, having independence in their daily work tasks, and having a clear understanding

of their organization’s values and mission, with turnover intentions so high, these positives do not

appear to be enough to cause workers to stay in their current jobs or even to aspire to promotions

within the field of child welfare (despite 71% of respondents agreeing that there were opportunities for

advancement if they worked toward them).

Interesting to note is that the majority of participants felt they were expected to be responsive to the

unique needs of their clients and to implement evidence-based practices, but responses were much

more divided when the question was posed as to whether quality of services was valued by the

organization over meeting set quotas or performance measures. In a separate category we also see that

staff overwhelmingly feel that paperwork and other bureaucratic concerns do tend to supersede client

interests. Thus, while the expectation and intention to do good is present, the supports necessary to

accomplish the work may be lacking. Additionally, and not surprisingly, half of survey respondents did

not feel like they were compensated fairly for the work they completed.

Another stress point seems to be emotional strain and staff burnout, which is currently not an area that

is addressed by pre-service training. Eighty-six percent of all respondents felt emotionally drained and

burned out without enough time to do their work. They also felt that their workload and job

expectations were not very reasonable. It may be beneficial for agencies to consider what supports

might be implemented during the first year on the job specific to handling and coping with emotional

strain. Another option might be to incorporate some time management strategies into the pre-service

training, or to have more experienced case managers be a part of pre-service training to talk about what

strategies they have developed to cope with the existing workload.

With turnover rate per year statewide for protective investigators averaging 44 percent (Department of

Children and Families, 2016) and case managers 37 percent (Florida Tax Watch, 2015), discussions

around retention of quality staff must be at the forefront. Case managers and CPIs may not have the


support and service array needed to meet the level of workload and expectations, making the job

overwhelming and stressful. Input from the self-assessment currently being distributed across sites may

help shed light on the specific supports and challenges workers perceive as they move through the mid-

point of their first year.

Next Steps

The next steps in the evaluation include the completion of all mid-point activities conducted at the six-

month point, which include the follow-up specialty track assessment, the self-assessments, and the

focus groups. The activities will all be included by early November 2019. Several final activities will be

completed at each site’s twelve-month mark, including a follow-up focus group, a second organizational

culture and climate survey, an electronic specialty track knowledge assessment, and an electronic self-

assessment. This final point of the data collection process will take place for each site between

December 2019 and May 2020. The organizational culture and climate survey sent at 12 months will be

contrasted with the three-month survey results to assess trends and changes in perspective regarding

turnover intentions and perceptions of organizational culture and climate. Analysis of the FFAs will be

ongoing throughout the project period, and data from this component will be integrated with data from

other assessments in the remaining reports.

Changes to the Evaluation Plan. This study began at each site when a new cohort of case managers

and/or CPIs were ready to go through the pre-service training. As such, the study timeline has been site

dependent and each site is at a slightly different point in the data collection process. Additionally, there

were some early barriers to implementing study activities on the timeline initially projected. The study is

now ongoing at all sites, with some adjustments made to streamline activities and better meet the goals

of the evaluation. One example of this type of adjustment is the change from six- and twelve-month

field observations to six- and twelve-month focus groups. The evaluation team determined that

completing observations at the six- and twelve-month intervals would not yield data that could be

directly correlated back to pre-service training. Once back in the units there are multiple factors, such

as the level of supervisory involvement, turnover, internal training opportunities, and other supports,

that influence a new staff’s performance. Therefore, focus groups will concentrate on what modules

within pre-service training prepared new staff to effectively perform their job responsibilities, what

content was missing, what content needed more emphasize, and what suggestions about the format

and/or structure would new staff recommend.

It is anticipated that data collection activities for the overall project will run through June 2020, with

data analysis complete by August 2020 and a final report submitted in September 2020.


References

Altman, J. C. (2008). Engaging families in child welfare services: Worker versus client perspectives. Child Welfare, 87(3), 41-61.

Chapman, M. V., Gibbons, C. B., Barth, R. P., McCrae, J. S., & NSCAW, R. G. (2003). Parental views of in-

home services: What predicts satisfaction with child welfare workers? Child Welfare, 82(5), 571-596.

Department of Children and Families, Office of Child Welfare. (2018). Child Protective Investigator and

Child Protective Investigator Supervisor Educational Qualifications, Turnover, and Working Conditions Status Report: Annual Report. Retrieved online from: http://centerforchildwelfare.fmhi.usf.edu/kb/LegislativeMandatedRpts/CPI_WorkforceStudy2018.pdf.

Florida Tax Watch. (2015). Challenges Facing Florida’s Community-Based Child Welfare System.

Retrieved online from: https://www.flchildren.org/wp-content/uploads/2018/07/TaxWatch.pdf Glisson, C. (2002). The organizational context of children's mental health services. Clinical child and

family psychology review, 5(4), 233-253. Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interorganizational

coordination on the quality and outcomes of children’s service systems. Child abuse & neglect, 22(5), 401-421.

Glisson, C., & James, L. R. (2002). The cross‐level effects of culture and climate in human service teams.

Journal of Organizational Behavior, 23(6), 767-794. Kemp, S. P., Marcenko, M. O., Hoagwood, K., & Vesneski, W. (2009). Engaging parents in child welfare services: Bridging family needs and child welfare mandates. Child welfare, 88(1), 101. Ortega, R. M., & Faller, K. C. (2011). Training child welfare workers from an intersectional cultural

humility perspective: A paradigm shift. Child Welfare, 90(5), 27-49. Radey, M. & Schelbe, L. (2017). From Classroom to Caseload: Transition Experiences of Frontline Child

Welfare Workers. Child Welfare, 95(2), 71-89.

http://centerforchildwelfare.fmhi.usf.edu/kb/LegislativeMandatedRpts/CPI_WorkforceStudy2018.pdf

http://centerforchildwelfare.fmhi.usf.edu/kb/LegislativeMandatedRpts/CPI_WorkforceStudy2018.pdf

https://www.flchildren.org/wp-content/uploads/2018/07/TaxWatch.pdf


Appendix A: Average Correctly Answered Core Curriculum Knowledge Assessment Questions

Average Correctly Answered Core Curriculum Knowledge Assessment Questions Pre-Test (n = 91)

𝑋 (Stand. Dev.)

Post-Test (n = 91)

𝑋 (Stand. Dev.)

Paired t-tests

The Practice Module (4 items) 2.57 (.92) 3.11 (.81) t (90) = - 4.96 ***

Child Development (3 items) 2.44 (.60) 2.53 (.67) ns

Trauma and the Child (3 items) 2.10 (.76) 2.23 (.73) ns

Family Conditions (4 items) 3.42 (.72) 3.52 (.72) ns

Understanding Child Maltreatment (5 items)

2.91 (1.13) 3.67 (1.10) t (90) = - 5.38 ***

Assessing and Analyzing Family Functioning (6 items)

4.32 (1.19) 5.20 (.93) t (90) = - 6.99 ***

Safety and Risk (5 items) 3.53 (1.13) 4.35 (.82) t (90) = - 6.11 ***

Safety Planning (5 items) 3.41 (1.14) 4.11 (.90) t (90) = - 5.08 ***

Overall (35 items) 24.69 (3.53) 28.71 (3.53) t (90) = - 11.48 ***

Appendix B: Field Observation Checklist for Case Managers

Blind File #: Case Manager: Transfer Staffing Home Visit Permanency Staffing

Circuit: Reviewer: Other Attendee Roles and Agencies (No names):

Region: Review Date:

INSTRUCTIONS: For each competency area, mark “Y” if the competency or skill was clearly observed and “N” if it was not. If there was no opportunity for the competency or skill to be addressed, mark “N/A” for not applicable. For each competency or skill that was observed, provide one brief case-specific example in the “Example” column. Provide a brief statement about the overall observation, including any situational abnormalities in the “Observer Notes” section at the bottom.

Competency or Skill

Assessment

(Mark Y, N, or N/A) Example






Conducts or has conducted on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.


Competency or Skill

Assessment


Engages and assesses families from a strengths-based perspective.



Discusses progress in recommended services.

Demonstrates active and ongoing assessment of safety management plan

Demonstrates use of safety analysis and planning criteria needed to modify safety plans

Demonstrates knowledge and application of Florida’s Child Welfare System Guiding Principals

Observer Notes:


Appendix C: Field Observation Checklist for CPIs

Blind File #: Case Manager: Transfer Staffing Home Visit Permanency Staffing

Circuit: Reviewer: Other Attendee Roles and Agencies (No names):

Region: Review Date:

INSTRUCTIONS: For each competency area, mark “Y” if the competency or skill was clearly observed and “N” if it was not. If there was no opportunity for the competency or skill to be addressed, mark “N/A” for not applicable. For each competency or skill that was observed, provide one brief case-specific example in the “Example” column. Provide a brief statement about the overall observation, including any situational abnormalities in the “Observer Notes” section at the bottom.

Competency or Skill

Assessment








(Continued from Page 1)

Competency or Skill

Assessment


Conducts or has conducted on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

Engages and assesses families from a strengths-based perspective.



Discusses progress in recommended services.

Demonstrates active and ongoing assessment of safety management plan

Demonstrates use of safety analysis and planning criteria needed to modify safety plans

Demonstrates knowledge and application of Florida’s Child Welfare System Guiding Principals

Observer Notes:


Appendix D: Observation Checklist Objective and Competency Key

Item Curriculum Component

Demonstrates sensitivity to cultural differences and ethnicity among clients.

Competency 15

Objectives 33, 44, & 53

Communicates with family members and collaterals by asking parents or guardians to elicit, define, and prioritize concerns and needs.

Competency 17

Core Labs 1 - 3


Competency 18

Core Lab 1

Utilizes interviewing techniques and strategies to prepare for and conduct age-appropriate interviews with the child, the caregiver, and the family.

Competency 19

Core Labs 1 - 4

Interprets the results of the department’s Family Functioning Assessment to make appropriate child safety determinations.

Competency 39

Objectives 133 - 147

Identifies, documents, and conducts on-going, age-appropriate assessment activities that ascertain if a child’s physical, mental, social, and educational needs are met.

Competency 43

Objectives 133 – 135

Core Lab 5

Engages and assesses families from a strengths-based perspective to develop and implement a case plan/safety plan based on this assessment.

Competency 64

Objective 141

Core Labs 1 & 5

Identifies and makes referrals to appropriate services based on the Family Functioning Assessment.

Competency 81

Objectives 41, 170 & 171

Discusses progress in suggested services. Competency 89

Objectives 170 & 171

Demonstrates general knowledge of Florida’s child welfare system. Competency 2

Objectives 1-14


Appendix E: Self-Assessment Survey

Introduction

The University of South Florida is conducting a study on the impact of pre-service training for newly

hired child welfare professionals in Florida. The goal of this survey is to collect responses from new CPIs

and Case Managers at different points in time after pre-service training. We are asking you to take part

in this research study because you are currently or have recently completed training to be a child

protective investigator or case manager. Your responses immediately after training may change as you

spend more time in the field and reflect on your experiences. At other times, a response may be so

important to you that it stays consistent over time. We will send this survey to you at 3 months, 6

months, 9 months, and 12 months post-training. We are interested in learning from you as you go –

whether it is immediately after the training or one year post-training.

Your participation is this study is voluntary. There are no anticipated risks to taking part in this study, as

participation will not affect your job status, and identifying information will be excluded from analyses

and reports. You may experience some benefits to participation, including reflecting on the training and

skill application process and heightened awareness of the importance of child welfare practice. You may

also benefit from the knowledge that your participation will help to improve child welfare training and

practice. There is no compensation for participating in this study, and there is no cost.

Funding for the study is provided by the Florida Institute for Child Welfare at Florida State University. If

you have any questions or concerns about this study, you may contact the Principal Investigator, Amy

Vargo, at 813-974-5356 or [email protected]. Thank you very much for your time!

Role

1. What is your current role?

Child Protective Investigator

Case Manager

Other (Please Specify) _____________

2. How long ago did you complete pre-service training?

Less than 3 months

Between 3-6 months

Between 6-12 months

More than 12 months

mailto:[email protected]


Self-Assessment – FFA Competency

For the following questions, please select the response you feel best represents your experiences

completing the Family Functioning Assessments.

1. I understand all of the components I am required to complete in the Family Functioning

Assessment:

a. Strongly Agree

b. Agree

c. Disagree

d. Strongly Disagree

2. I can provide sufficient descriptive information when completing the Family Functioning

Assessment:

a. Strongly Agree

b. Agree

c. Disagree


3. I am confident in my ability to rate the functioning level of children:

a. Strongly Agree

b. Agree

c. Disagree


4. I am confident in my ability to rate the caregiving capacity of adults:

a. Strongly Agree

b. Agree

c. Disagree


5. I have enough time to fully complete the Family Functioning Assessment:

a. Strongly Agree

b. Agree

c. Disagree


6. If there is anything you would like to elaborate on regarding your experiences with the Family

Functioning Assessment, please do so below:


Practice Competency

For the following questions, please indicate whether you feel you demonstrated each skill during your

last three months of work by selecting Yes or No. If there was no opportunity to demonstrate a skill,

mark N/A. Please provide a brief case-specific example of how you demonstrated the skill in the

Example box (i.e., specific things you said or did during a visit that represents a particular skill).

Competency or Skill

Assessment

(Mark Y, N,

or N/A)

Examples of how YOU have

used this skill in your Work

I demonstrated sensitivity to cultural differences and ethnicity among

clients (e.g., respecting communication style, acknowledging value

system, discussing culturally specific services, etc.).

I communicated with family members and collaterals by asking

parents or guardians to identify, discuss, and prioritize concerns and

needs.

I interacted with persons allegedly responsible for maltreatment in a

way that allows information to be communicated in a non-

confrontational manner.

I utilized interviewing techniques and strategies (e.g. demonstrating

respect, active listening, using exploring and focusing skills, etc.) to

prepare for and conduct age-appropriate interviews with the child,

the caregiver, and the family.

I interpreted the results of the Department’s Family Functioning

Assessment to make appropriate child safety determinations.

I conducted or have conducted on-going, age-appropriate

assessment activities that ascertain if a child’s physical, mental,

social, and educational needs are met.

I engaged and assessed families from a strengths-based perspective.

I developed and implemented (or have developed and implemented)

a case plan/safety plan based on strengths-based assessment.

I identified and facilitated referrals to appropriate services based on

the Family Functioning Assessment.

I discussed progress in recommended services.

I demonstrated general knowledge of Florida’s child welfare system.


Transfer of Learning (open-ended questions)

1. At your current point post-training, what do you feel were the most helpful aspects of the pre-

service training that helped you prepare for the work you’re doing in the right now in the field?

2. At your current point post-training, what do you feel were the least helpful aspects of the pre-

service training that helped you prepare for the work you’re doing in the right now in the field?

3. Please describe any resources or supports given to you given to you during your time as a new

case manager that have helped you apply what you learned in the pre-service training to your

everyday work. (Some examples may include, but are not limited to supervision, peer support,

caseload size, time to complete tasks, or follow-up training)

4. What, if any, are some of the barriers you have experienced as you have tried to implement

what you learned during the pre-service training to your everyday case work? (Some examples

may include, but are not limited to caseload size, turnover, agency policies, etc.).

5. Based on your most recent experiences in the field, what suggestions do you have for improving

pre-service training?

evaluation of department of children and families core and

Documents