Data Management: Let's Talk About Using Data to Evaluate 

 November 6th 3pm to 4:30pm

 Linkages Shared Learning Webinar #8

Stuart Oppenheim, CFPIC, Executive DirectorDanna Fabella, Linkages Project Director

Peer County Advisors• John Dufresne, MSW, Program Manager, 

Fresno• Johnny Alaniz, Social Services Program 

Manager, Fresno

• Sarah Whittington, MSW, SW Practitioner, Fresno on behalf of Madera

Agenda• Overview of Evaluation/Research:  Stuart Oppenheim• What’s on the Toolkit?  Danna Fabella• What are some counties doing?

– John Dufresne & Johnny Alaniz:  Fresno– Sarah Whittington:  Madera

• Q & A• Discussion

– Next Steps

Research/Evaluation• Research is directed toward increasing 

knowledge, the primary aim being more knowledge or understanding of a particular group, problem or issue. The strict definition of scientific research is to perform a methodical study in order to prove a hypothesis or answer a specific question.

• Evaluation is concerned with answering questions about issues that arise in everyday practice. A distinguishing characteristic of evaluation is that, unlike traditional forms of academic research, evaluation is grounded in the everyday realities of organizations. Evaluations can be conducted of programs, processes, products, systems, organizations, personnel, and policies.

• Evaluation therefore answers questions like: ♦ Does it work? ♦ Does it do what we want it to? ♦ How well does it work?

Why Evaluate?• Not required by the Statewide Project• To let you know if and how Linkages is being 

implemented• To understand if you are getting the outcomes 

that you want• To let Leadership, staff, and stakeholders know 

how it improves outcomes for families

Approaching Evaluation• Know what you mean by Linkages.• What is success?

– Create a Logic Model• Inputs (training, , etc)• Outputs (CCP, Family Engagement, TDM, etc)• Outcomes (children remain at home, sanction cured, etc.)

• How will you measure it?– What data will you use?

• How will it be collected?– Automated/case flags– Case Review– Surveys

• Who will analyze it?• How will you use it?

– Adjust program– Build support

What we can find on the Toolkit:http://www.cfpic.org/toolkit/

Fresno County Linkages Data Collection Process

Presenters: John Dufresne Program Manager

Johnny Alaniz – Social Services Program Supervisor

Heading

Child removed from home TDM 

Scheduled

EW completes EW/TDM Datasheet on all 

adults & children who are part of TDM

TDM Scheduler sends email the Intake 

Linkages EW

Datasheet goes into TDM

TDM SW Facilitator 

notifies Linkages OA of outcome

Parents accept VFM

OA fast tracks case 

to Linkages JS

Children Detained Court involvement

CCP developed by Linkages JS, VFM SW and 

parent(s)

CCP Reviewed by all parties monthly and recertified every 6 

months

OA notifies Linkages JS’s of eligible AB429 

client Detention Hearing, JS & 

SW attend Detention Hearing

  Linkages JS allows AB429/WTW post aid services for the 

next 6 months and will maintain monthly contact with 

FR SW on progress of adults

Linkages JS will review AB429 Recertification after 6months 

and will contact SW and determine if an additional 6 

months of services are to continue

Data Collection and

tracking database

Fresno County Linkages Data

Collection Process

After Hearing Linkages JS & SW go over WTW CCP with 

parents. If client agrees CCP is signed by all parties.

Data that is collected

Sample Client data collected

FRESNO COUNTY LINKAGES DATA COLLECTION PARAMETERS

CCP OA CCP CCP CCP CCP CCP CCP JS J

S(OA

CWS)

AB429 Linkage

s

Court Report Date

Client

Last Name

Client First name

Ethnicity CWS ID #

CWS SvcCom

p

SW Dist #

CWS Closure

Date

CWS Outcome

+ / -

L/AB429

XXXX

XXXX

Caucasian XXXX VFM OLB

D2/1/20

12 Positive

CCP CCP JS (CALWIN) JS JS CCP

OARemin

dJS JS JS CCP OA

CalWin Id #

Linkage JS

Dist #WTW STATUS

Empl Status

No Emp

PT Emp

FT Emp

Sanction

Reocc Y / N

Date CCP

signed

6 MO CCP Reviewdate

G/C/D Y / N

No Longer Linkage

s

Second CCP Signe

d

CCP Recertificatio

ndate

AB429 Closed

XXXX ABCD Registered

No Emp N 4/11/20

11 9/11/2011 2/1/2012

2/1/2012

Who and/or where data is located

EW TDM Datasheets

0

5

10

15

20

25

30

35

WTW Adults for Given Month

Cal

WO

RK

s / W

TW A

ctiv

e A

dults

Adults Active on CalWORKs 22 20 30 29 25 29 16 23 21 18 23 27

WTW Active Adults 13 13 12 15 17 17 11 12 13 12 13 11

WTW NON Participating Adults 9 7 18 14 8 12 5 11 8 6 10 16

WTW Exempt 7 2 11 9 5 6 2 7 7 6 8 14

WTW Never Enrolled 1 5 6 4 2 5 2 2 1 0 1 0

Teen Parent Cal Learn 0 0 0 1 0 1 0 2 1 0 1 2

NonNeedyPayee 1 0 1 1 1 0 1 0 2 0 0 0

01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11 11/11 12/11

Eligibility Worker TDM Datasheet Stats

EW TDM Datasheets

0

20

40

60

80

100

120

Number of TDM Datasheets Completed for Given Month

TDM

Ref

erra

ls S

ent t

o EW

TDM Data Sheets 55 48 78 70 84 84 64 71 74 80 85 84

Adults 70 58 101 101 114 103 82 88 99 97 95 103

Adults on CalWorks Case 36 36 48 64 61 58 47 44 54 50 55 57

Adults no CalWorks Case 34 22 53 37 53 45 35 44 45 47 40 46

01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11 11/11 12/11

EW TDM Datasheets

0

10

20

30

40

50

60

Adults / No Active CalWORKs Case for Given Month

Adu

lts N

o C

alW

OR

Ks

Cas

e

Adults No CalWORKs Case 34 22 53 37 53 45 35 44 45 47 40 46

No Public Assistance 10 13 19 13 27 16 15 18 16 15 17 17

Active Cal Fresh Case 3 3 5 2 2 11 1 2 6 5 7 3

Active Medi-Cal Case 7 4 12 4 4 7 4 7 6 11 4 8

Active Cal Fresh / Medi-Cal Case 10 2 14 12 15 10 13 16 10 13 11 16

Pending Applications 4 0 2 4 5 0 0 0 2 0 0 1

AAP./ Foster Care/ GR 0 0 1 2 0 1 2 1 5 3 1 1

01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11 11/11 12/11

#2302 - CalWORKs Referrals

0

5

10

15

20

25

30

35

CalWORKs Referrals / Applications for Given Month

Cal

WO

RK

s A

pplic

atio

ns

#2302 CalWORKs Referrals 19 25 20 16 10 29 28 32 9 19 11 18

Return to Parent 15 20 16 9 7 21 19 25 6 16 9 13

Placed with Relative 4 5 4 7 3 8 9 7 3 3 2 5

Processed Return to Parent 9 11 15 8 7 9 9 11 3 6 5 9

Processed Placed w/Relative 2 4 4 4 1 5 3 3 3 1 1 3

Applications Pending 0 0 0 0 0 0 2 5 0 4 2 1

No Application Taken 8 8 3 4 2 14 12 11 2 6 2 5

Cal Fresh Only Applications 0 0 0 0 0 0 0 0 0 0 0 0

Courtesy Applications 0 1 0 0 2 0 0 0 0 0 0 2

Denied 0 2 2 0 0 1 2 2 1 2 1 0

01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11 11/11 12/11

EW TDM Datasheets

0

10

20

30

40

50

60

70

CalWORKs Adults for Given Months

Cal

WO

RK

s A

dults

Adults with CalWORKs Cases 36 36 48 64 61 58 47 44 54 50 55 57

Adults Active on CalWORKs 22 20 30 29 25 29 16 23 21 18 23 27

Adults Not Active on CalWORKs 14 16 18 35 36 29 31 21 33 32 32 30

Sanction 5 5 7 9 6 8 4 3 5 7 9 2

Drug Felon 2 3 1 3 2 1 5 1 0 5 0 4

SSI 3 1 1 5 2 2 1 1 4 4 5 5

Time Out 3 6 6 9 15 7 16 9 9 11 12 11

Undocumented 1 1 1 5 6 5 1 4 7 2 6 3

Ineligible 0 0 2 4 5 6 4 3 8 3 0 5

01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11 11/11 12/11

EW TDM Datasheet - 2011 Year in Review

2011 Totals Monthly AverageTDM Datasheets 87773Adults 1,11192Adults w/CalWORKs Case 610 50Adults no CalWORKs Case     501 41Adults Active CalWORKs 283 23Adults Not Active CalWORKs     327 27Sanction 70

5Drug Felon       27

  2SSI         34

      2Time Out     114

  9Undocumented        42   3Ineligible        40    

  3

WTW Active Adults 159 13WTW Non-Participating Adults 124 10WTW Exempt 84 7WTW Never Enrolled 29 2Teen Parent        8

    Non Needy Payee     75   6No Public Assistance 196 16Active Cal Fresh   50   4Active Medi-Cal   78   6Active Combo-CF&MC 142 11Pending Applications   18   1AAP/ GR   17   1#2302 C/WORKs Referrals 236 19Return To Parent 176 14Placed w/Relative   60   5Processed Return to Parent 102   8Processed Placed w/Relative     34   2Applications Pending     14   1No Application Taken     77   6Courtesy Applications        5

    Denied   13   1 

2011 TotalMonthly Average

Lessons Learned

• The amount of time that the Linkages data collection takes to compile.

• Lack of communication at times between: – JS/EW and SW – JS & OA– Linkages clients going to Intake rather than being referred 

to Linkages Intake EW• Current Excel collection method is not user friendly and is 

difficult to access reports in a timely manner.

Future Plans

• Expand Linkages to Emergency Housing population

• Expand Linkages to include Tribal TANF• Development of a comprehensive Database 

allowing to access reports at any given point in time.

Linkages: A Collaborative Effort to Serve the Needs of Clients: An Analysis of the Implementation of the

Linkages Program in Madera County, Department of Social Services

Sarah WhittingtonMSW Graduate Student

California State University, Fresno In Collaboration with Madera County Department of Social

Services

Madera County at a GlanceBureau of Labor Statistics & madera-county.com

• LOCATION: Madera County is located in the exact center of California, in the heart of the Central Valley and the Central Sierras. Fresno County borders on the south, Mariposa and Merced counties on the north, and Mono County to the east.

• GEOGRAPHIC AREA: 1,374,160 acres; 2,147 square miles, stretching from the rich San Joaquin Valley to the crest of the Sierra Nevada, the highest mountains in the contiguous United States. Bordered on the north by the Chowchilla River and on the south by the San Joaquin River, the County includes some of the richest agricultural land in the nation.

• CITIES: Chowchilla and Madera. Unincorporated communities: Ahwahnee, Bass Lake, Berenda, Coarsegold, Fairmead, Madera Ranchos, North Fork, Oakhurst, O'Neals, Raymond, and Rolling Hills.

• ECONOMY & EMPLOYMENT : Agriculture is the largest industry in the county, accounting for 29.9% of the employment. Government, another significant sector, accounts for 19.5% and services makes up 16.8% of the total. The county’s leading commodities include: almonds, grapes, milk, and pistachios.

• UNEMPLOYMENT: The Bureau of Labor Statistics has the unemployment rate in Madera County at 11.6 % in September 2012, down from a revised 13.0% percent in August 2012.

Madera County At A Glance• The 2011 Madera County

Population as estimated by the US Census Bureau equals 152,925.

• Persons under the age of 18 equal 28.2% of the population. (CA = 24.6%)

• Hispanic/Latino = 54.5% of the Madera Co. population. (CA = 38.1%)

• White = 37.5% of the Madera Co. population. (CA = 39.7%)

• American Indian = 4.5% of the Madera Co. population. (CA = 1.7%)

• Black = 4.4% of the Madera Co. population. (CA = 6.6%)

• Asian = 2.2% of the Madera Co. population. (CA = 13.6%)

• Percentage of high school graduates in Madera Co. = 67.9% (CA = 80.7%)

• Percentage of Madera County population with a Bachelors degree or higher = 13.5% (CA = 30.1%)

• Median household income, 2006-2010 = $46,039 (CA = $60,883)

• Percentage of persons in Madera County living below the poverty level, 2006-2010 = 19.3% (CA = 13.7%)

Madera County Department of Social Services:2011 Madera County DSS Client Profile

• July 2011 (Client Profile) Program and Number of Clients

CALWORKS = 8,782FOOD STAMPS (NON-CALWORKS) = 17,559MEDI-CAL (NON-CALWORKS) = 33,165 CMSP = 2,437GENERAL ASSISTANCE = 114ADULT PROTECTIVE SERVICES = 106IN-HOME SUPPORTIVE SERVICES =1,426CHILD WELFARE SERVICES = 330PUBLIC GUARDIAN = 152Total Number of Individuals Served by our Department = 64,071

Linkages: A Collaborative Effort to Serve the Needs of Clients: An Analysis of the

Implementation of the Linkages Program in Madera County, Department of Social Services

Research Completed By California State University, Department of Social Work Education

Graduate Student: Sarah Whittington in Partnership with Madera County DSS

Background and Impetus

• The current Linkages Program/Policy that exists in Madera County (pilot Linkages project in 2003, formal Linkages policy approved 2011) is the result of many different progressions of research and a combination of various legislative efforts.

• Poverty and Child Maltreatment• Between 2000 and 2009, the number of poor children (children living in families earning 100% or less of the federal

poverty level) increased by 33% (Chau et al., 2010).

• The correlation between poverty and child maltreatment is a phenomena that has been widely studied and has come to be recognized as one the strongest predictors of child maltreatment.

• The Fourth National Study of Child Abuse and Neglect (NIS-4) explains that children from low socioeconomic status households experienced some form of maltreatment at a rate of more than 5 times the rate for children from higher socioeconomic statuses (Sedlak, Mettenburg, Basenaer, Petta, McPherson, Greene, & Li, 2010).

• It is of no great surprise that families experiencing more economic hardships, such as unemployment, homelessness, food insecurity and loss of utility services are at greater risk of coming to the attention of Child Welfare Services .

• It has been shown that within the state of California, 60% of the children involved in Child Welfare Services, have a history of Aid to Dependent Families (AFDC) and/or Temporary Aid to Needy Families (TANF) use (Garrison, 2011).

• Recent federal and state efforts and legislation have moved the Child Welfare System into a more empirical direction, in which specific outcomes based on the effectiveness of the agency in serving clients (safety, permanence and wellbeing) is assessed and reported to the state and federal levels (The 1997 Adoptions and Safe Families Act (ASFA) & The 2011 Child Welfare Systems Outcomes and Accountability Act, also known as AB 636).

• AB 429 mandates continued WTW services for families dually involved in CWS and CalWORKs (CA TANF).

Hypotheses/ Research Question

• Research Question• Is the Madera County Linkages Program/Policy, which includes interagency

collaboration between CalWORKs and Child Welfare Services to holistically serve dual clients with a plethora of service needs, being fully implemented by the parties involved (CalWORKS workers, social workers, supervisors, agency administrators) with current Linkages cases?

• Hypothesis One• If the written Linkages policy in Madera County Department of Social Services is

(or is not) being fully implemented by the designated employees, the quantitative data (secondary data analysis) will show the presence (or absence) of the elements of the policy in the documented work being done for clients.

• Hypothesis Two• If the written Linkages policy in Madera County Department of Social Services is

(or is not) being fully implemented by the designated employees, the qualitative data (worker surveys) will show the presence (or absence) of the elements of the policy in the work they describe being done for clients.

• Research Design/Rationale• Pre experimental, focused on one cohort in one agency, without control group.• Study is exploratory and descriptive in nature, due to sparse research on this topic.

• Instrumentation• The Linkages Case Audit Survey Using Secondary Data Analysis is a quantitative

case audit tool based directly on the Madera County Linkages Program Policy. (Approved by MCDSS leadership)

• The Linkages Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey were developed to collect qualitative data. (pilot studied)

• Procedures• The data used for secondary analysis as outlined in the Linkages Case Audit

Survey Using Secondary Data Analysis was gathered from CWS/CMS and CIV Databases and Child Welfare Services and CalWORKs hard cases.

• The Linkages Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey, the letters of informed consent and posted envelopes were distributed to the Madera County DSS worker participants via Madera County interdepartmental mail and mailed directly back to the student researcher.

Sample-Study Participants

• Secondary Data Analysis• A convenience sample of the 15 Linkages cases open between

9/12/2011 and 10/10/2011 were analyzed.

• Staff Surveys• 50% response rate (11 of 22 surveyed responded)• Participants included: 6 Child Welfare staff, 4 Welfare to

Work Staff and 1 Eligibility Worker (Total of 5 CalWORKs staff).

Data Analysis

• Secondary Data Analysis• The quantitative data gathered from the Linkages Case Audit

Survey Using Secondary Data Analysis was analyzed using the Statistical Package for the Social Services (SPSS) to provide descriptive statistics.

• Staff Surveys• Content analysis, specifically, thematic coding was used to

assess the Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey responses.

• Elements of grounded theory data analysis

ResultsSummary of Quantitative Data Analysis

• 8 of 15 cases had no documentation of the ER social workers consultation or tandem visit with WTW case manager at the onset of the Linkages referral.

• Of the 10 cases audited that did result in a protective hold on the children, only 1 of the CWS cases had documentation of the ER worker inviting the WTW case manager or the EW worker to the detention staffing.

• 14 of the 15 cases analyzed (whether opening to FM or FR), the CWS social worker did arrange for a Linkages Staffing with the family, WTW case manager and EW. However, all but three of these staffing were not held within the designated 7 day period from opening the case.

• A MAD 454 Coordinated Case Plan was documented in 13 of the 15 total cases reviewed.

• 8 of 15 had documentation of the CWS social workers notification to the parents for their attendance at the Linkages Staffing.

• 12 of 15 did not have the Linkages staffing appropriately documented as Family Engagement Efforts (FEE) in the CWS/CMS database.

• 7 of the 9 cases that were maintained in FR, all but 2 cases continued to receive CalWORKs services under AB 429 and had the appropriate FR aid code updated after 30 days.

Sample of Graphs and Tables Generated from SPSS

Sample of Graphs and Tables Generated from SPSS

Sample of Graphs and Tables Generated from SPSS

Sample of Graphs and Tables Generated from SPSS

Sample of Graphs and TablesGenerated from SPSS

Sample of Graphs and Tables Generated from SPSS

Sample of Graphs and Tables Generated from SPSS

ResultsSummary of Qualitative Data Analysis

• All but one of the workers of both CWS and CalWORKs indicated that they had attended at least one Linkages training and many had attended several within the last few years.

• All but two respondents from both CWS and CalWORKs indicated that they did have collateral contact with their counterparts when managing a Linkages case. However, the frequency and purpose of these collaterals was not consistently practiced or understood.

• 7 of the 11 respondents did state that they complete the MAD 454 Coordinated Case Plans and Detention Staffing C-IV Datasheets for all of their Linkages cases.

• The majority of the CWS staff respondents reporting that they indeed had failed to document Linkages activities that they had completed.

• the majority of both the CalWORKs and CWS staff agree that the coordinated case planning has made it easier for Linkages clients to comply with both the CalWORKs and Child Welfare case plans.

• there was mixed responses from both CWS and CalWORKs staff regarding what they saw as the impact of the Linkages Program on poverty and child maltreatment in the Linkages cases.

• Staff members from both CWS and CalWORKs indicated that more ongoing training, flexibility of funding, engagement of parents and increased communication across programs is needed.

Discussion • Both the Linkages Case Audit Survey Using Secondary Data Analysis and The Linkages Child Welfare

Services Social Worker Survey and the Linkages CalWORKs Worker Survey did provide quantitative and qualitative evidence of whether or not the Madera County Linkages Policy/Program is being fully implemented by the designated employees. In turn, the conclusions about these two hypotheses allowed the student researcher to draw conclusions about the overall research question.

• The analysis of the data suggests that the core elements of the Linkages Policy/Program, as written in Madera County Department of Social Services, are being carried out, such as the joint Linkages staffings and the coordinated case planning. However, other parts of the written policy, such as collateral contacts, tandem visits, specific Linkages documentation in the data systems, timely staffing with all required parties present, parental notification and communication regarding the detention staffings are not occurring on a consistent basis and or are not being documented properly.

• Staff also seemed to indicate that they understand the impact of Linkages on individual clients and cases but do not seem to recognize the philosophy behind the creation of Linkages, which includes the recognition of one of the prevailing root issues of child maltreatment, poverty, indicating more training and coaching may be needed.

• Overall, it appears that the core elements of the policy are being carried out; yet, consistent documentation, clear direction and purpose seem to be lacking. Therefore, although great progress has been made in terms of the implementation of the Linkages Program/Policy in Madera County, the quantitative and qualitative data reveals that the Linkages Program/Policy is not being fully implemented at this time.

Implications for the Future• The research findings may lead to more valid outcome studies of the Madera County

Linkages Program in the future, since this study informs the degree of fidelity to the policy itself.

• Also, further study may be done on the financial imperatives of inter-agency/inter-departmental collaboration and the cost savings to counties from reducing duplicate services and providing joint services through more flexible funding streams.

• Through the course of this study the need for more consistent and more worker friendly documentation methods for linkages related activities became apparent.

• Due to the increasingly important role of collaboration between child welfare agencies and other social service agencies for the provision of multiple and complex needs of the clients, this study helps to inform the departments and the academic community about the status of this collaborative effort between CalWORKs and CWS (Linkages) in Madera County in terms of implementing the Linkages policy and program.

• This study also extend the literature on inter-agency and inter-departmental collaboration, a growing filed of study.

– Copy of this PowerPoint– Updated Linkages Coordinators Directory

Follow Up: we’ll email with…

– December 4th 9:00 a.m. to 10:30 a.m. Helping Sanctioned Linkages Families Succeed: Best Practices From the Field

Next Webinar