CONSISTENT PROTOCOL, UNIQUE SITES: SEEKING CULTURAL COMPETENCE IN A

MULTISITE EVALUATION

Carolyn Sullins, Ph.D. Ladel Lewis, Ph.D. candidate

The Kercher Center for Social ResearchWestern Michigan University

National Study Comprehensive Community Mental

Health Services for Children and Their Families Program: “Systems of care.”

incorporates a broad, flexible array of effective services and supports for a defined, multi-system population that is organized into a coordinated network… is culturally and linguistically competent, builds meaningful partnerships with families and youth at service delivery, management and policy levels, and has supportive policy and management infrastructure. (Pires, Lazear, & Conlan, 2008).

Systems of Care 144 sites have been or are in the process

of being evaluated. Each 1-2 year cohort: +/- 30 sites.

Each SoC has distinct: geographic location and scope (e.g.,

statewide, county wide, city-wide, tribal) Ages of the youth served Mental health issues facing the targeted youth Racial, ethnic, and cultural factors

Core Values Child centered

and family driven

Community based

Culturally competent

Common Variables Presenting issues of the youth Youth’s level of functioning (strengths

and weaknesses) Family strengths and barriers Types of services family and youth are

receiving Satisfaction with services Cultural competence of services Youth and family input into services

Barriers to Evaluation Implementation

“One size fits all” battery of questions

HSIRB mandates re language on consent forms

Ensuring an adequate sample size

Ensuring retention in a mobile population

Barriers to Participation History of racist abuse by researchers (E.g.,

Tuskegee syphilis study) Misinterpretation of data, or no access to

results Sensitive or stigmatized topics even more

difficult  Families overwhelmed or embarrassed

Local System of Care:“Kalamazoo Wraps”

9%

90%

Insurance Private InsPublic Insn=244

Caucasian: 49.1%(including White Latino/a: < 5%)

African-American/Multiracial: 50.9%

Ages 7-17

Diagnosed with a Severe Emotional Disturbance

Academic/Professional vs. Local Population Perspectives

Informed consent

Voluntary participation

Confidentiality and its exceptions

Evaluation Work GroupParents, various social service workers, eval

staff Reviewed consent forms for clarity Gave opinions to HSIRB re child abuse

reporting Parent input re: communication among

clinicians, families, and interviewers Interpretation of data Reporting of results

Youth Group We came to their

group Help re local

language Helped us make it

more comfortable for participants

Info that later helped us interpret data

Academic Culture vs. Local Communities’ Culture

Does confidentiality mean… No interviews in public

places, even if that’s what participants request?

Kicking Grandma out of the room?

Pretending you don’t see participant in public?

Problems with Interviews Some found it emotionally draining Too long and redundant

Questionnaires with overlapping questions Categories of services – national vs. local

terms Keeping in touch with families every 6

months

Easing Stressful Processes “Evaluating System of

Care – not you” Yet up front about sensitive,

personal questions Non-judgmental attitude

for better rapport, retention, AND accuracy. Balance – we can’t act as

friends or counselors! If SoC not working, or not

working with all groups of people, we need to know.

Interviewers: Pay Attention – So Participants Will Too!

Offer breaks, gum, stress balls, etc. to participants

Coloring books, DVDs for young kids

Redundant questions from multiple surveys: propriety and accuracy trump methodological “letter of law.”

Keeping in Touch with Families Family address tracking

form Birthday and holiday

cards with coupons Incentives for families

to contact us Annual dinner as a

“thank you” Results in bimonthly

newsletter, website, and other venues

Recruitment/Retention Results

Recruitment Rates (N=224)

77.2%

22.8%

African Americans

Partcipants

Non-Partic-ipants 80.9

%

19.1%

Caucasians

Participants

Non-Partic-ipants

Chi Square (1, N=224) = .466, p = .495

6 Month Interviews (N = 145)

72.10

27.9%

African American

Partic-ipants

Non-Partic-ipants 77.9%

22.1%

Caucasian

Partic-ipants

Non-Partic-ipants

Chi Square= (1, N= 145) = .665, p = .415

12 Month Interviews (N=119)

63.2%

36.8%

African American

Partic-ipants

Non-Partic-ipants 69.4%

30.6%

Caucasian

Participants

Non-Partic-ipants

Chi Square= (1, N=119) = .511, p = .561

18 Month Interviews (N=91)

59.6%

40.4%

African American

Participants

Non-Partic-ipants 56.8%43.2%

Caucasian

Partic-ipants

Non-Partic-ipants

Chi Square= (1, N=91) = .071, p = .834

24 Month Interviews (N=52)

47.8%52.2

%

African American

Partic-ipants

Non-Partic-ipants

62.1%

37.9%

Caucasian

Participants

Non-Partic-ipants

Chi Square= (1, N=52) = 1.055 , p = .402

Conclusion Fulfilling national

evaluation requirements within a local context is a constant balancing act.

With enough input from local stakeholders at each stage, it can be achieved!

Questions? Comments? Please contact

carolyn.sullins@wmich.edu

Or Ladel_lewis@yahoo.com

For more info, please see also http://www.wmich.edu/sociology

/kzoowraps.html