Developing and pilot-testing a culturally-competent evaluation

model for minority health projects in Ohio

Manoj Sharma, University of CincinnatiRobert Fischer, Case Western Reserve Univ.Carla Clasen, Wright State UniversityPeter Leahy, University of Akron Betty R. Yung, Wright State UniversityLucinda Deason, University of Akron Fatima Perkins, United Way of Greater Cleveland

Session 3369, Board 2November 6 (M), 2006, 4:30 pm

Background of the Ohio Commission for Minority Health (OCMH)

In July 1987, the Ohio General Assembly, passed Amended Substitute House Bill 171, creating the Ohio Commission on Minority Health (OCMH)

The Commission is an independent state agency with a biennial appropriation of approximately $3.5 million from general revenue funds.

Innovative and culturally specific projects are funded for up to $200,000, for a two-year period.

Between 1988-2001 OCMH funded grants totaling approximately $12 million

OCMH has funded organizations in 22 counties within Ohio during the past 18 years

Ohio Commission on Minority Health: Mission

The Ohio Commission on Minority Health is dedicated to eliminating disparities in minority health through innovative strategies and financial opportunities, public health promotion, legislative action, public policy and systems change.

Website: http://mih.ohio.gov/index.stm

Present Priority Areas for Funding of OCMH Demonstration Projects

Cardiovascular disease (including Hypertension)

Cancer Diabetes Infant mortality Substance abuse Violence

Grants Funded by the OCMH

Demonstration Projects– Innovative and culturally specific projects that are funded for up to

$200,000, for a two-year period in one or more of the six diseases/ conditions/ risk factors,

Minority Health Month– Grants for conducting this celebration in April for an award of $2,000.

Lupus– Grants for lupus programs with a maximum of $14,000.

Health Priorities Trust Fund– Resulted from the Tobacco Master Settlement Agreement. Supports

Community Health Grants for up to $200,000 and Scientific and Community Partnership Grants for up to $400,000.

Focus of this project has been to improve the evaluation ofdemonstration projects.

Major Program Activities of OCMH Funded Projects: 1988-2001

Providing health education– Focusing on “hands-on” skill building

Providing health screenings– Primarily focusing on health screenings for cancer

Increasing access to health care– Providing public awareness campaigns– Providing referral & follow-up– Improving relations with primary care providers– Building cultural competence in health care providers

Distribution of OCMH Grants by Race: 1988-2001

Target Population Number of projects (n)

%

African Americans 126 66.0

Hispanic Americans 35 19.0

Asian Americans 13 7.0

Native Americans 4 2.0

Multiple ethnic groups 11 6.0

Distribution of OCMH Grants by Type of Organization: 1988-2001

Multipurpose CBO

Community healthagency

Health departments

Universities

Other

Ohio Commission on Minority Health: View on Evaluation

Evaluation methods should be appropriate to the intervention model.

Each evaluation should include measures of both process outcomes (implementation) and behavioral/health status outcomes (changes in participants).

Evaluation is seen as a vehicle for program improvement (internal use) and program accountability (external use).

Evaluation findings from OCMH-funded projects should contribute to the knowledge base of what works for minorities.

Purpose of this Project

OCMH identified a need to streamline the evaluation process being used by demonstration projects that could be adopted by all funded projects, thereby allowing for cross comparisons between projects.

The purpose of this project was to develop a culturally-competent and practically-feasible evaluation model specific to projects serving minority populations in Ohio and make recommendations for its implementation.

Evaluation Focus

Cultural competence: Behavioral (knowledge, attitudes, and skills) and environmental (policy & organizational change) level initiatives that help communication and effective relations across and among different cultures

Culturally-competent evaluation: Emphasis on community participation before, during, and after evaluation

Methodology for Streamlining Evaluation of OCMH Projects: 2005-2006

State-wide search & selection of six evaluators with expertise in working with minorities & program evaluation: May 2005

Review of existing models: June-July 2005 Selection of outcome measures & model: August-

September 2005 Evaluation Guidelines including changes to RFP &

Reporting developed: Sept. 2005-June 2006 Projects begin implementation of new guidelines at

next funding cycle (FY 2006 – 2007)

Recommendations for Specific Changes Related to Evaluation

1) Pre-proposal period (modifications to the RFP, technical guidance, pre-identified pool of evaluators, writing support)

2) Proposal review process (technical comments on evaluation plan by evaluation professionals)

3) Post award support (technical assistance)

4) Grant period (annual technical assistance meeting)

5) End of grant period (clear guidelines for evaluation report)

Salient Highlights of Evaluation Guidance Document

Presents a practically-feasible model Focuses on culturally-competent evaluation Addresses issues of health literacy Specifies key considerations in selecting an

evaluator, planning for an evaluation budget, choosing a design and instruments, and securing permissions, as needed

Provides examples of outcome indicators and sample instruments

Guidance Document’s Content Highlights

Rationale for Evaluation: Evaluation helps to determine if programs really make a difference in the health and lives of people in Ohio.

Cultural Issues in Evaluation: Culturally competent evaluation engages the community being evaluated with the evaluation itself. This evaluation approach places an emphasis on community empowerment or community participation is necessary for a culturally competent evaluation.

Guidance Document’s Content Highlights Cont’d

Given the increasing size of the ethnically/racially diverse populations and various cultures, intervention programs and evaluation designs must be culturally competent. Nearly all work involves communication or the act of sharing information, yet we are rarely taught how to communicate effectively, in a multicultural setting. Program staff and evaluators must develop the skills necessary for obtaining cultural information from participants, yet bearing in mind that each participant is different and may not share all of the cultural values of her/his cultural group.

Prior to designing a program and evaluation that will help to reduce health disparities, the project director and evaluator must take into consideration the cultural context for the intervention and the participants’ past experiences with the health care system.

Various strategies can be employed when conducting a culturally competent evaluation, including Pre-Evaluation and Post-Evaluation work which are very crucial.

Guidance Document’s Content Highlights Cont’d

Health Literacy: Health literacy is the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions (National Library of Medicine, 2000).

Although low health literacy is more prevalent among people who completed fewer years of education, persons of certain ethnic/racial groups, the elderly, and individuals with low cognitive ability, the Institute of Medicine (2004) report noted that people of all ages, races, income and education levels are challenged by low health literacy.

Instruments have been developed for measuring health literacy levels.

Some interventions that have helped to improve knowledge for the low health literacy population include using videotaped educational tools, illustrated materials, and designing pamphlets for easy readability.

Guidance Document’s Content Highlights Cont’d

Evaluation Logistics: The evaluation logistic section of the Guidance Document provides information about selecting an evaluator, the evaluation budget/contract, and the institutional review (IRB) board process.

Guidance Document’s Content Highlights Cont’d

Evaluation Approaches: This section of the Guidance Document consists of information about evaluation designs, instrumentation, reliability, and validity as well as data collection methods.

General Logic Model: Basic Depiction

Inputs Activities Outputs Outcomes Outcome Indicator(s)

What is used to deliver services

Things done to, for or with the target population intended to lead toward desired change

Number of people served, hours of instruction provided and received etc.

Benefits accrued as a result of the activities conducted

What we track and use to measure the benefits to program participants

An Example of Outcomes and Outcome Indicators

Type of Program

Outcome Outcome Indicator (s)

Cancer Education/ Behavior Change

Increased knowledge about cancer prevention strategiesIncreased number of people receiving preventive screenings for cancer

80% of program participants will demonstrate increased knowledge about how healthy diet choices reduce the risk of cancer based on a self-report test.30% increase in number of women over 40 in the target population who have received a mammogram in the past two years based on

self-report.

Concluding Thoughts

The evaluation model will be implemented beginning with grantees in the 2006-2007 OCMH funding cycle.

The model offers great potential for streamlining data collection, data analysis, and reporting of evaluation results from the health promotion projects serving minority populations in the state of Ohio.

The model contains many exportable components that can be replicable in Ohio and other states for projects pertaining to ethnic minority populations.