program monitoring and evaluation: the importance of...

Emmanuel A. Taylor, MSc., DrPHHealth Scientist Administrator / Project Officer

Center to Reduce Cancer Health Disparities

July 15, 2008

Program Monitoring and Evaluation: The Importance of National and Local

Data

Cancer Health Disparities Summit

‘08

Eliminating Cancer Health Disparities Through

Science, Training and Community

Purpose of This Panel SessionInform Audience about the Increased Emphasis on Program Monitoring and Evaluation at CRCHD;

Discuss the Importance of Local Program Evaluation, and

Present Examples of Methodologies being Employed by CRCHD and Grantees in Monitoring and Evaluating CHD Projects

Research and Programs to Reduce Cancer Health Disparities

CUREMICCPCNPPNRP

Training

CNPPNRPMICCP

Population/Community Based

PNRPMICCPCNP

Clinical

MICCPCURE

Basic Research

PROGRAMSCRCHD MISSION /RESEARCH FOCUS

Background

• Local Program Evaluation and Program specific goals and objectives• Completeness of local data for monitoring and national evaluation of

CRCHD programs:-- Annual Progress Reports;-- Data submitted to central program database-- Site visits and Site visit reports-- Other “non-traditional” matrix of

• Re-issuance of CRCHD Programs-- Evidence-based decision-making-- Sustainability of cancer health disparities research

programs

What is “Evaluation”?Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy.

Source: Carol H. Weiss, “Evaluation”, 2nd Edition, Prentice-Hall, Inc, USA. 1998

“ … systematic investigation of the merit (quality), worth (cost-effectiveness), or significance (importance) of an object.”Source: Shadish, W.R., Cook, T.D., Leviton, L.C., “Foundations of Program Evaluation: Theories of Practice. Sage Publications, California, USA. 1991

Monitoring

To check systematically or scrutinize (a program) for the purpose of collecting specified categories of data;

To keep watch over (for the purpose of ensuring that program implementation is on track and performing as intended or planned)

-- Webster’s Dictionary

Levels of Program Evaluation

Local and National Evaluations

Why Local Program Evaluation?

• Monitoring of grantee progress and performance (accountability; periodic and annual progress reports, site visits, communication with PDs, etc.)

• NCI does NOT set local-level targets (Grantees do! Needs Assessment, baseline data…)

• Diversity of Grantees (institutional/individual grants)• Variability of Local situations: Program context, “local stories”,

documentation of lessons learned, barriers overcame, other “qualitative data”

• Feedback to Community and CBO/ partners (CBPR/CBPE)• Credibility and sustainability (scientific merit and more $$$$$$)

Why Both National and Local Program Evaluations

• Document Program accomplishments of goals and objectives: “Aggregated”and/or “Disaggregated” Data

• Complementary (For Complete Picture)• Accountability! (Worthiness!)• Sustainability• Data-based (Management) Decision-making

– CRCHD (Program Monitoring; RFA Re-issuance; etc.)– NCI (EC, BSA); Official Reports (Congressional, etc.)

• Evidence-based Public Health– From Determinants to Possible Solutions (Interventions)– Models that Work (Dissemination and Replication)

The Proof Is In The Pudding!!!

(Not in the Recipe; Not in the Ingredients; But, in the Product!)

What Are Our Products?

(Long-term) Ultimate Product!

Our Collective Mission:Reduction (and Hopefully, Elimination) of

Cancer-related Health Disparities

Locally AND Nationally

What Are Our Products?(Short-term)

Accomplishment of 5-year Program Goals and Objectives

• CNP(CBPR):Established Community Partnership / Coalitions Increased Utilization of Beneficial InterventionsEpidemiologic Shift in Stage of Cancer at DiagnosisProduced More Professionally Competitive InvestigatorsSecured Additional funds for Cancer Prevention and Control

• PNRP (Clinical):Increased Timely Resolution of Abnormal Screening ResultsIncreased Timely Initiation of Cancer Treatment following DiagnosisProduced Competent Patient NavigatorsIncreased participation of Minority Populations in Clinical trials

• MICCP (Basic & Clinical; “Outreach” = CBPR?):Better Collaboration between MSI and CC Enhanced (Cancer) Research Infrastructure at MSIImproved Outreach by CCProduced More Professionally Competitive Investigators

• CURE (Training):Increased the Pool of Minority Trainees in the Pipeline Produced More Professionally Competitive InvestigatorsIncreased the Number/Percentage of Grants Awarded to Minority InvestigatorsIncreased the Number of Publications in Scientific Peer Reviewed Journals

Local Data Related Issues• Data submission (frequency and timeliness)• Training of data collection and data entry staff (error

rates, proactive, …)• Completeness of data (baseline, follow up,

demographic of participants, etc.)• Quality of self-reported data• Annual Progress Report format (measures / matrix of

progress)• Monitoring and On-going communication with PD –

be proactive, program improvement / adjustments• Sustainability (dissemination – community, local

decision makers

PROGRAM BUILDING BLOCKS

FOCUS OF ACTIVITIES SHORT-TERM (PROCESS) (1-2 years)

INTERMEDIATE (IMPACT)(3-5 years)

Establish and maintain infrastructure

to address cancer disparities



Develop and conduct community-based:

• Education program• Activities



Predisposing Factors

• Improve patient and public knowledge, beliefs, attitudes, value and perceptions about cancer related issues across the continuum of care



NCI provides program management and

technical assistance to promote program

improvements



improvements

LONG-TERM(5-7 years)

ULTIMATE(8+ years)

Develop and conduct community based:

• Provider training• Recruitment and

training of minoritystudents

• Relevant research(i.e. pilot projects)





Develop and conduct strategies to educate

policy makers


policy makers

Create and/or enhance local partnerships with

community-based organizations to

assess community resources and their value to program




Create and/or enhance partnerships with

organizations that can help reduce disparities in the community (e.g.,

private and government groups,

policy makers)




policy makers)

Develop NCI collaborations

(including with CIS) to expand local capacity



Reinforcing Factors

• Increase health professional knowledge and sensitivity related to cultural compassion

• Increase understanding of issues impacting cancer control among disparate populations

• Mobilize community to support efforts of CNP (i.e., address and improve community norms)

Reinforcing Factors




Reinforcing Factors

• Increase understanding among policy makers of issues impacting cancer control among disparate population

Reinforcing Factors


OUTCOMES

Individual Change

• Increase positive health behaviors (e.g., smoking cessation, improve nutrition, etc.) among disparate populations

• Increase utilization of screening diagnosis, treatment, and clinical trials services

Individual Change



Community Change(Enabling Factors)

• Improve local referral patterns, including to clinical trials

• Improve provider interactions with disparate groups

• Increase number of health professional representing disparate populations

• Increase access to preventive, screening, diagnostic and treatment services

• Leverage funding from other sources to enhance services

• Translation of research to practice








Policy Change(Enabling Factors)

• Implement effective policies to: increase access to such things as insurance, state cancer programs, Medicaid/Medicare coverage, adequate clinical care, etc.



Reduce disparities through:

• Achieving a shift in diagnosis from later to earlier stage

• Improve cancer survivorship

• Narrowing the gap between the discovery, development , and delivery of care for disparate populations and that for other groups

• Achieving sustainability of efforts for disparate groups






Decrease morbidity

from cancer among

disparate populations to equal

rates among other

groups

Decrease morbidity

from cancer among


rates among other

groups

Evaluation findings used to enhance program effortsEvaluation findings used to enhance program efforts

CNP Conceptual Framework

Develop, through a community-based

participatory process, an

understanding of relevant resources, assets, and needs to address cancer

disparities by creating a synergy

of efforts and leveraging local

resources

What data are required for Program Monitoringand Evaluation?

SOURCE (LEVELS)

DATA NEEDEDMEASURESGOALS AND OBJECTIVESPROGRAM

Local Level• Grantee Organization/ Institution (PIs)

–Annual Progress Report–Intermittent on-going communication with PD

•Grantee Partners/ clinics, CBOs, State HDs, etc.

(Disaggregated)

•CNP Organizational infrastructure (staffing and data management resources)•Number and types of formal partnerships formed (MOUs) – CBO, CIS, Primary and secondary prevention facilities•Composition of CAGs•Community-based needs assessment results and methodology•Community profile: demographics•Baseline data on targeted cancer disparity•Baseline data on utilization of beneficial cancer interventions at partner sites•Number and types of CBPR activities conducted

“Our Products!”

•Establish Community-based Partnerships

•Increase utilization of beneficial cancer interventions

•Significantly improve access to beneficial cancer interventions

•Train junior investigators in CBPR

•Ensure Sustainability

CNP


SOURCE (LEVELS)

DATA NEEDEDMEASURESGOALS AND OBJECTIVESPROGRAM

Local Level• Grantee Organization/ Institution (PIs)

–Annual Progress Report–Intermittent on-going communication with PD

•Grantee Partners/ clinics, CBOs, State HDs, etc.

(Disaggregated)

•Follow up data on utilization of beneficial intervention at partner sites•Number, demographics, and discipline of junior investigators•Training of junior investigators•Number of pilot projects submitted and awarded (research focus, etc)•Additional non-CRCHD funds leveraged, by source of award•Race and ethnicity, and other demographics of program participants•Applicable State level data (for comparison)

“Our Products!”

•Establish Community-based Partnerships

•Increase utilization of beneficial cancer interventions

•Significantly improve access to beneficial cancer interventions

•Train junior investigators in CBPR

•Ensure Sustainability

CNP (Cont’)

SOURCE (LEVELS)

DATA NEEDEDMEASURESGOALS AND OBJECTIVES

PROGRAM

Local Level•Grantee Organization / Institution (PIs)

–Annual Progress Reports–Intermittent on-going communication with PD

•Clinic and hospital partners•NGO partners – ACS

National Level•NCI database•Program database –Contractor, NOVA

–Site Visit Report

(Aggregated, Summary)

•Duration between abnormal screening result and resolution•Percent by diagnoses•Stage at diagnosis•Duration between diagnosis and initiation of treatment•Quality of cancer care•Number and demographics of PNs•Number of PNs trained•Mode of training•Results of training sessions (scores)•Types and categories of needs / health care barriers resolved, by PN•Length of time to resolve needs / health care barriers, by PN

“Our Products”

•Increase timely resolution of abnormal screening results

•Increase timely initiation of cancer treatment following diagnosis

•Produce Competent Patient Navigators

•Increase participation of racial and ethnic minority and underserved populations in clinical trials

PNRP


SOURCE (LEVELS)


PROGRAM



•Clinic and hospital partners•NGO partners – ACS

•Competency (in-service) assessment and Mode of assessment•Patient Satisfaction with cancer care and navigation•Recruitment and enrollment in PNRP (and other clinical trials)•Race and ethnicity, and other demographic data of study participants (case and control)•Cost-effectiveness of patient navigation services•Title, journal reference and cancer focus of publications, by author•Other healthcare system changes

“Our Products”

•Increase timely resolution of abnormal screening results

•Increase timely initiation of cancer treatment following diagnosis

•Produce Competent Patient Navigators

•Increase participation of racial and ethnic minority and underserved populations in clinical trials

PNRP (con’t)


SOURCE (LEVELS)


PROGRAM



•Clinic and CC partners

National Level•NCI Portfolio Analysis

•Number of joint grant applications submitted by the partner institutions•Number of grants awarded to partner institution, by Source and mechanism of funding•Technical Merit score of each application submitted, name and affiliation of PI•Portion of grant to MSI and CC•Type and title of research, cancer focus, etc.•Size and demographic profile of study populations•Institutional support (MSI, CC)•Composition of a Community Advisory Board (CAB)

“Our Products”

•Establish and improve Collaborations between MSI and CC

•Enhance (cancer) research infrastructure at MSI

•Improve outreach by CC

•Increase the pool of professionally competitive investigators

MICCP


SOURCE (LEVELS)DATA NEEDEDMEASURES

GOALS AND OBJECTIVESPROGRAM



•Clinic and Cancer Center partners

National level•NCI Portfolio Analysis

–Need timely data, especially for annual re-issuance

•Type and outcomes of outreach activities by CC•Number of partnerships with CBO•Number, demographic profile and academic rank of students trained•Mentor name, faculty rank, and institutional affiliation•Number, demographic profile and academic /faculty rank of junior investigators trained•Grant applications submitted by junior investigators (number, title, cancer focus, funding source and amount)•Grants awarded to junior investigators

“Our Products”

•Establish and improve Collaborations between MSI and CC

•Enhance (cancer) research infrastructure at MSI

•Improve outreach by CC

•Increase the pool of professionally competitive investigators

MICCP (cont’)

What data are required for Program Monitoring and Evaluation?




–Annual Progress Reports–Intermittent on-going communication with PD–Trainee tracking data–Exit interviews


•Number, demographic profile and academic rank of students trained•Mentor name, faculty rank, and institutional affiliation•Number, demographic profile and academic /faculty rank of junior investigators trained•Grant applications submitted by junior investigators (number, title, cancer focus and amount)•Grants awarded to junior investigators, by funding source and mechanism•Current location, rank/ employment, research focus of former trainees

“Our Products”

•Increase the pool of investigators representative of diverse populations in the pipeline

•Produce more professionally competitive investigators

•Increase the number/ percentage of grants awarded to the diverse group of investigators

CURE





–Annual Progress Reports–Intermittent on-going communication with PD–Trainee tracking data–Exit interviews


•Entry and exit dates of trainees (MICCP funded period)•Mentor name, faculty rank, and institutional affiliation•Graduation rate and major of students trained•Current location, employment/ rank, research focus of junior investigators sponsored•Title, journal reference, and cancer focus of publications, by author•Number of total grants awarded by NCI/ NIH, by mechanism, by demographic profile of awardees

“Our Products”

•Increase the pool of investigators representative of diverse populations in the pipeline

•Produce more professionally competitive investigators

•Increase the number/ percentage of grants awarded to the diverse group of investigators

CURE (Cont’)


PROGRAM BUILDING BLOCKS

FOCUS OF ACTIVITIES SHORT-TERM (PROCESS) (1-2 years)

INTERMEDIATE (IMPACT)(3-5 years)















improvements



improvements

LONG-TERM(5-7 years)

ULTIMATE(8+ years)










policy makers


policy makers










policy makers)




policy makers)





Reinforcing Factors




Reinforcing Factors




Reinforcing Factors


Reinforcing Factors


OUTCOMES

Individual Change



Individual Change































Decrease morbidity

from cancer among


rates among other

groups

Decrease morbidity

from cancer among


rates among other

groups

Evaluation findings used to enhance program effortsEvaluation findings used to enhance program efforts

CNP Conceptual Framework

Develop, through a community-based

participatory process, an

understanding of relevant resources, assets, and needs to address cancer

disparities by creating a synergy

of efforts and leveraging local

resources

Remember:

“The Proof is in the Pudding!”

But,We Need Data and Cooperation!!!

Thank You !!!!

program monitoring and evaluation: the importance of...

Documents