Creating National Performance Indicators that are Relevant to Stakeholders:

Participatory Methods for Indicator Development, Selection, and Refinement

Demia L. Sundra, M.P.H., Margaret Gwaltney, M.B.A., Lynda A. Anderson, Ph.D., Ross Brownson, Ph.D.,

Jennifer Scherer, Ph.D.

Evaluation 2004November 3, 2004

Contributors to Project DEFINE [Developing an Evaluation Framework: Insuring National Excellence]

Evaluation Advisory GroupRoss Brownson, Ph.D. (co-chair)

Alan Cross, M.D.Robert Goodman, Ph.D., M.P.H.

Richard Mack, Ph.D.Randy Schwartz, M.S.P.H.

Tom Sims, M.A.Avra Warsofsky

Carol White, M.P.H.

CDCLynda A. Anderson, Ph.D., Robert Hancock, Demia Sundra, M.P.H.

COSMOS CorporationJennifer Scherer, Ph.D., Margaret Gwaltney, M.B.A. (Currently with

Abt Associates), Thérèse van Houten, D.S.W.,Cynthia Carter

Concept Systems, Inc.Dan McLinden, Ed.D., Mary Kane

Goals of Presentation

Describe the context in which the performance indicators for the CDC’s Prevention Research Centers (PRCs) program were developed

Review the participatory methodology utilized to develop and select national indicators

Discuss the benefits, challenges, and lessons learned from developing performance indicators in an established, diverse, national program

Background and Context

Features of CDC’s Prevention Research Centers Program

33 academic-based extramural research centers across United States

Academic centers partner with community and state organizations to develop, conduct, and disseminate prevention research through participatory methods

Diversity across centers: When founded (newly funded to 18 yrs) Community setting and partners Focus of research

Context for Developing a National Evaluation

National evaluation planning project initiated in response to: Institute of Medicine (IOM) report on the PRC program Support for evaluation at CDC Growth of program and related need for accountability

Project DEFINE Goals (Planning Phase) Engage stakeholders, develop a logic model and

performance indicators, and draft an evaluation plan Maintain a participatory and utilization-focused

approach throughout

Intended Purposes of Performance Indicators

Individual data on each PRC Evaluation Monitoring Technical assistance needs

Cross-center summary data Accountability Program improvement Information sharing and communications with internal

and external stakeholders

Anticipated Challenges in Developing PRC Indicators

Centers strive to achieve diverse health outcomes Program had few previous cross-center

requirements Centers are at various stages of growth and

maturity Interests of diverse stakeholders had to be

considered Concern existed about how performance

indicators would be used Indicators had to be meaningful and impose

minimal burden on PRCs in terms of time and cost

Methodology

Basis of Project DEFINE

Concept Mapping Gained national and community perspectives on

PRC program through 2-tiered approach Engaged diverse stakeholders in brainstorming

statements describing PRC program Statements analyzed to create

visual maps of concepts Concepts used to

build draft logic models

Community and national logic models combined

Engage the Community

Diversity & SensitivityRelationships & Recognition

Active Dissemination

Technical Assistance

Training

Research Methods

Research Agenda

Core Expertise & Resources

Development of Draft Indicators

More than 70 indicators first drafted

Mapped to all components of program logic model

Some indicators dropped, others refined based on input received at regional meetings and PRC contextual visits

52 candidate indicators remained on the list

National, Regional, or Local Health

Priorities and Health DisparitiesEngage

theCommunity

Establisha Research

Agenda

Conduct Core and

Other Research

Using SoundResearch Methods

ProvideTraining, Technical

Assistance, orMentoring

• Researchers• Practitioners• Students• Community

Members

PRC Community Committee

PRC Capacity • Human resources:

core expertise, diversity, sensitivity

• Facility• Communication and

data systems• Administrative

capacity• Evaluation expertise

Relationships with Partners• State and Local Health Departments• Community Partners• University Partners• Other PRCs• CDC

Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust)

Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust)

Research andEvaluation FindingsCommunicated and

Disseminated

• Publications • Presentations• Media• Reports

Research andEvaluation FindingsCommunicated and

Disseminated

• Publications • Presentations• Media• Reports

Trainees andTechnical Assistance

Recipients

Programs and Interventions

OUTCOMES

Improved Community

andPopulation

Health and

Elimination of Health

Disparities

Improved Community

andPopulation

Health and

Elimination of Health

Disparities

EnhancedCommunity

Capacityfor

Prevention

Translation of Research

to Practice and Policy

Widespread Use of

Effective Programs and

Policies

Skilled Public HealthProfessionals

INPUTS ACTIVITIES OUTPUTS

Evaluation

ExpandedResources Recognition

CONTEXTUAL CONDITIONS(e.g., health services and service gaps, socioeconomic conditions)

Stakeholder Recommendations from Regional Meetings

Select a limited number of indicators focused on features common across Centers

Collect data on some components of logic model in other ways as part of the national evaluation, rather than through indicators

Develop indicators through an iterative process, with multiple opportunities for input

Link the performance indicators to the PRC Information System

Stakeholder Selection of the National Performance Indicators

52 indicators listed in structured feedback tool (workbook) All stakeholder groups provided feedback and comments

PRCs, Community, State, and CDC Planned on having core and optional indicators

Results of Performance Indicator Feedback

100% response rate received on workbooks Comments from workbook summarized within

each stakeholder group 3 of the 4 stakeholder groups recommended 8

indicators 2 of the 4 stakeholder groups recommended an

additional 11 indicators

Resulting National Performance Indicators

Evaluation advisory group selected and refined 13 indicators based on Results and feedback of workbook Map of indicators across the logic model Cross-walk of recommended indicators with IOM

report recommendations

Indicators correspond to various logic model components, e.g. Community input on selecting health priorities

(input) Existence of explicit research agenda (activity) Evidence of peer-reviewed publications (output)

Collecting the Information

Performance indicators were integrated into the information system that was in development Conceptualized from the beginning Reinforced through stakeholder feedback

Fields were created in the information system for each performance indicator

Information system was developed and reviewed by: Evaluation Contractors Centers and partners (usability and pilot tests) CDC staff Evaluation Advisory Group PRC Steering Committee

Core and Optional Indicators

Only core indicators developed through Project DEFINE. Consensus allowed us to: Focus on 13 indicators Not use resources for optional indicator development and

integration into information system

11 out of 28 PRCs developed center-specific indicators on their own Topics areas such as

community satisfaction with partnership; funding generated; web site hits; infrastructure measures; research methods appropriate for minority population

PRC Performance Indicators: Summary

Specific component requirements across all grantees

Indicators reflect both process and outcome measures, with focus on process Initial requirements as part of new funding cycle Prospective evaluation

Assess general information across PRCs rather than specific health topics Defining common outcomes, e.g. community capacity

Indicators will be refined during evaluation implementation

Challenges, Benefits, and Lessons Learned

Current Challenges with Performance Indicators

Requests for more guidance on how to further define and collect data

Development of summary reports and provision of feedback to all stakeholders

Need to increase specificity of indicators over time

Balance between participatory processes and program requirements

Buy-in, support, and ownership of indicators Evaluation advisory group was critical for trust

and support from larger stakeholder groups

Community voice is reflected in indicators

Perspective of the PRCs’ staff and partners reflected in utility and feasibility issues surrounding indicators and information system

Benefits of Participatory Approach for Performance Indicator Development

Lessons Learned and Recommendations

Utilize participatory methods for selecting and refining indicators to increase stakeholder support

Build sufficient time into schedule to allow multiple opportunities for stakeholder input

Acknowledge inherent challenge in developing indicators for an established program

Include community input in indicator development to increase accountability to partners

For more information on the Prevention Research Centers Program

http://www.cdc.gov/prc/

Click on “about the program” to view the Conceptual Framework (logic model) and narrative

Contact information: Demia Sundra: dsundra@cdc.gov

PRC IS: General Information Page

PRC IS: Community Committees

PRC IS: Health Priorities