evaluating community-based participatory research

EVALUATING COMMUNITY-BASED PARTICIPATORY RESEARCH OCTOBER 2015

Curtis Nash* *Research Shop Intern

Citation: Nash, C. (2015). Evaluating Community-Based Participatory Research. Guelph, ON: Community Engaged Scholarship Institute. https://atrium.lib.uoguelph.ca/xmlui/handle/10214/8902

https://atrium.lib.uoguelph.ca/xmlui/handle/10214/8902�

Evaluating Community-Based Participatory Research

p.2

Table of Contents Introduction ................................................................................................................... 3

Methods ......................................................................................................................... 3

What Do We Mean By “Evaluation?” ........................................................................... 4

Notes About The Resources And Tabular Approach ................................................. 4 Public Engagement With Research And Research Engagement With Society (PERARES) ... 4 Community Impacts of Research-Oriented Partnerships (CIROP) ......................................... 5 Access Alliance Multicultural Health & Community Services (Access Alliance) ...................... 5 Native American Research Centers for Health (NARCH V) .................................................... 5

Table 1: Summary of Questions Asked in Evaluations .............................................. 7

Table 2: Summary of Indicators Used in Evaluations ................................................ 8

Table 3: Summary of Measures Used in Evaluations ................................................. 9

Evaluation Questions Worth Noting .......................................................................... 10 Start-Point Evaluation:...........................................................................................................10 Mid-Point Evaluation: ............................................................................................................10 End-Point Evaluation: ............................................................................................................10 Post-Project Evaluation: ........................................................................................................10

Conclusions ................................................................................................................. 11

References ................................................................................................................... 12

Appendix: Selected Frameworks & Logic Models.................................................... 15


p.3

INTRODUCTION This project, an internal initiative prepared for the Community Engaged Scholarship Institute and The Research Shop, explores the evaluation mechanisms found within the literature that are used to assess the effectiveness of community-based participatory research projects.

The aim of this project was to review the questions, indicators, and methods used to assess the effectiveness of community-based participatory research projects for all parties involved – Research/Science Shops, Community Partners, and Student Researchers.

This report has two main sections. The first section provides the contextual lens through which to understand the second section through providing a background on evaulation and discussing the core resources and approach used in this report. The second section synthesizes the questions, indicators, and methods referenced within the literature into three tables.

It is hoped that the following information will provide readers with a preliminary understanding of the evaluation practices commonly found in community-based research institutes. This information is intended to inform any revisions that the Community Engaged Scholarship Institute and The Research Shop consider in their program evaluations.

METHODS This project conducted a literature scan to find the relevant academic and grey resources available relating to community-based participatory research evaluations. Google, Google Scholar, and Primo were used as the main search engines for this work. The search queries used included:

• “Community-Based Research” AND “Evaluate” OR “Evaluation”

• “Community-Based Research” AND “Assess” OR “Assessment”

• “Community-Based Research” AND “Measure” OR “Measurement”


p.4

• “Community Engagement” AND “Evaluate” OR “Evaluation”

• “Community Engagement” AND “Assess” OR “Assessment”

• “Community Engagement” AND “Measure” OR “Measurement”

• “Community-University Partnership” AND “Research”

• “Community-University Partnership” AND “Research” AND “Evaluate”

• “Participatory Research” AND “Evaluate” OR “Evaluation”

• “Participatory Research” AND “Assess” OR “Assessment”

• “Participatory Research” AND “Measure” OR “Measurement”

WHAT DO WE MEAN BY “EVALUATION?” Before examining the questions, indicators, and measures used in evaluations, it is worth briefly noting what evaluation refers to in order to provide some context.

“Evaluation” refers to a means through which the value of something is determined. It is a process where achievements, and the tactics through which these achievements were made, are assessed in a systemic manner. For community-based participatory research projects, it involves a critical analysis that asks what went well, what went poorly, and how the process and outcomes of a similar project could be improved upon next time.

NOTES ABOUT THE RESOURCES AND TABULAR APPROACH The queries used in the literature scan yielded a number of both academic and non-academic works of relevance for the purposes of this project. Four were of particular importance and comprise a significant portion of the tables presented on the following pages. As such, a brief overview of each of these four materials is worthwhile.

Public Engagement With Research And Research Engagement With Society (PERARES) PERARES was a four year project that began in 2010 and was funded by the European Community’s Seventh Framework Programme. Its aim was to involve researchers and Civil Society Organizations in the development of research agendas and research processes focused on improving public engagement in research activities. Three


p.5

documents focusing on evaluating community-based participatory research projects that emerged from PERARES are of particular use and have been heavily referenced in the tables below. An in-depth review of these three documents would be worthwhile in any future consideration of program evaluation revisions.

Community Impacts of Research-Oriented Partnerships (CIROP) Developed by a team of academics, CIROP is a 33-item tool that measures the perceived impacts of research partnerships among community members specificially in the social and health service sectors. It measures these impacts along four main areas: Personal Knowledge Development, Personal Research Skill Development, Organizational/Group Access To and Use of Information; and Community and Organizational Development. The likert-scale questions it poses within its survey have taken a prominent place in Table 1 below.

Access Alliance Multicultural Health & Community Services (Access Alliance) Based in Toronto, Access Alliance is a community health centre that promotes health and well-being services and attempts to improve the access immigrants and refugees in Toronto have to these services. In 2004 it established a community-based research program and has constructed a number of tools to help organizations undertake partnerships with marginalized communities effectively. Its Community-Based Research Toolkit has been particularly useful, with a range of chapters included discussing the Planning, Ethics, and Evaluation surrounding community-based participatory research.

Native American Research Centers for Health (NARCH V) The National Congress of American Indians Policy Research Center, the University of New Mexico Center for Participatory Research, and the University of Washington Indigenous Wellness Research Institute have partnered to undertake a four year project aimed at understanding the ways communities are engaged for health research initiatives. NARCH V explores the quantitative and qualitative methods that can be used to evaluate community-based participatory research projects, and has developed a matrix listing a significant number of questions used to evaluate these.

A number of academic articles discussing evaluations of community-based participatory research projects were also explored.


p.6

The findings from these materials have been depicted in the tables on the following pages. It is important to note that the questions listed in Table 1 are meant to be representative of the diverse phrasings of the questions used in each of the individual materials. Although the questions used in each of the materials might have different wordings, the main premise behind the questions are captured in those listed in Table 1.

TABLE 1: SUMMARY OF QUESTIONS ASKED IN EVALUATIONS PARARES CIROP Access Alliance NARCH V Braun Et

Al. (2012) Start-Point Evaluation Have realistic aims been set for the various stages of the project? X X X Have realistic societal aims been set for the project (i.e. policy change)? X X X Have the main research questions been formulated in a clear and concise manner? X Have the main deliverables been clearly identified (i.e. a research report)? X Have the roles of each partner been clearly defined at each stage of the project? X X X Has the individual responsible for ensuring the research gets completed been identified? X Do any of the individuals involved in this project need training or skill development? X X Have the project methods been clearly defined? X Have the amount of funds needed for this project been determined? X X X Has it been determined who will supply the funds needed for this project? X X X Has a mutually agreeable timeframe been constructed? X Mid-Point Evaluation Has the project been progressing in accordance with the original aims established? X X Have any changes made to the research questions been agreed to by all partners? X X Have any changes made to the methods selected been agreed to by all partners? X Have partners been able to contribute to the approach of the project? X X X X Have partners been able to contribute to the content of the project? X X X X Have research team members been communicating openly with one another? X X Have research team members been communicating through the use of plain language? X X Have partners developed a common understanding of the problem area? X X End-Point Evaluation Did the final deliverable meet partners’ needs? X Was the final deliverable useful to a broader audience than just the project partners? X Was the final deliverable understandable to non-specialist audiences? X Is the final deliverable likely to influence directions of future research? X X Were partners satisfied with the engagement process and the end deliverables? X Did the students involved improve their ability to perform research and present findings? X X X Did the students involved improve their networking skills and professional networks? X Did the students involved receive good supervision and feedback throughout? X

p.8


TABLE 1: SUMMARY OF QUESTIONS ASKED IN EVALUATIONS PARARES CIROP Access Alliance NARCH V Braun Et

Al. (2012) Post-Project Evaluation Has the project increased partners’ knowledge of how research is done? X X X Has the project increased students’ knowledge about the topic in question? X X X Has the project increased students’ interest in societal issues? X X X Has the project helped foster continuing relations between the Science Shop and partner? X X Has the project increased partners’ capacity to get project funding? X X X Has the project led to the development of new research collaborations or opportunities? X X Has publication of the end deliverables raised awareness about the issue(s)? X X Has publication of the end deliverables caused alternative policies to be considered? X X X Has publication of the end deliverables improved existing policies, programs, or services? X X X

TABLE 2: SUMMARY OF INDICATORS USED IN EVALUATIONS PARARES NARCH V Sandoval Et Al. (2012)

King Et Al. (2010)

Khodyakov Et Al. (2012)

Curriet Et Al. (2005)

Start-Point Evaluation Community Capacity X X Historical Context of Collaboration X X Organizational Capacity X X Process Evaluation Involvement In Grant Proposal Writing X X Involvement In Background Research X X Involvement In Choosing Research Methods X X Involvement In Developing Sampling Procedures X X Involvement In Recruiting Study Participants X X Involvement In Designing Interview/Survey Questions X X Involvement In Collecting Primary Data X X Involvement In Analyzing Collected Data X X Involvement In Interpreting Study Findings X X Involvement In Writing Reports/Journal Articles X X Involvement In Delivering Presentations X X

p.9


TABLE 2: SUMMARY OF INDICATORS USED IN EVALUATIONS PARARES NARCH V Sandoval Et Al. (2012)

King Et Al. (2010)

Khodyakov Et Al. (2012)

Curriet Et Al. (2005)

Output/Outcome Evaluation Academic Publications X X X X Non-Academic Publications X X X Conference Presentations X X X Community Presentations X X X Personal Knowledge Development X Personal Research Skill Development X Community & Organizational Development X Information Access & Use X

TABLE 3: SUMMARY OF MEASURES USED IN EVALUATIONS PARARES CIROP Access Alliance NARCH V Khodyakov

Et Al. (2012) Scarinci Et Al. (2009)

Quantitative-Based Dichotomus Questions/Questionnaire X X X Likert-Scale Questions/Questionnaire X X X X X Cost-Benefit Analysis X Logic Model X X Qualitative-Based Open-Ended Questions/Questionnaires X X X X Semi-Structured Interviews X X Semi-Structured Focus Groups X

EVALUATION QUESTIONS WORTH NOTING The Tables above provide a useful visual medium through which to understand which questions, indicators, and measures are referenced most often within the literature. Table 1 in particular offers a valuable list of questions that can be used to structure any community-based participatory research evaluation. The following questions from Table 1 are those deemed of most importance to any evaluation given their repetition within the literature:

Start-Point Evaluation:

• Have realistic aims been set for the various stages of the project?

• Have realistic societal aims been set for the project (i.e. policy change)?

• Have the roles of each partner been clearly defined at each stage of the project?

• Have the amount of funds needed for this project been determined?

• Has it been determined who will supply the funds needed for this project?

Mid-Point Evaluation:

• Have partners been able to contribute to the approach of the project?

• Have partners been able to contribute to the content of the project?

End-Point Evaluation: • Did the students involved improve their ability to perform research and present

findings?

Post-Project Evaluation:

• Has the project increased partners’ knowledge of how research is done?

• Has the project increased students’ knowledge about the topic in question?

• Has the project increased students’ interest in societal issues?

• Has the project increased partners’ capacity to get project funding?

p.11


• Has publication of the end deliverables caused alternative policies to be considered?

• Has publication of the end deliverables improved existing policies, programs, or services?

CONCLUSIONS This brief literature scan has reported the most important and most relevant findings from the literature surrounding the evaluation approaches used to assess community-based participatory research projects. It sought to determine what questions, indicators, and methods were being used to evaluate the effectiveness of community-based participatory research projects for Research/Science Shops, Community Partners, and Student Researchers.

The findings offered here offer some preliminary information about evaluating community-based participatory research. It is important to note that this report constitutes a brief literature scan, and that most attention was directed to those materials deemed of most importance to the project. To build on this preliminary resource, additional research exploring a larger breadth of materials is required. In addition, there seems to be a lack of academic publications on this topic. Correspondence and interviews with other Science Shops about the evaluations they use might help resolve this limitation.

p.12


REFERENCES Access Alliance Multicultural Health and Community Services (2011). Community- Based

Research Toolkit: Resource for Doing Research with Community for Social Change. Toronto: Access Alliance Multicultural Health and Community Services.

Blackstock, K., Kelly, G., & Horsey, B. (2007). Developing and Applying A Framework to Evaluate Participatory Research for Sustainability. Ecological Economics, 60, 726-742.

Boere, E., & Heijman, W. (2011). Discussion Paper: A Cost-Benefit Analysis and Evaluation of Science Shops. Public Engagement with Research and Research Engagement with Society – PERARES. Financed by the European Community's Seventh Framework Programme.

Braun, K., Nguyen, T., Tanjasiri, S., Campbell, J., Heiney, S., Brandt, H., Smith, S., Blumenthal, D., Hargreaves, M., Coe, K., Ma, G., Kenerson, D., Patel, K., Tsark, J., Hébert, J. (2012). Operationalization of Community-Based Participatory Research Principles: Assessment of the National Cancer Institute’s Community Network Programs. American Journal of Public Health, 102, 1195-1203.

CBPR Research for Improved Health Study Team. (July 2011). Focus Group Interview Guide, Research for Improved Health: A National Study of Community-Academic Partnerships. Qualitative Study Instrument, University of New Mexico Center for Participatory Research. From: NARCH V (Indian Health Service/NIGMS/NIH U261HS300293 2009-2013), a partnership between the National Congress of American Indians Policy Research Center (Sarah Hicks, PI); the University of New Mexico Center for Participatory Research (Nina Wallerstein, PI); the University of Washington Indigenous Wellness Research Institute (Bonnie Duran, PI); and CBPR projects nationwide.

CBPR Research for Improved Health Study Team. (July 2011). Partnership Interview Guide, Research for Improved Health: A National Study of Community-Academic Partnerships. Qualitative Study Instrument, University of New Mexico Center for Participatory Research. From: NARCH V (Indian Health Service/NIGMS/NIH U261HS300293 2009-2013), a partnership between the National Congress of American Indians Policy Research Center (Sarah Hicks, PI); the University of New Mexico Center for Participatory Research (Nina Wallerstein, PI); the University of Washington Indigenous Wellness Research Institute (Bonnie Duran, PI); and CBPR projects nationwide.

p.13


Currie, M., King, G., Rosenbaum, P., Law, M., Kertoy, M., & Specht, J. (2005). A Model of Impacts of Research Partnerships in Health and Social Services. Evaluation and Program Planning, 28, 400-412.

Khodyakov, D., Stockdale, S., Jones, A., Mango, J., Jones, F., & Lizaola, E. (2012). On Measuring Community Participation in Research. Health Education & Behavior, 40, 346-354.

King, G., Servais, M., Currie, M., Kertoy, M., Law, M., Rosenbaum, P., Specht, J., Willoughby, T., Forchuk, C., & Chalmers, H. (2003). The Community Impacts of Research Oriented Partnerships (The CIROP Measure). Available From: www.impactmeasure.org.

King, G., Servais, M., Forchuk, C., Chalmers, H., Currie, M., Law, M., Specht, J., Rosenbaum, P., Willoughby, T., & Kertoy, M. (2010). Features and Impacts of Five Multidisciplinary Community-University Research Partnerships. Health and Social Care in the Community, 18, 59-69.

King, G., Servais, M., Kertoy, M., Specht, J., Currie, M., Rosenbaum, P., Law, M., Forchuk, C., Chalmers, H., & Willoughby, T. (2009). A Measure of Community Members’ Perceptions of the Impacts of Research Partnerships in Health and Social Services. Evaluation and Program Planning, 32, 289-299.

Sandoval, J., Lucero, J., Oetzel, J., Avila, M., Belone, L., Mau, M., Pearson, C., Tafoya, G., Duran, B., Rios, L., & Wallerstein, N. (2012). Process and Outcome Constructs for Evaluating Community-Based Participatory Research Projects: A Matrix of Existing Measures. Health Education Research, 27, 680-690.

Scarinci, I., Johnson, R., Hardy, C., Marron, J., & Partridge, E. (2009). Planning and Implementation of a Participatory Evaluation Strategy: A Viable Approach in the Evaluation of Community-Based Participatory Programs Addressing Cancer Disparities. Evaluation and Program Planning, 32, 221-228.

Stedman-Smith, M., McGovern, P., Peden-McAlpine, C., Kingery, L., Draeger, K. (2012). Photovoice in the Red River Basin of the North: A Systematic Evaluation of a Community-Academic Partnership. Health Promotion Practice, 13, 599-607.

Trench, B., Kaiser, D., Vargiu, A., & Windt, H. (2013). Evaluation Guidelines and Instruments. Public Engagement with Research and Research Engagement with Society – PERARES. Financed by the European Community's Seventh Framework Programme.

p.14


Windt, H., Murphy, P., Smith, D., & Vargiu, A. (2014). Final Report on PERARES Work Package 9: Monitoring and Evaluation. Public Engagement with Research and Research Engagement with Society – PERARES. Financed by the European Community's Seventh Framework Programme.

p.15


APPENDIX: SELECTED FRAMEWORKS & LOGIC MODELS

Figure 1. The Evaluation Framework in Blackstock Et Al. (2007)

p.16


Figure 2. The Logic Model in Scarinci Et Al. (2009)

evaluating community-based participatory research

Documents