project evaluation and assessment - nsf · project evaluation and assessment nsf epscor pd/pa/eod...
TRANSCRIPT
![Page 1: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/1.jpg)
Project Evaluation and Assessment
NSF EPSCoR PD/PA/EOD Meeting January 23, 2015
![Page 2: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/2.jpg)
Our Presenters
• Jim Gershey – Louisiana • Pips Veazey – Alaska • Michelle Gregoire – New Hampshire • Amy Slocum, Deleware
![Page 3: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/3.jpg)
Introduction
![Page 4: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/4.jpg)
Objectives of Evaluation
• Assessment of progress toward goals listed in proposal and strategic plan
• Evidence of broader impacts • Effectiveness of diversity, workforce,
outreach efforts • Determine if changes/adjustments are
required
![Page 5: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/5.jpg)
Components of Evaluation Plan
• Strategic Plan • External Evaluation Team • Internal Evaluators • External Review Board (ERB) • Project Execution Team (PET) • Reverse Site Visit • EPSCoR Committee
![Page 6: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/6.jpg)
External Evaluation Team
• Developed an easy-to-use web-based data collection and survey tool for formative and summative evaluation
• Data is collected, collated and reported back to project management team
• Evaluators attend annual meetings
• Some templates are automatically completed
![Page 7: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/7.jpg)
Lessons Learned
• Helpful to have External Evaluation Team involved in the development of the Strategic Plan
• Ease of use of the data reporting system is key
• Plan way ahead for visits by External Review Board
• Have external evaluators attend state meeting
![Page 8: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/8.jpg)
Alaska EPSCoR Evaluation
![Page 9: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/9.jpg)
Objectives of Evaluation
Overall productivity – goals listed in strategic plan
Research capacity development
Production of scientific knowledge and research capacity
Institutionalization of research program across the state
Communication and learning
Impacts on students
Integration and impacts of stakeholders
![Page 10: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/10.jpg)
Components of Evaluation Plan
University of Alaska Program Advisory Council (university)
External Experts Advisory Committee (national)
External Evaluators (national)
State Committee on Research (statewide)
Reverse Site Visit (NSF)
Strategic Plan
![Page 11: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/11.jpg)
External Evaluation Team
Faculty surveys and interviews (annual meeting/teleconferences)
Students surveys and focus groups
Stakeholder interviews (internal and external)
Bibliometrics
Online metrics
Social network analysis (interdisciplinarity)
![Page 12: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/12.jpg)
Lessons Learned
Engage evaluators in significant developments
Plan ahead – communicate timing of major events
Identify people at each partner institution to serve in advisory capacity
Significant investment for every state – how can we as a community maximize the benefit of our evaluations?
![Page 13: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/13.jpg)
nhepscor.org
Objectives of Evaluation
• Assess research capacity development • Accurately assess project strengths and
weaknesses • Evaluate effectiveness of research
collaborations, educational programs • Recommendations for course correction
![Page 14: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/14.jpg)
nhepscor.org
Components of Evaluation Plan
• Strategic plan • External evaluation team • AAAS scientific advisory board • Reverse Site Visit
![Page 15: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/15.jpg)
nhepscor.org
External Evaluation Team
• Integration: Network analysis • Productivity: Bibliometric analysis • Outreach: Coordination, impact, diversity • Student impacts: Engagement in research
activities
![Page 16: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/16.jpg)
nhepscor.org
Lessons Learned • Evaluators: Measure impact • Advisory board: Guidance, national perspective • Both should contribute to strategic plan; annual
review of evaluation plan with management team • Attend team meetings: Presentations, observations,
interviews, feedback to management team • Reverse Site Visits: Can be useful but limited
perspective
![Page 17: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/17.jpg)
Objectives of Evaluation
• To insure that the needs of critical stakeholders (e.g., Delaware EPSCoR leadership and NSF) are met through the evaluation process and that results are useful and timely - Patton’s utilization-focused approach (Patton, 2008)
• Use information provided by the internal evaluation team and program leadership, along with stakeholder interviews during a site visit, to provide guidance on achieving benchmarks
• Assess progress as it relates to the relevant program elements and goals
• Offer perspectives on the local, state, and national impacts of programmatic elements.
![Page 18: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/18.jpg)
Components of the Evaluation Plan
Evaluation Foundations
• Utilization-Focused Evaluation (Patton 2008)
• Guiding Principles for Evaluators (AEA, 2004)
• Internal and External
• Summative and Formative
• Quantitative and Qualitative
Methods
•Faculty surveys and interviews
•Key informant interviews
•Pre/post student and teacher surveys
•National Student Clearinghouse – “on track”
•Comparison Groups
•Bibliometric/online metric analyses
•Social networking
•Technology usage data
![Page 19: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/19.jpg)
External/Internal Evaluation Team
External Evaluation Personnel
• State Steering Committee • External Advisory Board • AAAS Panel • NSF RSV Panel
Internal Evaluation Personnel • Internal Evaluator • Institutional Evaluation Support
![Page 20: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/20.jpg)
Lessons Learned
• Focus evaluation on specific components of the program for each visit
• Ensure we are using evaluation reports to help guide programmatic decisions
• Determine what is doable and within a realistic scope of time and available funding
• Internal evaluator who is a key member of the management team as they provide a critical eye and perspectives beneficial to keeping program leaders focused on goals, objectives and metrics
• Identify individuals at each partner institution to serve as evaluation contact to call that ensures ownership, consistent reporting and institutionalization
![Page 21: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/21.jpg)
Questions?
![Page 22: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/22.jpg)
Round Table Discussion
“Brain Forest”
![Page 23: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/23.jpg)
Discussion Format
• 2:00-2:10 – first round table discussion Q (10 min) • 2:10-2:15 – transition • 2:15-2:25 – second round of discussion Q (10 min) • 2:25-2:30 – transition • 2:30-2:40 – third round of discussion Q (10 min) • 2:40-2:50 – Return to seats. Note takers prepare report
out. Complete meeting evaluation form. • 2:50-end – Report out key findings
![Page 24: Project Evaluation and Assessment - NSF · Project Evaluation and Assessment NSF EPSCoR PD/PA/EOD Meeting January 23, 2015 . Our Presenters • Jim Gershey – Louisiana • Pips](https://reader034.vdocuments.net/reader034/viewer/2022042219/5ec4d5c82289635eef56016c/html5/thumbnails/24.jpg)
Discussion Questions