a valuable stakeholder engagement and evaluation design tool...a valuable stakeholder engagement and...
TRANSCRIPT
Focusing Sessions
A valuable stakeholder engagement and evaluation design toolVeronica S. SmithNational Science Foundation Research Traineeship Annual Meeting September 2019
1. Purpose
Donaldson’s theory-driven evaluation science
• Contributes to team learning and program improvement• Increases stakeholder engagement• Reduces evaluation anxiety• Hypothesis driven
Donaldson, Stewart, (2017) Program Theory-Driven Evaluation Science:Strategies and Applications
Focusing session goals• Create common understanding • Make program theory explicit and testable• Frame evaluation questions and identify key measures• Provide framework for data collection and analysis
2. Are you ready?
3. Be gameful!
Theory of Action
4. Focus on learning
together
Theory of Change
Community of Learning
Where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole (reality) together.
-Peter Senge
What are the 3 attributes you would like to contribute to our learning community?
Veronica Smith1. Diversity, equity,
inclusion best practices2. Facilitation skills3. Data science
How do we want to conduct the evaluation as co-creators of knowledge?
What kind of learning community do we want to create?
Norms• Experience Discomfort• Take Risk• Stay Engaged• Listen for Understanding• Speak Your Truth• Expect/Accept Non-Closure• No Fixing
17
5. Focus on change
together
Program Logic
Input Activity Output Short-termOutcome
IntermediateOutcome
Long-termOutcome
Process evaluation
A C T I O N
Impact evaluation
C H A N G E
Focus on C H A N G E
Increase Decrease DeepenBroaden Mediate ModerateDiversify Compress TemperAccelerate Decelerate
6. Map together
Need impact map to move beyond waving of hands
éSTEM Diversity,
Equity & Inclusion
YOUR NRT PROGRAM
Anchor Changes
5+ years
Timeline
é Trainee Knowledge & Skill
éInterdisciplinarity
& Convergence
Timeline2-4 years 5+ years1-2 months
Desired Change C
Desired Change E
leads to
Impact Map
leads to
Desired Change B
Desired Change A
Desired Change D
Anchor change
24
Insert the DIRECT maps
25
7. Focus on
knowledge needs
together
What if you could get a sound answer to a question about your program that would help improve outcomes?
What are the measures you need to track progress against goals?
Example questions• How confident are trainees about their career prospects and
competitiveness in the clean energy industry?• How effective are the NRT courses at preparing students for
the capstone project?• How capable are students at applying and communicating
about data science across domains?
31
“Concrete development of questions and group discussion.”
“It was great to learn the goals and general direction of the organization.”
“Healthy discussion and reframing of comments.”
“Enjoyed being able to share as a stakeholder.”
“The collaborative process allowed everyone to participate.”
“I felt like my opinion as a student and my contributions were really valued.”
“Having input into the program’s focus.”
32
Measurement Invariance Across Time
Cheryl Schwab PhDInternal Evaluator
Overview of the DS421 Program
Purpose
Test the assumption of measurement invariance when assessing change in trainee ability (latent variable) using the same instrument at several time points.
AssumptionResponses across time points for the same person are conditionally independent given the latent variable (i.e., the correlations between the responses at each time point are smaller than can be attributed to trainee ability).
Item Response TheoryUnderlying function called an item characteristic curve which specifies that as the level of a trait increases the probability of a correct response to an item increases.
(see Hambleton, Swaminathan & Rogers, 1991)
Outcome SurveyRate your ability to do the following in a collaboration (i.e., project, research, work).I am a(n)…novice, advanced beginner, proficient,
expert. ---contribute environmental science disciplinary knowledge.---conduct collaborative research.---write for audiences inside primary discipline.---present ideas to audiences outside primary discipline.
ICC
Differential Item Functioning (DIF)
Test if the difficulty of an item is different across groups or across time.
DataCollected between November 2015 and May 2018
Administrations 4 times across two year programCohorts = 3Departments = 11Cases n= 78
Modeling Items
MODEL DEVIANCE #PARAMETERSRating Scale Model
item + step 4314 33Partial Credit Model
item + item*step 4230 90
Map of Latent Distribution and Thresholds
Latent RegressionREGRESSION COEFFICIENTS
CONSTANT -0.226 ( 0.236) c2 0.007 ( 0.383) c3 0.414 ( 0.395)
Variance 2.002 Unconditional Variance 2.123 ~ 3%
Time
MODEL DEVIANCE #PARAMETERS
item + step 4319 35
item + step + time 4310 38*
* Significant change in deviance 0.05
Main Effect Time
TIME ESTIMATE ERROR 1 0.523 0.108 2 0.225 0.1653 -0.483 0.1754 -0.265*
*estimate constraintChi-square test of parameter equality = 32.77, df = 3, Sig Level = 0.000
Map of Latent Distribution and Parameters
Results• Distribution of item difficulties shifts as the students
move through the program.
Next Steps• Modeling differences between cases• Case dependence• Including more data per case
Faculty and Staff
David Ackerly (Co-PI), Integrative Biology (L&S) Max Auffhammer (PI and Director), Agricultural and Resource Economics (CNR) Maggi Kelly (Co-PI), Environmental Science, Policy and Management (CNR) Philip Stark (Co-PI), Statistics (L&S) David Anthoff, Energy and Resources Group (ERG) David Culler, Electrical Engineering and Computer Sciences (CE) Kristina Hill, Landscape Architecture and Environmental Planning (CED) Solomon Hsiang, Goldman School of Public Policy (GSPP)Laurel Larsen, Geography (L&S) Heather Constable (BiGCB Berkeley Initiative on Global Change Biology & DS421 NSF Research Traineeship: Data Science for the 21st Century & UC Berkeley Field Stations Campus Coordinator)
A Collaborative Approach to Program Evaluation for an NRT
Valerie DeckerAssistant Director, Center for Evaluation and Assessment
University of Iowa
Sustainable Water Development Program
Poster Session 2Number 211
waterhawks.uiowa.edu
Program Evaluation Overview
• Systematic data collection• Informing program improvements• Determining the success of a program• Driven by the information needs of stakeholders
The Program Evaluation Standards
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.
Dimensions of quality:• Utility• Feasibility• Propriety• Accuracy• Evaluation Accountability
www.jcsee.org
Outcome of a Collaborative Approach
Strong foundational plan with opportunities to adapt built in
Necessary Components
Evaluators as collaboratorsBudget for evaluation
Questions
Assessing Interdisciplinary Competency in the Disaster Resilience and RiskManagement (DRRM) Graduate Program using Concept MapsMarie C. Paretti, PhDEducation Director, DRRMVT
Professor, Engineering [email protected]
Image courtesy of NASA Goddard Space Flight Center
This material is based upon work supported by the National Science Foundation under Grant Number 1735139. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
DRRM Program: Purpose
▪ Promote collaboration through a transdisciplinary educational experience
▪ Train business analysts, educators, engineers, scientists, and urban planners to better prepare for, respond to, and recover from natural disasters
Concept Maps
● Measures domain understanding
● Captures depth (levels of hierarcy) & interconnections (number of links)
● Supports multiple scoring methods
○ Quantitative○ Holistic
Concept Map Scoring: Holistic Criteria
Complexity Density Organizational Strategy
Construct Depth of demonstrated knowledge
Breadth of demonstrated knowledge
Sophistication of organization
Measurement Number of hierarchies included in map
Number of concepts included in map
Organizational strategy
[1] M. Besterfield-Sacre, et. al., "Scoring Concept Maps: An Integrated Rubric for Assessing Engineering Education," J. of Eng. Ed., vol. 93, no. 2, pp. 105-115, April 2004, Art.[2] J. E. Sims-Knight et al., "Using concept maps to assess design process knowledge," in FIE 2004. 34th Annual, 2004.
Holistic Scoring: DRRM Scholars* Only
Topical Structured Topical Functional Topical Functional
Complexity
Dens
ity
A
ALow
Med
ium
High
Low Medium High
B
B
C
C
D
D
EE
F
F
G
GHH
Aug 2018
Dec 2018
J
KEY:
J
Density: Number and range of topics and subtopicsComplexity: Connections across topicsOrganizational Strategy: Topical, Structured Topical, Functional Topical, Functional
Organizational Strategy
*Scholars are those students participating in the full doctoral program
Strengths of Concept Maps as Assessment Tool in this Context
1. Provides a representation of student learning2. Supports analysis of depth of thinking and interactions among concepts
Student G
Pre-Map
Post-Map
DRRM concepts span, bridge, and intersect multiple disciplinary boundaries.
1. Interdisciplinarity creates a high level of complexity.
2. 2D Concept map may be insufficient to map knowledge.
3. Quantitative scoring complicated by dimensional limits, scope, and “between” spaces.
Challenges of Concept Maps as Assessment Tool in this Context
Suggestions for using Concept Maps as Assessment Tool in Highly Interdisciplinary Contexts
1. Written explanations can complement visual representations and expand our ability to assess learning.
Suggestions for using Concept Maps as Assessment Tool in Highly Interdisciplinary Contexts
1. Written explanations can complement visual representations and expand our ability to assess learning.
2. Global (holistic) scoring approaches may more readily capture nuances in the concept maps than quantitative (# of nodes, # of connections, # of levels) methods.
Questions?
11Icon made by Roundicons and freepik from www.flaticon.com
@DRRM_VTJessica [email protected]
@jess_deets
Dr. Marie C. [email protected]
@mparetti2
DRRMhttps://www.drrm.fralin.vt.edu
VT
Gili Marbach-AdCollege of Computer, Mathematical, and Natural
Sciences, University of Maryland
Using Formative Evaluation Practices to Promote Continuous Improvement in two NRT Projects
Mission:
To enhance graduate students’ following skills:
• Interdisciplinary Research
• Communication
• Teamwork/Leadership
• Mentorship
• Cohort 1: Spr2017 - Spr2019 (N=11)
• Cohort 2: Fall2017 - Fall2019 (N=12)
• Cohort 3: Fall2018 - Fall2020 (N=15)
• Cohort 4: Fall2019 - Fall2021 (N=9)
• Cohort 5: Fall2020 - Fall2022
Poster #212 Session 2
Mission:program that trains students to effectively work and communicate across disciplines to address food/energy/water nexus issues
Cohort 1: Spring 2019Poster #110 Session 1
STEM Training at the Nexus of Energy, WAter Reuse and FooD Systems
COMBINE Change Model
�� �����# "�����#! �! ��#������# � :*/$@*2@9 5(%9%$@%$:# 9*32@ 2$@95 +2*2(@�@%2) 2#%@(5 $: 9%@79:$%297@*2@ *293@95 $*9*32 /@�)���@453(5 17�@ +2#/:$*2(�@ � �@�#)3/ 57@<*9)@79+4%2$@�135%@<*9)3:99)%@&3/03<*2(@7-*//7�@ � ��0��'��#���*���#�=�%6(0�:%, 79*4%2$�
���8#���=�#1�(��0��'�#�(< � �#�(��0��=5%436%$@#32&*$%2#%@*2@9 5(%9%$� �29%5$*7#*4/+2 5>@�%7% 5#) �%6/�:&,=06��=!8@�=1��"���0�� 7-*//7� �311:2*# 9*32@ �� � �#�(��0��=2:1"%5@3&@4:"/*# 9*327@ 2$"%�6��=�%6/�=�#�=�#=�#1�#0�7�=��1��� 1<35.�% $%57)*4@
'(%���1=�%6(0� 45%7%29 9*327@3&@*29%5$*7#*4/*2 5>@5%7% 5#)� �%29357)+4@ � �#�(��0��= $;*735@ < 5%2%77@ 2$� ���=��=0�"�#�/ 445%#+ 9*32@&35@*29%5$*7#*4/+2 5>@5%7% 5#)� ��(=1%=��(=161%*��=:���� � �#2�.0��'0=;4=��='�(3$�/ ��
� �#�(��0��=2:1"%5@3&@*29%527)+47� ��#1%+� � ��7��%'= @13$%/@&35@39)%5@&:9:5%��##6��=��(��(=��7��%'"�#1 *29%5$+7#*4/*2 5>@453(5 17�%-�%'��##6��=��=�<"'%0�6" ���� #����#! ������#� �65����=9= ��=���##�!= �� ���7��%'=453(5 1@#31432%297@�%"'���=�%##��1=�#�=���� �� ��� #��(�(��6�1�=(�0��)�="�#2%+�
� �35-@<*9)@ $;*7357@ 2$@�,7@ ���� #����#������ ����#� �2#0:$%@��� ���@& #:/9>@1%1"%5@&5313:97*$%@3&@#35%@$+7#*4/*2%@32@9)%7*7@#311*99%%
������� #���!��# �
� �)���@79:$%297@*2@� 9)%1 9+# /�)>7*# 0�@�314:9%5@�2(*2%%5+2(�314:9 9*32 /�@ 2$@
�� �������#�5 $*9*32 /@�)���@453(5 17@& */@93@$%;%/34@+29%5$*7#*40*2 5>@-23</%$(%�@7-*/07�@ 2$@5%7% 5#)@ 4453 #)%7@9) 9@ 5%@2%#%77 5>@93@#:55%29@<35-40 #%7�@��������#�53(5 17@3&9%2@*2#/:$%@*27:&'*#*%29@1%2935*2(�@ 933@1:#)@%14) 7*7@35@7*/3%$@5%7% 5#)�@ & */:5%@93@ #-23</%$(%@:7%&:/@1%9)3$3/3(*%7@&531@3:97+$%@ # $%1+#@&*%0$7�@���������� �����#�%;%/34@ @13$%/@&35@*29%5$*7#*4/+2 5>@(5 $: 9%@5%7% 5#)@453(5 17@9) 9@%14) 7*?%@9)%@ "3;%@$%7*5%$@7-*//7�@ �*&%@�#*%2#%7@ 9@9)%@�2+;%57*9>@3&@� 5>/ 2$�@
� !77%77*2(@453(5%77@3&@#3:57%@#31432%297@ 2$@9)%+5@439%29* /@795%2(9)7@ 2$@<% -2%77%7@��%%$" #-@&531@79:$%297@ 2$@*2795:#9357�� �5( 2*?*2(@&:9:5%@7#)%$:/%@3&@��� ���@453(5 1@ 2$@*$%29*&>@#) 2#%7@&35@&:59)%5@*2;%79*( 9*32� !$ 49*2(@933/7@93@"%99%5@5%&/%#9@%=4%5*%2#%@*2@��� ���@453(5 1@@
FrameworkforAssessment:MeasuringFiveLevelsofProgramEvaluation
ØParticipationØSatisfactionØLearningØApplicationØImpact
Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler.Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin.Colbeck, C. L. (2003). Measures of success: An evaluator’s perspective. Paper presented at the CIRTL Forum, Madison, WI.Connolly, M. R., and Millar, S. B. (2006). Using workshops to improve instruction in STEM courses. Metropolitan Universities 17.4 : 53-65.Marbach-Ad, G., Egan, L. and Thompson, K. (2015). A discipline-based teaching and Learning Center: A model for professional development. Springer
Jan 2017
May/Aug 2017
Dec/Jan 2018
May/Aug 2018
Dec/Jan 2019
Cohort I (N=13-1=12)
Data Practicum(n=13)
Research & Progress Seminar (n=12), Network Analysis (n=21)
Inte
rvie
ws
Surv
eys
Pre-
Surv
ey
Post
-Sur
vey
Post
-Sur
vey
Peer to Peer (n=5)
Post
-Pro
gram
Sur
vey
Int. with instructors
(n=2) Int. with students
(n=4)
Peer to Peer
Int. with instructor
Focus group with
students (~15)
Follow-up
interview with
students (n=3)
Mixed Model Research DesignCOMBINE: Cohort 1 (Jan 2017- Jan 2019)
Research & Progress Seminar
Lit. Survey Seminar
Interview with
Advisors (n=3)
Data Collection:• Observations• Artifacts• Surveys• interviews
Jan
2017
May/Aug
2017
Dec/Jan
2018
May/Aug
2018
Dec/Jan
2019
May/Aug
2019
Dec/Jan
2020
May
2020
Cohort I
Cohort II
Data Practicum
Lit. Survey
Seminar & Data
Practicum
Network Analysis
& Research &
Progress Seminar
Inte
rvie
ws
Su
rve
ys
Pre
-Surv
ey
Post
-Surv
ey
Pre
-Surv
ey
Post
-Surv
ey
Peer to Peer
Post
-Pro
gra
m S
urv
ey
Post
-Pro
gra
m S
urv
ey
Peer to Peer
Post
-Surv
ey
Int. with
students
(n=4) Follow-
up
interview
with
students
(n=4)
Pre
-Surv
ey
Post
-Surv
ey
Int. with
instructors
(n=2) Int. with
students
(n=4)
Int. with
instructor
Focus
group
with
students
(~15)
Follow-
up
interview
with
students
(n=3)
Mixed Model Research DesignCOMBINE: Cohort 2 (May 2017- May 2019)
Research &
Progress Seminar
(n=12), Network
Analysis (n=21)
Lit. Survey
Seminar
Interview
with
Advisors
(n=3)
Jan
2017
May/Aug
2017
Dec/Jan
2018May/Aug
2018Dec/Jan
2019
May/Aug
2019
Dec/Jan
2020
May
2020
Cohort I
Cohort II
Cohort III
Data Practicum
Lit. Survey
Seminar & Data
Practicum
Lit. Survey
Seminar
& Data
Practicum
Network Analysis
& Research &
Progress Seminar
Inte
rvie
ws
Su
rve
ysPr
e-Su
rvey
Post
-Sur
vey
Pre-
Surv
ey
Post
-Sur
vey
Symposium Peer to Peer
Prog
ram
Sur
vey
Prog
ram
Sur
vey
Po
st-P
rog
ram
Su
rve
y
(+co
ntr
ol g
rou
p)
Peer to PeerPo
st-S
urve
y
Interview
with
students
(n=4)
Pre-
Surv
ey
(+co
ntro
l gro
up)
Post
-Sur
vey
Post
-Sur
vey
Follow-up
interview with
students
(n=4)
Int. with
students (n=4)
Follow-
up interview
with
students (n=4)
Int. with
instructors
(n=2) Int. with
students (n=4)
Int. with
instructorFocus
group with
students
(~15)
Follow-
up interview
with
students (n=3)
Mixed Model Research DesignCOMBINE: Cohort 3 (May 2018- May 2020)
Network Analysis
& Research &
Progress Seminar
Interview with
Advisors
(n=3)
Research &
Progress Seminar
Mixed Model Research DesignUMD Global STEWARDS
Interdisiplinary Research Communication Career Preparation
COMBINE Components
Lear
n ab
out i
mpo
rtan
t res
earc
h re
sults
in n
etw
ork
scie
nce
Lear
n ab
out t
he th
eory
and
met
hods
of
net
wor
k an
alys
is
Deve
lop
an a
ppre
ciatio
n an
d un
ders
tand
ing
of th
e qu
estio
ns a
nd
met
hods
of o
ther
fiel
ds
Appl
y m
etho
ds o
f net
wor
k an
alys
is to
bi
olog
ical d
ata
Deve
lop
skill
s for
ora
l pre
sent
atio
ns
Lear
n to
bui
ld e
ffect
ive
visu
aliza
tions
Deve
lop
writ
ten
com
mun
icatio
n sk
ills
Com
mun
icate
to a
div
erse
aud
ienc
e
Build
team
wor
k sk
ills
Gain
exp
erie
nce
men
torin
g ot
hers
Deve
lop
lead
ersh
ip sk
ills
Gain
exp
osur
e to
div
erse
care
er
optio
ns
COMBINE Courses Data Practicum with Communication Training (PHYS798N)
6 2 10 5 10
9 8 10 7 2 3 3
Computational and Mathematical Analysis of Networks (CMSC8280)
5 11 5 10 5 6 3 6 9 2 4 1
Seminar Courses Network Science Literature Survey Seminar
8 4 5 1 8 0 2 5 5 1 2 3
Network Biology Research-in-Progress Seminar
6 4 9 2 9 3 2 7 2 0 0 5
Elective Courses Discipline-bridging 3-4-credit elective course (chosen from list of options)
5 4 6 3 2 5 5 3 3 0 1 3
Discipline-bridging elective seminar (chosen from list of options)
3 2 4 2 1 1 2 1 2 0 0 3
Other Activities Peer-to-peer tutorial 2 1 3 2 7 1 2 6 5 5 2 2COMBINE Annual Symposium 7 4 6 1 6 3 2 4 1 0 1 7Career Development Workshop 2 1 3 0 3 0 0 1 1 0 0 4COMBINE Outreach 1 0 3 2 3 1 0 5 4 2 3 0Outside research project/Internship 0 0 1 0 1 0 0 0 0 0 0 0Other COMBINE-related element 1 0 1 1 2 0 0 2 1 2 2 1
Feedback from First CohortFor the COMBINE components that you participated in, check those that helped you with each of the following objectives (check all that apply)
Feedback from First Cohort• Not enough opportunities for fellows to connect
with each other outside of their scheduled courses or the required COMBINE activities. üLeadership team began hosting a series of “coffee talks”
wherein fellows were invited to socialize and collaborate with each other in a more social, non-academic setting.
COMBINE faculty (~25)
Weekly seminars- COMBINE faculty research talks- Literature presentations/discussion- COMBINE trainee research talks
CMCS828O Network Analysis tools
Advanced coursework
Discipline-brindging coursework
Minimum 2 courses (4 credits) in complementary areas
ACTIVITIES
PHYS798N - Intensive data project and communication
Peer-to-peer tutorials
InternshipsSmithsonian, NIH, NIST, UMD School of Medicine, Genentech, NVIDIA, Epic, Dexcom...
Outreach/Mentoring
- K-12 student education- Public engagement- Undergrad mentoring- Peer mentoring
Annual Career Workshop
Annual COMBINE Symposium
Trainee oral and poster presentations
PARTICIPANTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES
PI
Research educatorProgram manager
? ability to communicate research and findings to diverse audiences (including researchers outside their immediate field and the general public)
PhD graduates with
? knowledge of opportunities and effective approaches for outreach and mentoring
? knowledge-base and skills to develop, propose, and execute research at the intersection of P/M, CS, and BIO*
? knowledge of career options in academia, industry, and national labs
? professional networks in academia, industry, and national labs
Increased number of interdisciplinary researchers prepared to address problems at the intersection of P/M, CS, and BIO* (in academia, industry, and national labs)
Student committee or Executive committee
? teamwork skills for interdisciplinary research
*The three COMBINE disciplinesP/M: Physical/Mathematical SciencesCS: Computational SciencesBIO: Biological Sciences
Evaluation categoriesP: ParticipationS: SatisfactionL: LearningA: ApplicationI: Impact
co-PIs
COMBINE research faculty (mentors/advisers)
COMBINE administration
COMBINE evaluation
L
A
L
L
L
A
I
Internal evaluatorExternal evaluatorAdvisory board
Regular participant
COMBINE trainees (~50)
x5
Evaluation methods done Survey Student Interviews Instructor Interviews Focus Group Class artifacts
designates COMBINE components that are in progress
PROPOSED EVALUATION
Surveying of studentsTo investigate:- baseline characteristics-educational history-strength and confidence in skills incorporated in COMBINE program-evaluation of program components
Student InterviewsTo investigate:- more in-depth classroom experience and individual growth-effects of COMBINE not evidenced in classroom like conference attandance or publications
Instructor InterviewsTo investigate:- effectiveness of course components- potential changes to the course or program
Focus Groups To investigate:-- effectiveness of course components- potential changes to the course or program
Class Artifacts To investigate:- frequency and characteristics of course components- student participation and response to course
P, S, L
S, L, I, A
P, L, I
P, S, L
P, L
Expansion of interdiciplinary graduate programs based on advocacy for COMBINE's effectiveness
Create a repository for evaluated and established courses for interdisciplinary research and communication
I
I
COMBINE: Logic Model
Acknowledgements
• COMBINE and UMD Global STEWARDS Fellows, their advisors, and the leadership team
• The Evaluation Teams:• Jack Marr• Frances Turner• Megan Gerdes• Hannoori Jeong
Questions
• COMBINE: Poster #212 Session 2• UMD Global STEWARDS: Poster #110 Session 1
Evaluation of graduate program courses: going beyond the typical end of course survey
2019 NRT Annual Meeting
9.26.2019 • Andrew Grillo-Hill, Josh Valcarcel
Evaluators for RIPTIDES program at San Francisco State University
Poster # 124 (session 1)
2
Context:
3
Challenge
• Y1 evaluation focused on newly developed courses
• Useful & actionable feedback for the program
4
Challenge
• Y1 evaluation focused on newly developed courses
• Useful & actionable feedback for the program
• Some issues related to the existing courses
• Not enough information • Disagreed with university course
evaluation
5
Challenge
• Y1 evaluation focused on newly developed courses
• Useful & actionable feedback for the program
• Some issues related to the existing courses
• Not enough information • Disagreed with university course
evaluation• For Y2 look at ALL the core courses
• Student focus groups• Student surveys• Limited observations• Internal memo of findings• Faculty interviews
6
Methodology:
7
Key Findings:
• Improve interdisciplinarity• Expectations didn’t always
match• More coordination across
the courses• More statistics training• Less than ideal meeting
day and time
8
Program Action
• Setting clearer expectations • Writing course now specific for
program students• One course changed day & time• One course redesigned to be a
statistics course• Faculty planning meeting for
team taught course
