api.ning.comapi.ning.com/.../copyofserppostassessmentreport_o… · web viewthe data from these...

SERP 2014 – Survey Results Report – last updated October 26, 2014

SERP Post-Institute Report

The purpose of the post-institute data analysis is to determine how the Institute impacted team members and the teams overall to embark on a plan of action and implement their plan successfully.

The items used for this report were:1) Pre-Institute STEM Department Evaluation survey2) Post-Institute Surveys: Part 1 & 23) Team Action Plans4) Follow-up Interviews

The data from these items were aggregated and, in some cases, summarized to determine how the Institute impacted individuals and teams, specifically through the use of the STEM Department Rubric, the workshops, and other sessions. These results were then reviewed to identify trends and compare pre-post to discover what changes resulted from their participation in the Institute.

None of the data were analyzed for statistical significance.

There is a signed IRB form on record for each participant in the 2014 SERP Institute.

1


Pre-Institute Surveys

As part of the application process for the teams, each team leader had to complete a team application and each team member had to complete an individual survey. The individual survey asked potential participants what kind of things they wanted to learn at the Institute.

See bullet list in Appendix A – should I pull out common keywords and then compare with post institute? There is a lot of data.

1) Pre-Institute: Rubric

A few weeks prior the Institute, selected teams were asked to review the STEM Department Rubric on their own (i.e., without consulting with their team members) and identify where they thought their department currently stands, from Baseline to Exemplary. Seventy-one participants responded to this question.

Their responses illustrated that the majority of individuals saw their departments as Developing for most of the Rubric factors.

Only one factor was rated as Beginning by just over 50%: “Faculty Development.”

Three factors that received highest rating as Accomplished were: “Mentored Research by Students” (32%), “Integration of Department with GenEd in Institutional Mission and Goals” (32%), and “Assessment” (23%).

Authenti

c Rese

arch i

n Clas

ses/L

abs

Student

Cognit

ive Skill

s Dev

elopmen

t

Studen

t Meta

cogn

itive Skill

s Dev

elop...

Student

Core Com

petenc

ies

Indepe

ndent/

Mentored R

esearc

h by S

t...

Use of

Best Peda

gogie

s

Facu

lty Deve

lopmen

t

Assessm

ent

Dispositi

ons o

f Fac

ulty a

nd Adm

inistra..

.

Integrat

ion of D

epart

ment Prog

ram with Gen

Ed Go...

-10%0%

10%20%30%40%50%60%70%

Where each participants saw his/her institution's life sciences department on the scale from Baseline to Exemplary before the Institute

BaselineBeginningDevelopingAccomplishedExemplary

2


2) Pre-Institute: Rubric factors of highest priority for improvement

Individual team members were also asked to identify the top 3 Rubric factors that they felt were the highest priority to improve. The question specifically asked:

We recognize that the Rubric's factors are not mutually exclusive, but as of now, if you were to give highest priority for improvement to up to three factors on the rubric, what would they be? Please select no more than three!

The data in the following chart illustrates the actual number (not a percentage) of individuals who picked those rubric factors. These data show that “Use of best pedagogies” (49% of 71 individual team members) and “Authentic research in classes/labs” (45%) were given the highest priority by individual participants. Three additional factors each were given high priority rankings by 35% of the 71 respondents: “Student cognitive skills development,” “Student metacognitive skills development,” and “Student core competencies.”

Authentic

research in

classe

s/labs

Student co

gnitive

skills d

evelopment

Student m

etacogni

tive sk

ills dev

elopment

Student core

compete

ncies

Independen

t/mentored re

search by

students

Use of best

pedagogie

s

Faculty deve

lopment

Assessm

ent

Disposi

tions o

f faculty a

nd adminis

tration

Integra

tion of de

partment pr

ogram with Gen E

d goals a

nd Institu

tional missi

on

0%20%40%60%80%

100%

45%35% 35% 35%

24%

49%30% 27%

15%4%

Individual team members' highest priority for improvement of up to three factors on the rubric (n=71)

3


3) Pre-Institute: Rubric factors requiring least effort

Individual team members were also asked to identify the top 3 Rubric factors that they felt require the least effort to improve, specifically the question was stated as follows:

Regardless of the priority rating you selected above, please select up to three factors that you think would require the LEAST effort from the department to improve. Please select no more than three!

Of the 71 respondents, 51% selected the “Integration of department program with Gen Ed goals and Institutional Mission” factor as the one that they felt requires the least effort from the department to improve. Other factors that were selected most commonly by respondents were “Independent/mentored research by students” (37%) and “Faculty development” (30%).

Authentic

research

in cla

sses/la

bs

Student c

ognitive

skills

devel

opment

Student

metacog

nitive

skills

develo

pment

Student

core c

ompet

encies

Indepe

ndent/m

entore

d resea

rch by

studen

ts

Use of b

est peda

gogies

Faculty

devel

opment

Assessm

ent

Disposi

tions o

f facul

ty and

adminis

tration

Integra

tion of de

partment p

rogram

with Gen

Ed goal

s and

Institu

tional m

ission

0%20%40%60%80%

100%

24% 21% 17% 24%37%

17%30% 28% 25%

51%

Individual team members' choices of their top three Rubric factors requiring the least effort from the department to improve (n=71)

4


4) Pre-Institute: Rubric factors requiring the most effort/most difficult

Individual team members were asked to select the three Rubric factors that they felt would require the most effort or were the most difficult for their department to improve. Specifically, they were asked:

In contrast, please select up to three factors that you think would require the MOST effort or be the MOST DIFFICULT for the department to improve. Please select no more than three!

The was no one Rubric factor that stood out in the responses. Instead, there were a number of factors that were selected by over 30% of the 71 respondents: “Use of best pedagogies” (38%), “Authentic research in classes/labs” (37%), “Student cognitive skills development” (35%), “Faculty development” (34%), and “Dispositions of faculty and administration” (32%). The factor that was selected least frequently was “Integration of department program with GenEd goals and Institutional Mission” (11%).

Authentic re

search in

classe

s/labs

Student co

gnitive sk

ills devel

opment

Student m

etacogniti

ve ski

lls development

Student c

ore com

petencie

s

Independen

t/mento

red researc

h by s

tudents

Use of best p

edagogies

Faculty developm

ent

Assessm

ent

Dispositi

ons o

f facul

ty and a

dminis

tration

Integration of de

partment p

rogram

with Gen E

d goals

and Institu

tional m

ission

0%20%40%60%80%

100%

37%18%

35%17%

28%38% 34%

24% 32%11%

Individual team members' choices of Rubric factors requiring the most effort or most difficult for department to improve (n=71)

5


5) Pre-Institute: Comparison of highest priority, least effort required, and most difficult/most effort

Authentic

researc

h in classe

s/labs

Student cognitive

skills d

evelopment

Student meta

cognitive

skills d

evelop

...

Student core

competen

cies

Independen

t/ment

ored researc

h by st..

.

Use of b

est pedago

gies

Faculty d

evelopment

Assessm

ent

Disposi

tions of f

aculty and ad

ministra...

Integra

tion of departm

ent progra

m with Gen E

d g...

0%

10%

20%

30%

40%

50%

60%Pre-Institute: Comparison of perceptions of Rubric factors that have highest

priority, are least effort, and are most difficult

Highest prior-ity

Least effort

Most difficult

6


Post Institute Survey

In the first part of the post-Institute survey, participants were asked to share their overall impressions of the Institute, the Rubric, and the Plenary sessions. The second part of the survey addressed the Workshops and those data follow this section. Participants were told that the names and institutions would be kept confidential, unless SERP Fellows received specific permission to attribute a “personal testimonial” to its provider.

1) Overall Impressions

Participants were asked to consider their overall impressions of the SERP Institute. Their responses are aggregated below, from most positive to least. Responses revealed that all participants either strongly agreed or agreed that the atmosphere of the Institute was one of mutual respect and support and that the work accomplished at the Institute will benefit the regions students.

Strongly Agree75%

Agree25%

The atmosphere of the Institute was one of mutual respect and support.

Strongly Agree67%

Agree33%

I believe the work accomplished at this Institute will benefit our region's students.

7


Strongly Agree71%

Agree27%

Neutral2%

It was good to have these days to work and socialize with members of my own team

Strongly Agree59%

Agree40%

Neutral2%

I am proud of the work my team accomplished at the Institute.

8


Strongly Agree63%

Agree29%

Neutral8%

I and/or my team learned a lot from the talks and workshops.

Strongly Agree51%

Agree38%

Neutral10%

Disagree2%

My team's Action Plan is heavily influenced by what we learned at the Insti-tute.

9


Strongly Agree61%

Agree32%

Neutral6%

I want to continue to work within this Southeast PULSE network of institu-tions and organizations.

Strongly Agree48%

Agree43%

Neutral6%

Disagree2%

Strongly Disagree2%

I enjoyed my time at the Institute.

10


Strongly Agree38%

Agree44%

Neutral13%

Disagree5%

I and/or my team learned a lot from talking with other participants.

Strongly Agree33%

Agree46%

Neutral19%

Disagree2%

My team and I formed new relationships with teams from diverse institution types.

11


When asked to clarify low scores to the previous questions to help the SERP Fellows understand where they fell short in order to improve, participants shared the following information:

Many individuals said that they felt there was not enough time for reflection, interactions with other teams, between team members, and/or breaks; the schedule was jam-packed.

o Twelve specifically mentioned that there should have been more un-programmed time with other teams and/or more promotion of interactions/relationships/collaborations between teams, especially to listen to one another, learn from each other, for networking and conversations. The breakfast topics were a start, but not enough to let the true networking take place.

o Fifteen specifically mentioned there should have been more time for conversations between team members to reflect on what they heard, exchange thoughts after plenary sessions and workshops, share emerging ideas, work on action plans, etc.

Seven indicated that the accommodations could have been better and detracted from the overall experience.

Two people noted that the PULSE Fellows were not a diverse group of people.

Other comments made by single participants were:o The entire Institute use a more synergistic approach.o S/he did not understand the ongoing benefit of being involved with PULSE.o S/he would have liked to see more of his/her peer institutions (i.e. research) present.o His/her team would have benefited from having a 'consultant' from the institute sit in on

some brainstorming and plan formulation meetings during team planning time. o Fellows should sit with the team members more during meals to interact with them.o S/he did not find the panel discussion or administrators particularly helpful.o It would have been nice for those who were on campus without a vehicle to have access

to a van to go out for one of the meals.o The titles of sessions were sometimes misleading and the content wasn't what we

thought it would be. o The teams’ administrators should have met together at some point during the institute to

share ideas.

When asked what was particularly helpful and/or pleasantly surprising about the Institute, participants highlighted the following:

Opportunities to interact with and learn from other institutions:o Opportunities to interact with other institutions (e.g. Breakfast discussions) (5)o Benefits of interactions with a diversity of institution types (5).o Awareness that other schools are committed to refocusing their life science curricula and share

common goals (3)o Awareness that there are similar issues or concerns shared by institutions with varying

backgrounds (3)o Energy & enthusiasm of other institutions (1)

Plenary and workshop session presenters:o Learning about student metacognition through Saundra McGuire (9) was very inspiring and made

a big impressiono Workshops with practical applications (4).o The speakers and presenters (8). There was something for everyone, even those who already

were implementing reform efforts. Workshops were informative, interactive, and inspiring, especially:

12


o Integrated interdisciplinary workshop (3)o Michelle Withers

Team aspect:o Team members having time together, getting excited and engaged about their work together (4)o Pre-homework prepped them for Institute (2)o Action plan template (1)o Opportunity to interact with administrator (1)

Overall:o The enthusiasm, commitment, and support of the facilitators (5)o The resources collated at the SERP Institute/PULSE website are quite helpful (2)o Realizing there is a massive amount of peer-reviewed evidence for high-impact STEM pedagogy

that we do not useo Greg’s drawings (3)o Overall – just how useful, well-organized, collegial, valuable, productive, refreshing, engaging,

exciting (11) the Institute was. o The U of Richmond campus & people (4)o Good to have ASB follow-up (1)

TestimonialsQUESTION: Do you want these listed?

Impact of Greg Gersch’s work

Participants were asked to describe how the work of Greg Gersch (the graphic facilitator) impacted them, their team, and our collective work together. Seven categories emerged from their open-ended responses and are listed here from most frequent response to least, with the actual number of individuals in parentheses.

o His work helped to capture specifics what took place, served as a reference/reminder, new way to verbalize/visualize challenges, allowed us to reflect. Identified important concepts and easily represented them (23)

o His work served as an overall recap/summary or a collective memory of the complete message of the Institute, the system, the connected nature of the discussions, the atmosphere, the emotions (16)

o It was unique/interesting, but did not make an impact (8)o It was fun, added an element of play, enjoyable (5)o The panels/images were a conversation starter, will serve as inspiration for future team

conversations (4)o The panels/images will be useful to include in our own workshops or conversations for others

back home (3)o They were inspired to use the technique with students (3)

13


2) Efficacy of Rubric

Participants were next asked to provide reactions to three statements in order for the SERP Fellows to better understand the efficacy of the STEM Department Evaluation Rubric. They were asked to state the extent to which they agreed that:

1. Using the STEM Department Evaluation Rubric in advance of the Institute was good preparation for the Institute.

2. The STEM Department Evaluation Rubric is an effective tool for helping departments establish a shared vision for future work.

3. The Rubric’s ten factors are well-articulated across the scale from baseline to exemplary.

Participants were also asked to provide specific feedback on the use of the Rubrics QUESTION: Do you want me to go through those open-ended responses?

Strongly Agree68%

Agree26%

Neutral5%

Not applicable2%

Using the STEM Department Evaluation Rubric in advance of the Institute was good preparation for the Institute (n=63).

14


Strongly Agree49%Agree

44%

Neutral2%

Disagree2%

Not applicable3%

The STEM Department Evaluation Rubric is an effective tool for helping departments establish a shared vision for future work (n=63).

Strongly Agree; 35%

Agree; 48%

Neutral; 10%

Disagree; 5% Not applicable; 2%

The Rubric's ten factors are well-articulated across the scale from baseline to exemplary (n=63).

15


3) Impact of Institute on Rubric Topics

3.a. Participants were asked to rate the level to which the topics on the STEM Department Evaluation Rubric were more important to them and/or their team after participating in the institute.

The one topic that was “much more important” post-Institute to the most individuals was “Student Metacognitive Skills Development.”

The topics that were “more important” post-Institute to over 40% of respondents were “Authentic Research in Classes/Labs,” “Student Cognitive Skills Development,” “Student Core Competencies,” “Use of Engaging Pedagogies,” “Faculty Development.” Those that had no change were “Student/Mentored Research,” “Assessment,” and “Dispositions of Faculty/Administrators.”

Authent

ic Rese

arch in

Classes

/Labs

Student

Cognitiv

e Skills

Develop

...

Student

Metacog

nitive

Skills Deve

l...

Student

Core Com

petenc

ies

Indepe

ndent/M

entore

d Rese

arch b

y ...

Use of E

ngagin

g Peda

gogies

Faculty

Develop

ment

Assessm

ent

Disposi

tions o

f Facu

lty and

Adminis

t...

Integra

tion of

Departm

ent pr

ogram

with GE go

a...

0%10%20%30%40%50%60%70%80%90%

100%

Rate the level to which the topics on the STEM Department Evaluation Rubric are more important to you and/or your team now than they were prior

to being invited to participate in the SERP Institute.

Much More ImportantMore ImportantNo Change

When encouraged to provide clarifications of choices, o (8) stated that they were already focused on some of the factors, and therefore selected “no

change.” o (2) stated that they already knew that the Vision and Change recommendations, such as core

competencies, were important. They came to the Institute looking for tools (1) and inspiration (1). o They needed to consider what they can take on, need to focus on what’s feasible, and are

focusing on small changes, instead of everything at the same time (3)

16


o The sessions provided a lot of valuable insight and increased their motivation to work on particular factors (4)

o The team discussions, time together, allowed them to identify which were most important to them as a group (1)

o the plenary on the power of increasing student metacognition made a surprising (1) and strong impact (2)

o The plenary about SACS helped them to understand how to use it regarding high impact practices (1)

o (2) stated that the rubric was helpful in shaping prioritieso (1) stated the rubric will be helpful in verbalizing the factors to others back home.

o Most all items on the list are important (4)

o The institute overall increased factors’ meaning, significance (1)

o The institute provided tools (2), which makes it more likely they will implement

17


3.b. Participants were asked to rate the impact of the Institute on empowering them and their team to improve their department's work in each area identified on the Rubric. They were told that their reactions to these statements will help SERP Fellows understand the extent to which the Institute helped empower them and their team to enhance and/or implement new strategies upon return to your campus.

o Sixty percent of respondents stated that they were “much more empowered” to improve their department’s work in “Student Metacognitive Skills Development.”

o Over 50% of respondents were “more empowered” to improve their department’s work on “Authentic Research in Classes/Labs,” “Student Cognitive Skills Development,” and “Student Core Competencies.”

Authentic

Research

in Clas

ses/...

Student C

ognitiv

e Skill

s Deve

lo...

Student M

etacog

nitive

Skills D

eve...

Student C

ore Com

petenc

ies

Indepe

ndent/M

entored

Researc

h by...

Use of E

ngaging Peda

gogies

Faculty

Development

Assessm

ent

Disposi

tions o

f Faculty

and Adm

inis...

Integrati

on of D

epartment p

rogram

with GE g...

0%10%20%30%40%50%60%70%80%90%

100%Please rate the impact of the Institute on empowering you and your team to

improve your department's work in each area identified on the Rubric.

Much more em-poweredMore empowered

When encouraged to provide clarification for their selections, respondents said that they now felt:- In general, institute helped, feel more committed, increased awareness (3)- Workshops in general inspired them (1) - Learned a lot of valuable information (2)- Were empowered by having time to discuss as a team (1)- Were empowered by the rubrics (2)- Were empowered by Saundra McGuire’s presentation/workshop (1)- Understanding the use of SACS helped regarding importance of high impact practices(1)- Any factors marked “no change” reflect already a strong commitment/empowered (2)

18


4) Plenaries

There were 62 individuals who completed the questions about the plenaries. Participants were asked to share the degree to which each of the plenary sessions: 1) inspired them to learn more about the topic, 2) positively influenced their team’s action plan, 3) was important and/or thought provoking regardless of its influence on their action plan, 4) helped their team formulate practical ideas to implement on their campus, and 5) had information/lessons learned that their team plans to share with their whole department upon returning to campus.

a) Plenaries: Inspired them to learn more about the topic

Over 80% of the 62 responding participants strongly agreed that Saundra McGuire’s plenary on metacognition inspired them to learn more.

Additionally, almost 40% of participants both strongly agreed and agreed that Beth Ambos’ plenary on undergraduate research inspired them, and almost 40% of participants agreed that Sara Brownell and Erin Dolan’s plenaries inspired them to learn more. Approximately 30% of participants agreed that Daniel Wubah’s plenary talk and the PULSE presentation inspired them to learn more.

1: McG

uire

2: Ambo

s

3: Brow

nell

4: Dola

n

5: Wub

ah

6: Adm

in P...

7: PULS

E0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Inspired me to learn more about the topic

Strongly Agree

Agree

Neutral

Disagree

I did not attend this session

19


b) Plenaries: Positively influenced their team’s action plans

Just over 70% of the 62 responding participants strongly agreed that the plenary given by Sandra McGuire positively influenced their teams’ action plans.

Just over 30% strongly agreed and 40% agreed that the plenaries given by Beth Ambos and Erin Dolan positively influenced their plans.

1: McG

uire

2: Ambo

s

3: Brow

nell

4: Dola

n

5: Wub

ah

6: Adm

in P...

7: PULS

E

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Positively influenced my team's action plan

Strongly Agree

Agree

Neutral

Disagree


20


c) Plenaries: Was important and/or thought provoking regardless of influence on action plan

Just over 80% of the 62 responding participants strongly agreed that Saundra McGuire’s plenary on student metacognition was important and/or thought provoking, regardless of its influence on their teams’ action plan.

1: McG

uire

2: Ambo

s

3: Brow

nell

4: Dola

n

5: Wub

ah

6: Adm

in P...

7: PULS

E

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Was important and/or thought provoking regardless of its influence on our action plan.

Strongly Agree

Agree

Neutral

Disagree

I did not at-tend this session

21


d) Plenaries: Helped formulate practical ideas to implement on campus

Almost 70% of the 62 participants who responded to this question strongly agreed that Saundra McGuire’s plenary on student metacognition helped them formulate practical ideas to implement on their campus.

Just over 50% of participants agreed that Beth Ambos’ plenary on undergraduate research helped them in the same way.

1: McG

uire

2: Ambo

s

3: Brow

nell

4: Dola

n

5: Wub

ah

6: Adm

in P...

7: PULS

E

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

This session helped us formulate practical ideas to implement on our campus.

Strongly Agree

Agree

Neutral

Disagree


22


e) Plenaries: Takeaways from the Institute

In addition to the plenary speakers, the participants were asked to provide feedback on the final “Take-Aways” session, and 62 people responded to three questions about the session.

Was a good way to end the Institute.

Participants' ideas were welcomed and valued in

this session.

Revealed a sense of shared mission across

the teams.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

What are the Takeaways from the Institute?

Strongly AgreeAgreeNeutralDisagreeI did not attend this session

23


5) Workshops

Once participants completed the first survey, they were sent a link to the second survey to answer questions about the workshops and 57 individuals responded it.

1A 1B 1C 1D 2E 2F 3G 3H 3I 4J 4K 4L 5M 5N 5O0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

This workshop positively influenced my team's action plan (n=57).

Strongly Agree

Agree

Neutral or Disagree

The workshops that over 50% of respondents strongly agreed most strongly influenced team action plans were:

1A: Motivating Students to Use Metacognitive Strategies: It Might Be Easier than You Think! - Saundra McGuire

3G: Course-based undergraduate research experiences (CUREs): Definition, Outcomes, and Assessment - Sara Brownell and Erin Dolan

5N: Scientific Teaching and Engaging Pedagogies: "OK, We Know We Should, but How?" - Ellen Goldey

4J: Embedding authentic research and discovery into all levels of the curriculum - Nitya Jacobs, Judy Awong-Taylor, Alix Fink

1B: Using Vision and Change to Map and Plan Your Department's Curriculum - Nitya Jacobs 2F: Integrated and Interdisciplinary 1st year Course Models - April Hill, Laura Runyen-Janecky,

Will Case, Kathy Hoke, and Students 4L: Using Assessment Evidence to Inform, Transform, and Inspire Your Department (it's not just

for SACS!) - Ellen Goldey

24



10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Workshop inspired participant to learn more about the topic (n=57)

Strongly Agree

Agree

Neutral or Disagree

The workshops that over 50% of respondents strongly agreed inspired them to learn more were: 1A: Motivating Students to Use Metacognitive Strategies: It Might Be Easier than You Think! -

Saundra McGuire 5N: Scientific Teaching and Engaging Pedagogies: "OK, We Know We Should, but How?" - Ellen

Goldey 3G: Course-based undergraduate research experiences (CUREs): Definition, Outcomes, and

Assessment - Sara Brownell and Erin Dolan 4J: Embedding authentic research and discovery into all levels of the curriculum - Nitya Jacobs,

Judy Awong-Taylor, Alix Fink 2F: Integrated and Interdisciplinary 1st year Course Models - April Hill, Laura Runyen-Janecky,

Will Case, Kathy Hoke, and Students 4L: Using Assessment Evidence to Inform, Transform, and Inspire Your Department (it's not just

for SACS!) - Ellen Goldey 1B: Using Vision and Change to Map and Plan Your Department's Curriculum - Nitya Jacobs

25



10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Team plans to share what was learned with whole department (n=57)

Strongly AgreeAgreeNeutral or Disagree

The workshops that over 50% of respondents strongly agreed they would share what they learned with their whole department were:


1A: Motivating Students to Use Metacognitive Strategies: It Might Be Easier than You Think! - Saundra McGuire


2F: Integrated and Interdisciplinary 1st year Course Models - April Hill, Laura Runyen-Janecky, Will Case, Kathy Hoke, and Students

1B: Using Vision and Change to Map and Plan Your Department's Curriculum - Nitya Jacobs

26



10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Workshop was interesting or thought provoking regardless of its influence on team's action plans (n=57)

Strongly Agree

Agree

Neutral or Disagree

The workshops that over 50% of participants strongly agreed were interesting or thought provoking, regardless of their influence on their team’s action plan, were:


IA: Motivating Students to Use Metacognitive Strategies: It Might Be Easier than You Think! - Saundra McGuire

2F: Integrated and Interdisciplinary 1st year Course Models - April Hill, Laura Runyen-Janecky, Will Case, Kathy Hoke, and Students


1B: Using Vision and Change to Map and Plan Your Department's Curriculum - Nitya Jacobs 4L: Using Assessment Evidence to Inform, Transform, and Inspire Your Department (it's not just

for SACS!) - Ellen Goldey

27


Pre-Post Comparisons

In order to get a sense for how the Institute influenced participants’ perspectives and abilities to work on advancing the rubric factors, pre and post participants responses were compared.

The comparison of pre-Institute to post-Institute responses of the factors they perceived as highest priority for their department to improve revealed the impact of the Institute on some of the factors, as illustrated in Table 1.

The top two factors rated as highest priority prior to the Institute, “Authentic Research in Classes/Labs” (45%) and “Use of Engaging Pedagogies, (49%) both were regarded as more important by 48% and 46% respectively, and much more important by 25% and 29% respectively.

There were three factors that were rated second highest in priority by 35% of respondents prior to the Institute. One of those, “Student Metacognitive Skills Development,” was recognized as being much more important by 65% of participants after the Institute. 25% of respondents felt that “Student Cognitive Skills Development” were much more important, and only 8% felt that “Student Core Competencies” were much more important.

It is important to note that a number of participants provided further clarification of their choices, and some stated that they selected the “no change” option because certain factors were already high priorities for their department to improve. See section above for a summary of other reasons for their choices.

Table 1. Comparison of highest priority prior to Institute vs. degree of importance post-Institute. (73 people responded pre-Institute, 62 post-Institute.)

Rubric FactorPre-Institute

Highest priority

Post-institute no change in importance

Post-Institute more

important

Post-Institute much more important

Authentic Research in Classes/Labs 45% 27% 48% 25%

Student Cognitive Skills Development 35% 24% 51% 25%

Student Metacognitive Skills Development 35% 5% 30% 65%

Student Core Competencies 35% 43% 49% 8%

Independent/Mentored Research by Students 24% 56% 37% 8%

Use of Best/Engaging Pedagogies 49% 25% 46% 29%

Faculty Development 30% 35% 49% 16%

Assessment 27% 49% 33% 17%

Dispositions of Faculty and Administration 15% 46% 33% 21%

Integration of Department program with GenEd goals and Institutional Mission

4% 44% 44% 11%

28


There were five Rubric factors that over 30% of the 73 respondents prior to the Institute said were most difficult or required the most effort for their department to improve. These are in bold font in Table 2. The table illustrates their pre-Institute responses and the impact the Institute made on the degree to which participants felt empowered to improve the Rubric factors.

60% of respondents felt much more empowered, and 37% felt more empowered to improve “Student Metacognitive Skills” in their department, which was identified by 35% pre-Institute as most difficult or requiring the most effort for their departments to improve.

43% said they felt much more empowered and 37% felt more empowered to improve the “Use of Best/Engaging Pedagogies” in their departments, which was identified by 38% pre-Institute survey as most difficult/requires most effort for their department.

27% of respondents felt much more empowered and 54% more empowered to improve their departments’ work to provide “Authentic Research in Classes/Labs,” which was selected by 37% of respondents pre-Institute as most difficult/requires the most effort.

“Student Metacognitive Skills Development”

Table 2. Comparison of factors described as most difficult of requiring the most effort to improve in their department prior to Institute vs. degree of to which the Institute empowered them. (73 people responded pre-Institute, 62 post-Institute.)

Rubric FactorPre-Institute Most effort or most difficult

Post-Institute

No change

Post-InstituteMore

Empowered

Post-InstituteMuch More Empowered

Authentic Research in Classes/Labs 37% 19% 54% 27%

Student Cognitive Skills Development 18% 22% 54% 24%

Student Metacognitive Skills Development 35% 3% 37% 60%

Student Core Competencies 17% 37% 52% 11%

Independent/Mentored Research by Students 28% 41% 41% 14%

Use of Best/Engaging Pedagogies 38% 21% 37% 43%

Faculty Development 34% 30% 44% 25%

Assessment 24% 44% 37% 19%

Dispositions of Faculty and Administration 32% 43% 37% 21%

Integration of Department program with GenEd goals and Institutional Mission

11% 40% 46% 14%

29


Action Plans

In order to determine the impact of the Institute on the teams’ plans for improving their department, their action plans were reviewed.

First, each “Priorities and Vision for Department Transformation/Improvement” section was read to identify whether the teams identified, in these statements and lists, a particular factor on the STEM Department Rubric.

Percent of teams’ action plans “Priorities and Vision” statements that reference the STEM Department Evaluation Rubric factors.

Authentic in-class/lab research

Student Cognitive Skills

Student Metacognitive Skills

Student core competencies

Independent research

Best/Engaging Pedagogies

Faculty Development

Assessment

Dispositions of Facluty and Admin

Integration of Department program with GE goals and Institutional

Mission

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

60%

45%

55%

50%

25%

55%

35%

15%

30%

5%

It is important to note that some teams stated their priorities or vision in very general terms. Therefore it was more difficult to identify a specific Rubric factor. For example, one department wrote that they “want greater integration and assessment of all PULSE/V&C priorities, state-level efficiencies, and university initiatives & Gen Ed goals” which means they want to do it all. Others were much more specific and therefore recorded in the right-hand column. In addition, some team’s priorities and visions did not specify the extent to which they want to work on a particular Rubric factor. For example, one team wrote: “develop a repository of effective teaching practices,” however it is not clear the extent to which the team would like the repository to be used, or whether faculty development will also be offered to support faculty in their implementation of these items. Therefore, the table represents, as best as possible, the number of teams who included Rubric factors as part of their priorities and vision statements.

NEXT: Scan entire action plan

FINALLY: summary of interviews

30

api.ning.comapi.ning.com/.../copyofserppostassessmentreport_o… · web viewthe data from these...

Documents