Assessment & Evaluation

NSF CISE REU Sites PI Meeting

March 18, 2011

Project Supporters for 2010 Implementation

2009 Working Group MembersGuy Alain Amousou Chris Aberson Wendy Cooper Teresa Dahlberg Andy Fagg Stephen Gilbert Manfred Huber Niels Lobo Sanjay Madrias Joan Peckham Eric Wong Yu-Dong Yoa

Kevin Zeng

2010 Implementation MembersBrooklyn College, CUNY Iowa State University University of Alabama UNC Charlotte

Colorado Springs Jackson State University University of Central Florida University of South Carolina

Dakota State University Louisiana State University University of Texas Austin

Depaul University Montclaire State University University of Houston University of Wisconsin Oshkosh

Depauw University Marshall University University of Illinois Urbana-Champaign

Hope College Oklahoma University University of Massachusetts Amherst

13 used Common Application 20 used Survey

2010 Assessment Project

• Common Application • a la carte Survey • Evaluation Toolkit

– Coitweb.uncc.edu/reu/toolkit

• Future Directions– Refinements– Longitudinal Follow up

Common Application 2010

• 13 sites utilized• Total of 1,006 applicants• Most students applied to only 1

– One student applied to 30– 2 to 3 schools was common

• Application Range: 29-152• Average Number of Applications per Site: 77• ~ 130 accepted = 18% approximate acceptance rate

697

309

Unique Applicants Applied to 1+Sites

Unique Applicant Gender

Gender

Female27%

Male72%

Unspecified1%

Males: 500

Females: 187

Unspecified: 10

Unique Applicant Ethnicity

68124

524

426

149

Ethnicity

10%

18%

7%1%61%

2%1%Asian

African American

Hispanic/Latino

Pacific Islander

Caucasian

Native American

Multi-ethnic/Other

Applicant Level in School

86

191

294

123

Level

Freshman12%

Sophomore28%

Junior42%

Senior18%

Freshman

Sophomore

Junior

Senior

Degrees Considering

Degree most interested in pursuing

Masters programs: 479 (69%)

Phd programs: 180 (26%)

Undecided: 3 (<1%)

31 (5%) not considering graduate programs

A la Carte Survey 2010• Pre survey: May-June 2010• Post survey: July-August 2010• Total N: 339 REU Students; 20 Sites

Pre Demographics (N=197):

Gender- 69% Male (135), 28% Female (56), 3% Unspecified (6)

Ethnicity- 61% Caucasian, 10% African American, 10% Asian, 7% Other, 6% Hispanic, 5% unspecified, 1% Native American

Level- 57% Senior, 31% Junior, 11% Sophomore, 1% Freshman

Post Demographics (N=142):

Gender- 70% Male (99), 30% Female (43), Unspecified (0)

Ethnicity- 63% Caucasian, 10% African American, 10% Other, 7% Asian, 5% Hispanic, 1% Native American, 5% unspecified

Level- 62% Senior, 27% Junior, 10% Sophomore, 1% Freshman

Methodology

• Items– 4 point Likert type scale, 4 being positive– Some items were reverse scored– Collapsed into construct means representing 4 variables– Ethnicity collapsed into URM status

• Reliability

– Coefficient alphas above .547 • MANOVA

– To test hypothesis that there would be differences between means based on time, gender, URM status

What was measured• Self Efficacy (15 items)

– I can formulate a research problem

• Intent to attend graduate school (15 items)– I plan to apply to graduate school in a computing discipline

• Attitudes towards computing (38 items)– I like to use computer science to solve problems

• Help seeking/coping skills (30 items)– When I do poorly on an exam, typically I….skip class

Outcomes• Increases at Post Assessment were not significant• No main effects, no interactions

Construct Time Mean Std. Deviation

Self- Efficacy pre 3.05 0.47

post 3.43 0.40

Intent pre 3.10 0.46

post 3.19 0.47

Attitude pre 3.59 0.32

post 3.65 0.29

Help- Coping pre 2.94 0.33

post 2.98 0.37

What does this mean?

• We know there is impact from the REU experience

• More investigation is needed

• How to detect impact– Cultural nuance may be missed by survey with small

sample of women and URMs relative to men

Post Program Evaluation

• 38% reported plans to participate in future REU

• For 18%, this was not their first REU

QuickTime™ and a decompressor

are needed to see this picture.

Implications & Limitations

• Effects are small given the time frame

• Effects may be delayed• Are we measuring the

right things?• Need more exploration

• Self-report• Sample• Possible ceiling effect• Consider time series

design

Next Steps

Common Application• Track offers• Follow up with

applicants• Compare

accepted/declined on key indicators

Survey• Deeper analysis• Larger sample• More variables, or

different variables• Control groups?• Learning outcomes?

Collaborations for Research & Writing Circles

DiscussionWhere do we want to go from here?

What do we want to know?

CISE REU Evaluation Toolkit

http://www.coitweb.uncc.edu/reu/toolkit/