the stars alliance students & technology in academics, research, and service stars alliance...

The STARS AllianceStudents & Technology in Academics, Research, and Service

STARS Alliance Evaluation

Tiffany Barnes

Evaluation Team

Kim Buch, Audrey Rorrer

Web: Anthony Chow

Qualitative & PP: Sally Berenson

The STARS AllianceStudents & Technology in Academics, Research, and Service 2Barnes, Eval Team

NSF Site Visit 2007

Evaluation• Successes:

– Refined approach to evaluate Organizational, program, individual level outcomes

– Instrument development– Combined qualitative and quantitative measures for:

• SLC• Alliance effectiveness

– Articulated new goals and measures (STARS/Computing Identity)

• Challenges:– Changing team– New research questions arise all the time– Comprehensive program has many goals - too many surveys for same

people– Data collection from different people/institutions– Small team


NSF Site Visit 2007

Impact on pipeline

• Q1: The STARS Alliance features a broad scope (middle school to graduate school) across three categories of population (gender, ethnic, disability), with a variety of outreach, research, mentoring, and career counseling components. Characterize, quantitatively if possible, what has been attempted within this scope of activity and what has been achieved to date. To what extent have each of the target groups been reached? Each of the pipeline stages?

• Approximately 110 college SLC students each year– About 40% returning in Year 2

• Over 500 K-12 students at outreach events• Over 500 parents• Over 100 teachers and counselors• > 5 major community events (AU, FSU, UNCC)


NSF Site Visit 2007

Measures

• Q3: What new measures of impact/outcome have been developed to capture what the Alliance is seeking to achieve, i.e., beyond traditional pre-post test measures appropriately disaggregated (by gender, ethnicity, disability, and cohort) such as GPA, year-to-year retention, and degree attainment? For example, how have the observations in students' electronic journals been used by the Alliance institutions and Advisory Boards?

• Selected interviews each semester• Journaling currently not being analyzed• Climate survey to measure impact of STARS on

computing culture at schools• New outreach evaluation instruments


NSF Site Visit 2007

Comparison

• Q7: What data is the Alliance using for comparison purposes? Are they national sources, local institutional information on students, some combination, or other? Explain.

• Comparison of applicants, enrollments across years

• Comparison with NSF and Taulbee survey data


NSF Site Visit 2007

Celebration impacts

• Q4: Describe the benefits of the summer workshop to different categories of participant.

• Faculty- networking, project collaborations, information exchange, e.g.

– o Junior Faculty Round Table– o Implementing Pair Programming

2007 I tem 100% My expectations of the conference were met. 100% I w as satisfied with the conference overall.

100% I have developed helpful professional collaborations through attending the conference.

100% I believe that the celebration had a positive impact on individual student who attended.

90% I think that the conference prepared students to conduct successful SLC projects.

90% The STARS Celebration inspired me to become more involved in BPC efforts.


NSF Site Visit 2007

STARS Celebrations

The STARS Celebration emphasizes STARS core values of excellence, leadership, civic engagement, service, and community – values intended to foster student success.

•2007 at UNC Charlotte– 113 students– Over 40 faculty– Over 50 partners and community

•2006 at Georgia Tech– 105 students– 23 faculty/staff


NSF Site Visit 2007

STARS Celebration Results

• Students felt the Celebration provided opportunities for community building. (2006: 97% , 2007: 92% )

Students learned how computing professionals can improve quality of life. (2006: 83%, 2007: 89%)

Students learned about disparity of representation in IT: by gender and race (2006: 98%, 2007: 88%) by persons with disabilities (2006: 86%, 2007: 69%)

Students had ample time to select leadership assignments to fit their professional needs (2006: 80%, 2007: 88%)


NSF Site Visit 2007

Demonstration Projects

The STARS Alliance serves as an

incubator for new demonstration projects and the

scaling and replicating of best practices among

the diverse alliance

institutions.

STARS Leadership Corps

Pair Programming

Tiered mentoring model

Teaching Math to the Visually Impaired

African American Researchers in Computer Science

Culturally Situated Design Tools


NSF Site Visit 2007

STARS Leadership Corps Model


NSF Site Visit 2007

STARS Tiered Mentoring and Role Models

High school student

Middle school student

Professional or Grad student

Undergrad student

Regional Partnerships -A computing

community from Kindergarten to the Workforce


NSF Site Visit 2007

Evaluation

• Three-Tiered Outcomes Model:– Organizational Outcomes (e.g., Alliance)– Program Level Outcomes (e.g., Demonstration

Projects)– Individual Level Outcomes (e.g., Students)

• Evaluation model: CIPP– Context (setting of project)– Input (project resources)– Process (activities & strategies)– Product (ultimate results)


NSF Site Visit 2007

Alliance Outcomes

• Increased student enrollment in computing and IT programs

• Increased student awareness about computing and IT• Increased student readiness to enter computing and IT

graduate school and workforce• Increased participation of undergraduates entering

computing and/or IT graduate school or workforce• Increased graduate rates of computing and IT students• Increased persistence and declaration of students

majoring in computing or IT• Increased college adjustment and GPA for students in

computing or IT


NSF Site Visit 2007

Alliance Outcomes

• Sustained Alliance Efficacy• Institutionalize Alliance Partnerships• Increased national awareness of effective practices for

Alliance development• Serve as a model and repository for BPC


NSF Site Visit 2007

Year 1 Accomplishments

• Steering Committee:– Jan. 2006-2007, & Aug. 2006-2007– Regular teleconferences for planning & update

• STARS Celebration– Successful Annual Conference held in August 06-07– 2006: 105 students, 23 faculty, 2007: 113 students, >40 faculty– 2007: New evening for partners, >50 additional attendees

• Student Leadership Corps– 2 universities implemented in Summer 2006– All implemented in Fall 2006, Spring 2007, Fall 2007

• Workshops– Pair Programming, Mentoring, IBPC: Jan & Aug 2007– CSDT at Celebration Aug 2007– Qualitative methods workshops Aug 06-07, April 2007


NSF Site Visit 2007

Accomplishments

• Web portal: StarsAlliance.org• Evaluation

– Instrument development– IRB exploration: applying for waivers for K-12

outreach evaluation– Interviews & Training

• Coordination with UNC Charlotte REU Site• Faculty recruiting at UNC Charlotte!


Results


NSF Site Visit 2007

Institutional Enrollment


NSF Site Visit 2007

SLC ’06-’07 Participants

SLC Participants

Females Males Total

African American 20% 23% 43%

Asian 7% 7%

Caucasian 29% 19% 48%

Total 58% 42% 100%


NSF Site Visit 2007

2006-2007 SLC

107 students participated in 2006-2007 STARS Leadership Corps

K-12 Outreach18%

Peer Ambassadors

13%

Campus Organization

17%Community

Service7%

Research25%

Internships3%

Peer Mentors17%


NSF Site Visit 2007

SLC Survey Results

• Retention of SLC students: Over 95% are forming meaningful peer & faculty relationships 88% more committed to computing majors 94% interested in graduate education 83% felt the SLC helped them develop computing skills 91% would recommend the SLC to others

Recruiting: 90% are helping others understand the value of computing Hundreds of K-12 students have participated in outreach events

Building a “like” community:88% felt the SLC gave them opportunities to work with people like themselves


NSF Site Visit 2007

Who is SLC?

• NOT the “average” UG student in Computing– Older, average age= 24 years– Higher GPA in Computing, M=3.3– Higher SES

• 85% “middle class”• 64% had parents with at least some college; 53% were college graduates

– Much more diverse• Women 51%• Under-represented minorities 62%• Physical disability 14%

– Extremely confident that they will obtain UG degree (4.7 on 5-pt scale)– Very likely to attend graduate school

• 66% believe they will obtain Masters• 50% believe they will obtain PhD


NSF Site Visit 2007

SLC & Computing

• 93% are confident that Computing is the right major• 87% plan to stay in the field of Computing long-term• Have a very strong “Computing Identity” (M=4.4/5)

– “Feel I fit in field of Computing” M=4.2– “Expect to be treated fairly in Computing career” M=4.1– “Can relate to the people around me in my classes” M=4.3– “Committed to applying Computing to benefit society” M=4.5– “Committed to promoting Computing to others” M=4.3

• Have a very strong “Computing Efficacy” (M=4.3/5)– “Feel someone like me can succeed in a Computing career” M=4.6– “I can complete the programming requirements for Computing majors” M=4.5– “I can complete the math requirements for Computing majors” M=4.6– “I can complete the science requirements for Computing majors” M=4.6– “I can complete my current Computing degree” M=4.6


NSF Site Visit 2007

SLC & Computing departments?

• 64% at least weekly contact with Computing faculty• 77% satisfied+ with the degree of interaction with faculty• 67% satisfied+ with the availability of extra-curricular

activities, eg, ACM/IEEE• 73% satisfied+ with “welcoming” atmosphere in the

department• 47% satisfied+ with availability of

“cutting-edge”/innovative courses• 70% feel satisfied+ that “I am part of the department”• 73% overall satisfied+ with the department


NSF Site Visit 2007

SLC & Diversity in Computing

• 66% satisfied+ w/ the diversity of dept. students and faculty• 64% agree that dept. students are sensitive to minority

issues• 77% agree that dept. faculty are sensitive to minority issues• 71% agree that dept. students are sensitive to women’s

issues• 78% agree that dept. faculty are sensitive to minority issues• 69% agree that dept. students are sensitive to

disability/accessibility issues• 77% agree that dept. faculty are sensitive to

disability/accessibility issues


NSF Site Visit 2007

SLC Concerns• Computing faculty contact per semester

– 6% reported NO contact– 5% reported they only had ONE contact– 12% reported they had one contact per month

• Students rated their high school technology/ computing program as:– 36% Not good– 23% Somewhat good– 14% Very good, 14% Excellent

• Students felt that their depts were NOT sensitive to:– 36% minority issues– 30% women’s issues– 31% disabilities/access issues


NSF Site Visit 2007

SLC quote

• “The STARS Alliance has engaged me to use computing to make a positive impact by exposure to other areas of technology and to be of service to other students and community members. It has provided them (middle school students) access to computing resources. It has allowed them to see women and minorities as role models.”


NSF Site Visit 2007

SLC Interviews

• Emergent themes toward careers are altruism, creativity, and mentorship.

• Initial early interest in computing: game playing that requires problem solving, along with a motivation to be self-sufficient.

• Overall, students stated that a family member or mentor had been influential in their decision to pursue a career in computing.

• The overwhelming majority of students indicated that their reason for persistence in computing is due to a commitment to giving back to their communities and an excitement about the creativity inherent in computing careers.


NSF Site Visit 2007

Celebration Results

• 97% of students felt the workshop provided adequate opportunities for community building.

• 83% agreed with the statement that the SLC Workshop emphasized how computing and IT professionals can use their skills to improve quality of life


NSF Site Visit 2007

Alliance structure

• Collaborations are beneficial to both students and faculty

• Participation facilitates faculty and student career development

• Alliance goals are being achieved and represent meaningful impact to the computing field.

• Training, dissemination, and sustainability goals are being met


NSF Site Visit 2007

Presentations

• Grace Hopper Celebration, Oct 2006• SIGCSE presentation to computing K-12

teachers, funded by Microsoft• Presentation at NIU, August 2007• Panel at GHC 2007• Panel & Birds of a Feather at Tapia 2007• Paper at Frontiers in Education 2007


Successes & Challenges


NSF Site Visit 2007

Alliance infrastructure

• Time is major issue for both faculty & students• Need tools to facilitate:

– Collaboration– Communication– Management– Timelines & deadlines– Tracking– Reporting– Sharing resources


NSF Site Visit 2007

Cross-institutional collaboration

• Faculty see this as very successful

• Students excited to meet students from other places

• Recruiting across Alliance!

• Looking forward to building more


NSF Site Visit 2007

Synergy with existing programs

• Essential to success

• Examples:– Charlotte REU & CRA-DMP Summer 2006– Mentoring on NCSU campus– Intel Opportunities at Georgia Tech

• Share resources and minimize work


NSF Site Visit 2007

Recruiting students

• A challenge at first: What is STARS?

• Now: More students want to join!

• Creating ways for non-funded students to join– STARS Alliance club– Course credit, other avenues

• Creating ways to recognize participation


NSF Site Visit 2007

Replication of best practices

• A Challenge!• How do you make professors do something

new?• You must show how it benefits them• Need a cookbook approach


NSF Site Visit 2007

Sustaining changes

• Student STARS Alliance club• STARS course added at UNC Charlotte• Faculty recruiting at UNC Charlotte:

– 2 new black faculty, 2 new women

• Existence of Alliance is raising awareness:– Deans and departments are using STARS Leaders

for recruiting and involving them in dept activities and policy-making, marketing


NSF Site Visit 2007

Effects of Alliance on faculty careers

• Writing circles– Planned for Year2

• Mentoring within the Alliance– happening informally

• Collaborations– Landmark College, REU Site proposal, Gaming

programs at UNCC, Auburn

STARSAlliance.orgStudents & Technology in Academia, Research, and Service

PI: Teresa Dahlberg [email protected]: Tiffany Barnes [email protected]

Evaluation TeamKim BuchAudrey RorrerSarah BerensonAnthony ChowJennifer Thomas

mailto:[email protected]




the stars alliance students & technology in academics, research, and service stars alliance...

Documents

impact of stars

stars celebrations

eval team nsf site visit

team new research questions

college slc students

students electronic

slc alliance effectiveness

diverse alliance institutions