Who Are Our Students???Assessment Forum

Monday, Feb. 9, 2004

Donald J. Lerew, Jr. – Institutional Researcher

Ronald Burwell – Professor of Sociology

Overview

• Welcome – Review of Agenda

• IR & Assessment Website & Data Sets

• Assessment Calendar

• AICUP First-Year Student Survey 2003

• Senior Interviews – Some Highlights

• NSSE Student Engagement Survey 2002

• CSS Senior Survey 2003

Talking Points

• How do seniors view their time at Messiah? • How many hours do our students spend in

academic pursuits in a given week? • How do students’ views change regarding gender

roles? • Who stays and who leaves…and why? Social vs.

Academic Factors? • How much effort do our students put into their

college experience?

Research/Assessment Web Site

• General, college-wide statistics

• IR Publications Listing

• Assessment Calendar

• Assessment data– CIRP, CSS, NSSE, CSEQ, SSI, FYSS

• Location: Messiah Home > Campus Offices > Institutional Research

Survey Data Sets

• Most analyses have datasets available

• Datasets typically created for SPSS

• Contact Ron or I for a copy

Assessment Calendar

• Schedule of planned institutional assessment activities over next several years

• Presented and reviewed by Assessment Committee

• Goal is to not overburden students, while gathering sufficient, meaningful data

• Calendar is available on IR website

Assessment Survey Calendar November 2003

Year

HERI CIRP

[1st-Yr]

HERI YFCY [1st-Yr]

AICUP First-Yr St.

[1st-Yr]

Ind. U. NSSE

[Fr. / Sr.]

HERI CSS

[seniors]

Ind. U. CSEQ

[seniors]

Noel Levitz SSI

[all yrs] Senior

Interviews

Ind. U. FSSE

[Faculty]

1994-95

1995-96

1996-97

1997-98

1998-99

1999-00

2000-01

2001-02

2002-03

2003-04

2004-05

2005-06

2006-07

– survey was conducted – survey is scheduled shaded cell – survey was not available CIRP – Cooperative Institutional Research Program Freshman Survey YFCY – Your First College Year AICUP First-Year Student Survey NSSE – National Survey of Student Engagement CSS – College Student Survey

CSEQ – College Student Experiences Questionnaire SSI – Student Satisfaction Inventory Senior Interviews FSSE – Faculty Survey of Student Engagement

Assessment Calendar

• CIRP – Cooperative Inst. Research Program– Survey of New Freshmen at orientation– Baseline picture of who students are before

college

• AICUP First-Year Student Survey– Survey of first-year students in spring semester– Assesses academic and personal development

of students during their first year of college

Assessment Calendar

• NSSE – National Survey of Student Engagement– Given to 2nd-semester freshmen and seniors

– Studies quality of UG learning on campus by way of student engagement

• CSS – College Student Survey– Surveys Messiah seniors

– Students’ self-reported academic and campus life experiences

Assessment Calendar

• CSEQ – College Student Experiences Questionnaire– Surveys seniors

– Assesses the quality of student effort in utilizing resources/opportunities at college

• SSI – Student Satisfaction Inventory– Surveys subset of each class level

– Measures student importance and satisfaction to various campus issues

Assessment CalendarOther Possible Instruments

• YFCY – Your First College Year– Supplement to CIRP

• FSSE – Faculty Survey of Student Engagement

• Others???

AICUP – First-Year Student SurveySpring 2003

• Survey of first-year students

• Surveyed during spring semester

• Surveyed through campus mail

• General and retention analyses conducted

AICUP – First-Year Student Survey--Self-Assessment--

• MC men are significantly less prepared & are significantly more challenged in the area of study skills during the first year

• MC students indicate study and writing skills as their two biggest first-year skill challenges

• Both self-rating and first-year assessment of spirituality traits were significantly better than comparison institutions

AICUP – First-Year Student Survey--Transition to College--

• MC women experience significantly more homesickness than men

• MC students more likely to be homesick than at comparison institutions

• MC men report significantly less perceived help in areas of:– Dealing with roommate problems– Dealing with personal problems– Working with advisor

AICUP – First-Year Student Survey--Academic Advising--

• MC men relied significantly more on themselves for academic advice than did women (M-28.6%, W-15.4%)

• MC students placed more emphasis on soliciting academic advice from family (MC-25.9%, Comp-15.1%)

• MC students placed less reliance on the advisor (MC-21.4%, Comp-29.0%)

AICUP – First-Year Student Survey--Academic Advising--

• MC women met with advisor more often during first year (M-2.55, W-3.26 mtgs.)– Comparison group averaged 3.80 meetings

• MC students rated advisors significantly better than comparison group students in:– Knowledge level, availability, & overall

satisfaction

AICUP – First-Year Student Survey--Use of Time--

• MC students spend nearly 22 hrs/wk attending class or studying/doing homework

• MC men spend significantly more time than women watching TV and playing video/computer games

• MC women spend significantly more time in prayer and meditation

AICUP – First-Year Student Survey--Use of Time--

• MC students spend significantly more time than comparison group students in:– Prayer/meditation– Studying and completing homework– Participating in student clubs/groups

AICUP – First-Year Student Survey--Satisfaction--

• MC women significantly more satisfied in:– Quality of academic advising, quality of

academic offerings, availability of academic offerings, Internet access, faculty attitude toward students, and personal counseling

• MC students indicated significantly higher satisfaction than comparison students in:– Library facilities/resources, food services

AICUP – First-Year Student Survey--Satisfaction--

• 93.7% of MC students indicated that they would “definitely” or “probably” return for the second year (Comp. Group - 90.2%)

• Prominent reasons for not returning– Transferring– Dissatisfaction with social atmosphere– Lack of financial resources

AICUP – First-Year Student Survey--Retention Analysis: General--

• Attrited students reported significantly…– Higher levels of homesickness and loneliness– Lower levels of belongingness

• Attrited students indicated significantly less college help in the areas of:– Personal problems, student club involvement,

organized social events, rec. athletic activities

AICUP – First-Year Student Survey--Retention: Skills Assessment--

• Attrited students rated themselves significantly lower in the areas of:– Intellectual self-confidence, social self-

confidence, ability to function independently, leadership ability, reading analytically, and spirituality

• While incoming skill levels differed, no significant differences in reported growth during first year

AICUP – First-Year Student Survey--Retention: Other Items--

• Attrited students were significantly more likely to indicated the advisor as the primary source of academic advice; equally less on self-reliance

• Retained students spent more time socializing with friends (7 vs. 4 hrs/wk)

• Both student groups indicated similar academic, residence hall, and social life expectations– Attrited students indicated significantly lower levels of

having those expectations met during the first year

Senior Interviews – Spring 2002

• Approximately 27 completed interviews with seniors during the spring of 2002

• Transcriptions created for 24 interviews (sound quality created some difficulties in transcription)

• Topics: Educational Experiences; Religion; Gender

Educational Experiences

• Based on questions from Richard Light– Academic regrets– Unforgettable experiences– Critical moments– Important books– Interactions in the residence hall

Religion

• Awareness of and introduction to college’s religious heritage [related to Lilly Project]

• Important formative faith experiences

• Experiences with doubt

• Understanding of what is a “mature Christian” [James Fowler]

Gender

• Views on male / female roles with society and the family

• Views on ordination of women and role in the church

• Impact of Messiah College on ideas regarding gender

Senior Voices

• Academic regrets

• Unforgettable experiences

• Residence Hall interactions

• Views on gender roles

• Ordination of women

NSSE – Student EngagementSpring 2002

• Survey of 1st-yr. and senior students

• Determine extent to which students engage in educational practices that are associated with learning outcomes

• Surveyed during spring semester

• Web administration

• General and school analyses

NSSE – Student EngagementPercentile Ranks

Benchmark Cl MCScore

Bac-Gen

Bac-Lib Arts

NSSE2002

Level of academicChallenge

FrSr

57.662.2

90-10090-100

40-5060-70

80-9090-90

Active/collaborativeLearning

FrSr

45.850.8

70-8040-50

60-7040-50

80-9060-70

Student/FacultyInteractions

FrSr

35.945.1

40-5040-40

10-2010-20

50-6060-70

Enriching Educ.Experiences

FrSr

65.257.6

90-10090-100

40-5050-60

80-9080-90

Supportive Campus Environment

FrSr

65.566.0

70-8080-90

60-7080-90

80-9090-100

NSSE – Student Engagement--Acad/Intellectual Experiences--• Only 21% of MC seniors worked with other

students on projects during class “often” or “very often” (Bac-Gen & NSSE – 43%)

• MC first-year students worked significantly more often with classmates outside of class to complete class assignments

• MC students discussed class ideas with others outside of class significantly more often than comparison students

NSSE – Student Engagement--Acad/Intellectual Experiences--• MC women were more likely than men to:

– Use e-mail to communicate with instructor

– Work harder than they thought they could

– Discuss ideas with others outside of class

– Work with classmates outside of class

• Students w/ a “B- or lower” average were more likely to:– Come to class unprepared

– Work with others on projects during class

NSSE – Student Engagement--Mental Activities--

• Five Levels: (1) Memorize, (2) Analyze, (3) Synthesize, (4) Make Judgments, (5) Apply

• MC students had similar category breakdowns to comparison students– Exception: both first-year and senior MC

students indicated lesser levels of memorization

• MC students w/ “B- or lower” indicated significantly higher usage of memorization

NSSE – Student Engagement--Reading & Writing--

• MC students indicated a significantly higher number of assigned textbooks/readings than comparison students

• MC students indicated writing significantly more 5-19 and under-5 page papers

NSSE – Student Engagement--Educational Experiences--

• MC students indicated significantly higher participation/plan to participate rates in:– Practicums, internships, or field experiences

– Community service or volunteer work

– Foreign language coursework

– Study abroad

– Culminating senior experience (exam, capstone, project, etc.)

• MC women are more likely to attend

NSSE – Student Engagement--Time Usage--

• MC students spend significantly more time preparingpreparing for class than do comparison students (FY-14.5, Sr.-14 hrs/wk)

• MC FY students spend 11 hrs/wk relaxing/socializing; 9 hrs/wk for seniors

• MC students work significantly more on-campus, but significantly less off-campus

NSSE – Student Engagement--Time Usage--

• MC first-year women spend significantly more time preparing for class; senior men, however, spend more time than women

• MC students with a “B- or lower” spend:– Less time preparing for class (9 vs. 14 hrs/wk)

NSSE – Student Engagement--Educational/Personal Growth--

• MC students indicated significantly more growth in:– Acquiring a broad general education

– Developing personal code of values/ethics

– Contributing to welfare of community

– Writing clearly and effectively

– Thinking critically and analytically

• MC first-year students reported significantly less growth in the use of computer/info. technology than did comparison students

NSSE – Student Engagement--Institutional Emphasis--

• MC students reported significantly more emphasis by the college on:– Spending significantly more time studying and on

academic work (seniors only)

– Encouraging diversity contacts/interaction

– Providing support needed to thrive socially

– Attending campus events and activities

• Messiah women indicated higher perceived emphasis toward academic and social support

NSSE – Student Engagement--Satisfaction--

• Over 90% of MC men and 95% of women rated the educational experience as “good” or “excellent”– Significantly higher than comparison groups

• A high majority of students indicated they would “probably” or “definitely” return– FY (W-94.7%, M-90.6%)– Sr. (W-93.8%, M-84.6%)

• Students w/ “B- or lower” evaluated educational experience significantly lower than “B or better”

NSSE – Student Engagement--Analysis by School--

• Summarizes mean responses by school

• Compares each school mean by the aggregate of all other schools

• Compares each school mean by class level

College Student Survey - 2003

Sample

• Sample size – 267 graduating senior

• 32.2% male

• 67.8% female

• Slight oversample of females

Plans for fall 2003

• Approximately 1 in 5 seniors plan on graduate study in the fall

• Most of the remaining students plan on working either part time or full time

Academic Activities

• MC students report frequent peer interaction in connection with academic activities (e.g., studying with other students and discussing courses with other students)

• MC students were more likely to be guests in professors homes compared to other private colleges

Other Activities

• Some 38.7% report drinking beer during the last year

• 56.8% report consuming wine or alcohol during the past year

• While these are lower than comparable rates at private colleges they are higher than in 2001

• About 40% of MC students begin to use alcohol while in college

Other experiences

• A majority of students reported that they occasionally or frequently were homesick (64.7%) or depressed (63.9%)

• About one third of the students reported frequently feeling overwhelmed (38.7%)

Time Usage

• MC students report that they spend about 25 hours per week on academic pursuits (classes/studying)

• A comparison sample from private colleges report about 22 hours per week on the same activities

• In 2001 MC students reported slightly more time spent on academic pursuits (28 hours)

Interactions with Professors

• MC students report that professors frequently provided intellectual challenge (55.6%) and emotional support (36.5%)

Satisfaction with college

• Some 91% of MC students are satisfied with their overall college experience (private – 86%)

• Most MC students say they would choose the same college again (MC – 89%, private – 80%)

Objectives of importance

• The top four objectives were:– Integrating spirituality into my life– Helping others in difficulty– Raising a family– Developing a meaningful philosophy of life

Political Identity

• By their senior year, MC students are more likely to say they are “middle of the road”– First year – 32.4%– Senior year – 46.3%

• Comparing 2001 with 2003, the 2003 seniors seem to be less conservative and more moderate (even upon entering college)

Questions regarding majors

• In both 2001 and 2003 we asked twenty supplementary questions regarding experiences in the major

• In general, MC students report very high levels of satisfaction with their experiences in the major; the class of 2003 is even more positive than the class of 2001

Comparisons: 2001 vs. 2003

• In a number of areas, the 2003 seniors are more positive than those of 2001

• Given the discussion regarding alcohol, it is interesting that the use of alcohol among seniors is higher in 2003

• Less time spent on academic pursuits

• Less conservative students

Questions??