Using the National Survey of Student Engagementto Assess Educational Effectiveness at AICAD

schools

AICAD Consortium Meeting

Pratt Institute, NYJune 12, 2007

Agenda

• Introduction & NSSE overview

• What can you learn about your students and their experience from NSSE?

• NSSE Reports

• Benchmarking

• Consortium comparison

• AICAD interests: marketing and institutional improvement

• NSSE details

• Timeline and Administration

• Questions – What questions do you have right now?

Introduction Activity:

Assessment at your institution:

What do you want to know about your students?

Why do you want to know this?What is the purpose of your assessment initiative(s)?

To what extent have you used NSSE data?

What is NSSE?

Student Engagement The time and energy students devote to

educationally purposeful activities and the extent to which institutions emphasize effective practice

Engagement is a reliable predictor of student learning and personal development

Institution can shape curriculum and resources for learning to promote engagement

What Really Matters in College: Student Engagement

Because individual effort and involvement are the critical determinants of impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement. Pascarella & Terenzini, Pascarella & Terenzini, How College How College Affects StudentsAffects Students, 2005, p. 602, 2005, p. 602

Foundations of Student Engagement

Time on task Time on task (Tyler, 1930s)(Tyler, 1930s)

Quality of effort Quality of effort (Pace, 1960-70s)(Pace, 1960-70s)

Student involvement Student involvement (Astin, 1984)(Astin, 1984)

Social, academic integration Social, academic integration (Tinto,1987, 1993)(Tinto,1987, 1993)

Good practices in undergraduate Good practices in undergraduate education education (Chickering & Gamson, (Chickering & Gamson, 1987)1987)

Outcomes Outcomes (Pascarella, 1985)(Pascarella, 1985)

Student engagement Student engagement (Kuh, 1991, 2005)(Kuh, 1991, 2005)

Good Practices in Undergraduate Education

(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)

Student-faculty contactStudent-faculty contact

Active learningActive learning

Prompt feedbackPrompt feedback

Time on taskTime on task

High expectationsHigh expectations

Respect for diverse learning stylesRespect for diverse learning styles

Cooperation among studentsCooperation among students

NSSE Survey & Results

Survey offers an annual snapshot of student participation in programs and activities that institutions provide for their learning and personal development.

Results provide an estimate of how undergraduates spend their time and what they gain from attending college.

NSSE items represent empirically confirmed ‘good practices’; they reflect behaviors associated with desired outcomes of college.

NSSE 2006 Participating Colleges & Universities by Carnegie Classification

0%

5%

10%

15%

20%

25%

30%

DRU-VH DRU-H DRU Master-L Master-M

Master-S Bac-AS Bac-Div

National

NSSE 2006

Core Survey: NSSE

Research based on effective educational practices

Designed and tested for high validity and reliability

Relatively stable over time

High credibility of self-reported data

Over 275,000 students at 600 institutions annually

NSSE Survey Item Organization

Q.1 – Academic activities

Q.2 – Learning mental activities

Q.3 – Reading & writing

Q.4 – Homework

Q.5 – Academic challenge

Q.6 – Co-curricular activities

Q.7 – Enriching educational experiences

Q.8 – Campus relationship

Q.9 – Time usage

Q.10 – Institutional emphasis

Q. 11 – Gains

Q.12-14 – Satisfaction

NSSE Results

Are diagnostic; to help institutions look holistically at undergraduate experience

Help pinpoint aspects not in line with mission, or what institution expects

Identify weaknesses and strengths in educational program

Help institutions know what to focus on to improve student learning and success

Questions to answer with NSSE results

How many hours per week do first-year students spend studying? Do women study more than men?

What % of seniors work with faculty members on activities other than coursework (activities, committees)? Does this differ by major?

What % of FY and SR spend 0 hours in co-curricular involvements? Is this more than at peer institutions?

Do FY students work more frequently with classmates on assignments outside of class than their counterparts at peer institutions?

Questions to answer with NSSE results

Do NSSE results match our mission and what we say about a [INSTITUTION] experience?

Are we meeting our own expectations for having a supportive campus environment?

Since implementing a new multicultural education initiative and expanding diversity programming, has our score on the diversity scale changed?

Are FY who withdraw from the institution different in terms of engagement than students who are retained?

How are we performing compared to our select peers (normative benchmarking) or to our institutionally identified standards (criterion benchmarking)?

NSSE Deliverables

Institutional Report (August) Comparison Reports

Respondent characteristics (Demographic Information)

Means and Frequencies (item averages and response percentages)

Benchmarks of Effective Educational Practice

Additional Reports (If Applicable) FSSE Report BCSSE Combined Report

Data file (student-identified)

NSSE Institute InformationUsing NSSE Data Accreditation ToolkitData Facilitator’s Guide

Sample NSSE results: Frequency comparisons

Frequency Comparisons: About how many hours do you spend in a typical 7-day week participating in co-curricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc.)

(1=0 hrs/wk, 2=1-5 hrs/wk, 3=6-10 hrs/wk, 4=11-15 hrs/wk, 5=16-20 hrs/wk, 6=21-25 hrs/wk, 7=26-30 hrs/wk, 8=more than 30 hrs/wk

0 Hours on co-curricular activities = 61% FY vs. 56% seniors compared to 43% and 46% at Select Peer Institutions – is this what NSSEville expects??

COCURR01

0 hr/wk 190 61% 2539 43% 3,440 52% # 43% 170 56% 2558 46% 3,341 52% 50,704 47%

NSSEville State Selected Peers Carnegie Peers NSSE 2006NSSEville State Selected Peers Carnegie Peers NSSE 2006

Sample NSSE results: Mean comparisons

Bench-mark Class Mean a Mean a Sig b

Effect

Size c Mean a Sig b

Effect

Size c Mean a Sig b

Effect

Size c

1. Academic and Intellectual Experiences

FY 2.11 2.34 *** -.27 2.33 *** -.24 2.36 *** -.28

SR 2.52 2.70 ** -.20 2.65 * -.15 2.75 *** -.26

FY 1.47 1.55 1.59 ** -.14 1.56

SR 1.83 1.83 1.85 1.81

s.

Worked with faculty members on activities other than coursework (committees, orientation, student life activities, etc.)

SFI

In your experience at your institution during the current school year, about how often have you done each of the following? 1=never, 2=sometimes, 3=often, 4=very often

h.Worked with classmates outside of class to prepare class assignments

ACL

NSSEville State

NSSEville State compared with:

Selected Peers Carnegie Peers NSSE 2006

NSSEville State score on 1h. (working with peers outside of class) is significantly LESS than SELECT PEER institutions for FY and Seniors

Benchmark Report

Level of Level of Academic Academic ChallengeChallenge

Active & Active & Collaborative Collaborative

LearningLearning

EnrichingEnrichingEducational Educational ExperiencesExperiences

SupportiveSupportiveCampusCampus

EnvironmentEnvironment

StudentStudentFaculty Faculty

InteractionInteraction

Sample NSSE Results: Benchmark Report

First-Year

60.957.8 59.4 59.1

0

25

50

75

100

NSSEville State Selected Peers Carnegie Peers NSSE 2006

Class Mean a

Sig bEffect

Size c Mean a

Sig bEffect

Size c Mean a

Sig bEffect

Size c

First-Year 57.8 ** .17 59.4 59.1Senior 55.8 ** .21 59.4 56.6 * .16

Carnegie Peers

59.7

Mean a

NSSEville State

Supportive Campus Environment (SCE)

Selected Peers

Benchmark Comparisons

NSSE 2006

60.9

NSSEville State compared with:

First Year Students

NSSEville State Strength – significantly higher score

for FY and SR on Supportive Campus Environment

AICAD Consortium Questions

•Value in developing consortium specific questions

•Comparison options

•Potential Data sharing

•Establish Core questions & others that rotate in

NSSE Use

“The NSSE data is one among several pieces of information that is used to organize discussions about enrollment management, curricula, retention, and faculty development.”

—Christopher Cyphers, Provost, School of Visual Arts

Using NSSE Data

Context Setting – paint a picture of the institution

Evidence of outcomes & processes

Refocus conversation about collegiate quality

Provides lexicon for talking about collegiate quality in an understandable, meaningful way

Benchmarking – longitudinal, criterion, normative

Problem Identification- results point to things institutions can do something about – almost immediately

Mobilize Action - to change/improve

Helps inform decision-making

Making Sense of Data: Benchmarking

Three Approaches:

Normative - compares your students’ responses to those of students at other colleges and universities.

Criterion - compares your school’s performance against a predetermined value or level appropriate for your students, given your institutional mission, size, curricular offerings, funding, etc.

Longitudinal – year to year comparison of your students to assess improvement

Benchmarking within AICAD consortium - Writing

Assessment Issue: Insure high quality writing experiences in the first year. Are we using writing center/tutors effectively?

Relevant NSSE items: 1c, d; 3c,d,e; 11c. Provide student learning process & outcome indicators

NSSE results: First year students write short papers comparable to AICAD schools; but fewer med & long papers, are less likely to prepare 2+ drafts & also report lower gain in writing than AICAD peer schools.

Interpretation: Benchmarking with AICAD schools indicates institution is underperforming; what other data might you gather to assess writing? What first year writing initiatives might help? What goals might you set for improvement?

Example: 1

Benchmarking – longitudinal (performance indicators & co-curricular improvements)

Assessment Issue: Maintaining effectiveness and making targeted improvements with upper division students

Relevant NSSE items: NSSE items identified as key performance indicators, gains items for seniors (11 a-p); and targeted improvements in co-curricular experiences (1h,s,t; 6a; 7b, 10f & diversity scale, 1e,u,v; 10c)

NSSE results: Baseline NSSE = 2006, monitor indicators in 2008; assess impact of co-curricular enhancements & diversity initiatives started in 2006 by comparing NSSE 2006 SR to 2008 SR scores.

Interpretation: Longitudinal benchmarking (2006-2008); could also benchmark with AICAD schools. Did you meet performance goals? Did the enhancements have an impact?

Example: 2

Multi year comparison

Using NSSE to “market” AICAD schools

Demonstrate AICAD consortium and institutional strengths (items, NSSE benchmarks) in undergrad program

Use results to show mission effectiveness i.e., gains items (11 a-p) & comparison

peers show liberal education gains; use consortium results to focus on arts school mission

Provide results to prospective students and families

Share results with current students, development office and alumni

Institutional Example: NSSE and Enrollment Management

The enrollment management area at Meredith has used NSSE results to help guide the enrollment marketing strategies. They look closely at trends and make adjustments to programs and campus visitation days to ensure that students are more cognizant of student involvement and engagement opportunities.

An academic dean reports using NSSE information when speaking to parents at an admissions event. "Parents seemed impressed that there was data to support the points that I was making about what we say about the student/faculty relationships and educational opportunities at Meredith."

Institutional Example: Hanover College

A detailed summary of NSSE is sent to the faculty as well as the Admission and Student Life staffs to ensure the results, both good and bad, are understood by key folks on campus. Last year, Admission requested an additional presentation and discussion of findings to help them better understand the strengths of the Hanover experience and how that impacts student fit.

Helping Students and Families Focus on What Matters to Success

Pocket Guide helps prospective students ask the “right” questions

Good questions to ask of all schools, not just those that participate in NSSE

School counselors can request up to 1000 free pocket guides per year. Colleges and non-profit education organizations can request up to 300 copies free per year.* *Request via the NSSE Web site

“A Pocket Guide to Choosing a College:Are You Asking the Right Questions…”

Connecting NSSE Data to Accreditation Standards - Example

Accreditation standard: “Demonstrate effectiveness of student academic and social support services”

Evidence for institutional self study:

Information about availability and student use of tutoring, writing support, peer study groups, counseling services

NSSE indicates FY & SR believe institution emphasizes spending time studying and support for student success; 79% seniors tutored or taught peers; positive correlation between peer collaboration outside of class, satisfaction and first-year retention

Positive student satisfaction data about support services

Area for improvement - seniors indicate low gains in writing and completing drafts of papers; institution responds with examination of writing requirement in senior capstone and targets seniors for increased use of writing center

NSSE and AICAD consortium

Consider data sharing agreements Potential for additional comparison studies;

prepare papers/presentations; examine shared concerns (retention, outcomes)

Use consortium to explore common concerns Coordinate survey schedule

Ideas to improve participation rate (incentives, persuasive to promote that survey is occurring at other AICAD schools??)

Identify focus for additional questions (up to 20!) Develop stable core of questions? or change focus?

Or a mix? Rotate new questions in?

Administration Details

What challenges have you faced in your NSSE administrations?

What concerns do you have about your next administration?

Questions about the details?

NSSE Timelines (15 mos.)

May NSSE/FSSE registration opens

September NSSE/FSSE registration deadline

NSSE materials due two weeks after registration confirmation

October NSSE pop. files, oversample, and

consortium decisions due

December FSSE materials and pop. files due

Mid-January early February NSSE administrations open

BCSSE registration begins

Mid-March early April FSSE administration opens

June NSSE & FSSE administrations close

BCSSE administration begins at many campuses

August Institutional Reports sent, including

raw data and printed reports for NSSE, FSSE, and the prior summer’s BCSSE

BCSSE administration continues

September BCSSE data and reports sent to

participating institutions

NSSE Administration

Administration Mode

Paper: We need accurate mailing addresses, letterhead, signatures

Web+: 4x the paper sample, we need e-mail and mailing addresses

Web: 5x the paper sample, we need e-mail addresses

NSSE Administration

Sample Size

Numbers are based on mode and school size

Oversampling can increase sample size or ensure adequate representation of populations of interest

NSSE Administration

Things that we need from you

Contact persons Campus Project Manager (required) Campus Administrative Contact (required) Auxiliary Contact (optional)

Population File All First-Year and Senior Students Accurate mailing and/or e-mail addresses

Institutional letterhead and signature file (Paper mode only)

NSSE Administration

Things for you to consider

Broad buy-in from others at your institution (informal word-of mouth)

Web-mode institutions: Good partnership with IT department

Consortium

NSSE Administration

Things you to consider (cont.)

Administration Plan • Follow the IRB rules of Indiana University

Bloomington• Allowed up to 5 institutional contacts• Promotion plan• Incentive programs• Tips to boost response rates http://

nsse.iub.edu/html/tips.cfm

NSSE: Only one step in assessment process

Step #1: Survey Data

• Survey students• Review results• Develop preliminary

list of strengths and opportunities for improvement

Step #2: Feedback

• Share results with faculty, administrators & students

• Identify themes & priorities

• Design action plan

Step #2: Feedback

• Share results with faculty, administrators & students

• Identify themes & priorities

• Design action planStep #3: Action Plan

• Finalize plan• Share plan with

appropriate groups• Link to strategic plan• Implement action

Step #4: Follow-up

• Use results as benchmarks to monitor progress

• Faculty & student focus groups

Step #4: Follow-up

• Use results as benchmarks to monitor progress

• Faculty & student focus groups

NSSE in your assessment plan

How often should I administer NSSE?

Every Year: Gives you a snapshot of each class

Every Three Years: Gives you a picture of a cohort at the beginning and the end of their college experiences

Every Five Years: Works well with most accreditation cycles (Accreditation and Interim Reports)

Other factors to consider Establishing a baseline Costs (using all core surveys) Additional Surveys/Sources of Data Time to take absorb results, make changes

Updates for 2007 and 2008

No changes to survey content

Select up to three customized comparison groups on your reports

Electronic report delivery

Executive Summary Report

Pocket Guide Report

45

NSSE results showed FY students were less engaged than seniors

New FY interdisciplinary, inquiry-based seminars; better integration of disciplines; engaging introductory courses

Associate Dean appointed to Office for the First Year

Assessment plan in development with NSSE indicators as key component

Institutional Examples: Worcester Polytechnic Worcester Polytechnic InstituteInstitute

Institutional Example: NSSE & Assessing General Education goals

Used NSSE items in 11a-p to assess Used NSSE items in 11a-p to assess institutional impact on college-level institutional impact on college-level competencies competencies (a.k.a., indirect measures of (a.k.a., indirect measures of student learning outcomes) student learning outcomes)

Undergraduate Undergraduate seniors seniors 2005 NSSE 2005 NSSE results confirmed findings from 2004results confirmed findings from 2004

Most seniors (75%+) reported that Most seniors (75%+) reported that KSU experience had “substantial KSU experience had “substantial impact” (VM+QAB) in 9 or 16 college-impact” (VM+QAB) in 9 or 16 college-level competencieslevel competencies

KSU rank ordered competencies, KSU rank ordered competencies, showing connection to mission, and showing connection to mission, and compared to other master’s instit compared to other master’s instit where KSU was sig. higher, where KSU was sig. higher, comparable, sig. lower on comparable, sig. lower on competenciescompetencies

Institutional Example: Program Development and Strategic Planning

NSSE results framed a “Sophomore Experience”

2005 = Pace’s 5th year of participation

Concern regarding SP- JR persistence; FY results offers context for understanding exp. as students enter SP year

Established “SP Experience Working Group” to investigate if FY exp. carried over in SP year. Focused on low NSSE score items, conducted focus groups, created sophomore survey. Led to pilot of “Pace Plan” (mentoring), includes Career Exploration Course, Sophomore Kick-Off Day

NSSE also used in strategic indicators, Accred, NCATE, AACSB, Faculty Development/Colloquia, items used by offices (Technology, Multicultural Affairs), studies performed by Enrollment Mngmt.

NSSE suite

The “NSSElings”

The Faculty Survey of Student Engagement (2003)

The Beginning College Survey of Student Engagement (2004)

Additional Surveys

The Law Student Survey of Student Engagement

The College Student Experiences Questionnaire

The College Student Expectations Questionnaire

The High School Survey of Student Engagement*

The Community College Survey of Student Engagement*

*Not administered by the Center for Postsecondary Research

FSSE

Faculty perceptions of how often their students engage in different activities

Importance faculty place on various areas of learning and development

Nature and frequency of interactions faculty have with students

How faculty members organize class time

FSSE Instrument: Survey Options

Course-based (default)

Responds to questions based on one particular undergraduate course section during taught during the current academic year

Typical-student

Responds to questions based on the typical first-year student or senior taught during the current academic year

Beginning College Survey of Student Engagement – BCSSE

Designed for entering first-year students as a companion to NSSE

Measures:pre-college academic and

co-curricular experiencesexpectations for

educationally purposeful activities during college

New!New!

BCSSE Instrument

Three pilots: ‘04, ‘05, and ‘06

2004 pilot with 28 schools, 15,890 students

2005-2006 pilots with 80 institutions, 39,986 students

Study effect of students’ background on NSSE scores

Use to examine gap between expectations and engagement

Registration now open

Creative Campus initiative and Experimental questions developed by Arts Consortium in NSSE 2007

1 )In your experience at your institution during the current school year, about how often have you done each of the following? (Very Often, Often, Sometimes, Never)

a) Attended an art exhibit, play, dance, music, theater, or other performance

b) Talked about an art exhibit, play, dance, music, theater or other performance with other students, friends, or family

c) Participated as an artist, performer, or crew member in an art exhibit, play, dance, music, theater, or other performance

d) Used your experiences and interest in the visual and performing arts in class discussions or assignments

e) Explored a new subject area as a result of your attendance or participation in the arts.

2) To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the following areas? (Very much, Quite a bit, Some, Very little)

a) Developing an understanding and enjoyment of an art exhibit, play, dance, music, theater, or other performance

b) Developing a commitment to be involved in the arts (attendance or participation)

3) In a typical 7-day week, about how many hours do you spend in arts experiences on or off campus? [0, 1-5, 6-10, 11-15, 16-20, 21-25, 26-30, more than 30]

4) How many arts courses (art, music, theater, dance) have you taken since coming to college? [0, 1, 2, 3, 4 or more]

5) What motivates you to attend arts events (art, music, theatre, dance)? (Select all that apply)

a) Class requirement b) Encouragement from friends c) Enthusiastic professor d) Personal interest e) Proximity f) Quality of the facility g) Opportunity to meet people h) To be involved on campus i) Promotion of the event on

campus j) Quality of the event

6) What keeps you from attending arts events (art, music, theatre, dance)? (Select all that apply):

a) Expense b) Parking availability c) Lack of awareness of events d) Difficulty getting tickets e) Coursework demands f) Facility is rundown g) Location is not convenient h) Job demands i) No one to go with j) Extracurricular activities/meetings k) Lack of quality events l) Limited interest in arts events7) Select the three (3) places you perceive to be

the most active spaces on campus a) sports venues (football stadium, basketball

arena) b) public plaza (main campus quadrangle or

green) c) arts venues (performing arts center,

museums/galleries, rehearsal halls, art studios)

d) student union e) dining hall f) residence hall g) library h) coffee shop or restaurant i) campus main street j) recreation area (fitness center, intramural

fields)

8) Where have you attended arts events (art, music, theatre, dance) while attending this institution? Select all that apply.

a) Theater buildings (performing arts center, recital hall)

b) Museums/galleries c) Coffee shop or restaurant d) Large concert venues (stadiums, arenas, band shell,

etc) e) Random places, such as street performances f)\ Residence halls g) Arts festivals h) Off-campus; downtown or in the local community9) To what extent have arts experiences at this institution

contributed to your abilities in the following areas? [very much, quite a bit, some, very little]

a) Thinking critically and analytically b) Thinking imaginatively or creatively c) Understanding fundamental concepts in my major d) Communicating clearly and effectively e) Working effectively with others f) Learning effectively on your own g) Taking intellectual risks10) To what extent do you agree with the following

statements? [strongly agree, agree, disagree, strongly disagree ]

a) The arts are visible on my campus b) My institution encourages students to study the arts c) The arts curriculum at my institution is open to

everyone d) My institution encourages students to participate in

arts events e) The university community values the arts f) The arts contribute to the vitality of this campus g) The presence of the arts enhances my collegiate

experience

Creative Campus initiative and Experimental questions developed by Arts Consortium in NSSE 2007

Discussion and Questions

Jillian KinzieAssociate Director, NSSE

Institute

Web site: www.nsse.iub.eduE-mail: nsse@indiana.edu

Indiana University Center for Postsecondary Research1900 East 10th StreetEigenmann Hall, Suite 419Bloomington, IN 47406-7512 Ph: 812-856-5824 Fax: 812-856-5150