OFF-THE-SHELF OR HOMEGROWN?

SELECTING THE APPROPRIATE TYPE OF SURVEY FOR YOUR

ASSESSMENT NEEDS

Jennifer R. Keup, Director

National Resource Center for The First-Year Experience and Students in Transition

keupj@mailbox.sc.edu

Institutional data are

meaningless without a

comparison group.

My institution is unique in its programs and

goals.

The main outcome of

interest on my campus is student

development.

GOALS FOR TODAY Introduce & discuss Ory (1994) model

for comparing and contrasting local vs. commercially-developed instruments Identify elements of your institutional culture &

structure that would influence decision Discuss myths with respect to survey

administration Share examples of:

…the most prominent national surveys for first-year assessment

…software and services available to facilitate institutional assessment

WHAT DO WE MEAN

Often commercially-developed

Scope to include multiple institutions

Primarily pre-set content

Examples: CIRP NSSE EBI

Developed locally Focused on institution Content developed and

adapted by the campus/unit

Examples: Program review Utilization/satisfaction

surveys for specific programs

“Off-the-Shelf” “Homegrown”

Continuum

QUESTIONS TO ASK

What is my

budget?

What is my

timeline?

What are my

analytical capabilities

?

Who needs to see these

data?

How will this fit with my

other responsibilitie

s?

Who needs to make decisions

with these data?

ORY (1994) MODEL FOR COMPARING AND

CONTRASTING LOCAL VS.

COMMERCIALLY-DEVELOPED

INSTRUMENTS

SIX FACTOR COMPARISON

PurposeMatchLogistics*Institutional AcceptanceQualityRespondent Motivation to Return

the Instrument

PURPOSE

Allows for comparison to national norm group

Examples: Comparison to peer

or aspirant group Benchmarking Contextualize a broad

higher education issue or mandate

Allows for a thorough diagnostic coverage of local goals and interests

Examples: Satisfaction with

campus program Achievement of

departmental goals Program review

“Off-the-Shelf” “Homegrown”

Why are we doing this study, and how will the results be used?

MATCH

May provide in- complete coverage of local goals & content

Tailored to local goals and content

“Off-the-Shelf” “Homegrown”

o What are the program/institutional goals, outcomes, and areas of interest?

o Does an existing instrument meets my needs?

o Does the survey address my purpose?o Can the existing instrument be adapted

to meet my needs?

Local Questions!

INSTITUTIONAL ACCEPTANCE

Professional quality & national use may enhance acceptance

Failure to completely cover local goals and content may inhibit acceptance

Local development can encourage local ownership and acceptance

Concerns about quality may interfere with acceptance

“Off-the-Shelf” “Homegrown”

o How will the results be received by the intended audience?

o Who needs to make decisions with this data?

o What is the assessment culture? Politics

IRB

QUALITY

Tend to have better psychometrics

Professional quality may compensate for incomplete coverage of local goals and objectives

Must fully test psychometric properties

Create a professional appearance

Lack of professional quality may affect results and institutional acceptance

“Off-the-Shelf” “Homegrown”

o What is the track record of quality?o What is the psychometric soundness of

the instrument?

RESPONDENT MOTIVATION TORETURN THE INSTRUMENT

Can create instant credibility

Sometimes provide institutional or individual incentives

Local specificity may yield greater respondent “buy in”

Local instruments may not “impress” people

Can create student perception of immediate impact

“Off-the-Shelf” “Homegrown”

o What will yield the highest response rate?

Incentives

LOGISTICS (10 CONSIDERATIONS) Availability Preparation time Expertise Cost Scoring Testing time Test & question types Ease in administration Availability of norms Reporting

The Devil is in the

details!

LOGISTICS (CONTINUED)

Does a survey currently exist for our needs?

If you can afford it, the survey is available

“If you build it they (i.e., data) will come”

Takes time & resources to develop

OTS: Availability HG: Availability

OTS: Prep time HG: Prep time

Short Can take considerable time

What is the survey timeline? Is it feasible? Have you considered administration

planning?

LOGISTICS (CONTINUED)

Fully-developed protocol allows one to administer after reading manual

Takes content, measurement, and administrative experience

Psychometrics!!!

OTS: Expertise HG: Expertise

OTS: Scoring HG: Scoring

Can be delayed if scoring off campus

Need to adhere to the administration cycle

Can be immediate

Related to expertise

LOGISTICS (CONTINUED)

Fixed based upon content and administration protocol

Flexible as long as the survey meets institutional & programmatic needs

OTS: Testing time HG: Testing time

OTS: Test type HG: Test type

Type of test and questions are predetermined

Allows for flexibility in type of test (objective /open-ended) and type of question (MC, rank ordering, etc.)

If administering in class do you

have faculty buy-in?

LOGISTICS (CONTINUED)

Requires standardized administration

Special training for testers

Allows for greater flexibility

OTS: Ease of Admn HG: Ease of Admn

OTS: Norms HG: Norms

National & inter-institutional comparison

IRB

Intra-institutional comparison

OTS: Reporting HG: Reporting

Standard formats that don’t always relate to institution

Institutional tailoring of results and reporting

LOGISTICS (CONTINUED)

Primary costs associated with purchase price

Other costs: Scoring Data Specialized reporting Human resources to

coordinate campus administration

Recurring cost

Primary costs associated with development costs Instrument development Ensuring psychometric

properties Scoring & recording data Reporting findings

Other costs Software/hardware Training

Primarily one-time investment

OTS: Cost HG: Cost

Purpose

MatchLogistics

Accept-anceRespons

e

Quality

O-T-S VS. HG MYTHS You can only gather comparison data

from national (OTS) surveys It is cheaper to develop and administer

a homegrown survey Off-the-shelf surveys don’t require any

work Homegrown surveys are hard. You don’t need IRB approval for local

assessment Off-the-shelf surveys study all the

important topics

FYE ASSESSMENT EXAMPLES

CIRP Freshman Survey

Your First College

Year (YFCY) Survey

NSSE Educational

Benchmarking Incorporated

Services Eduventures

Student Voice

Software Zoomerang

Survey Monkey

“Off-the-Shelf” “Homegrown”

CONTINUUM OF ASSESSMENT

CIRPNSSEEBI

Eduventures Student

Voice

Survey MonkeyZoomeran

g

Off-the-Shelf

Home Grown