Jen SweetOffi ce for Teaching, Learning & Assessment

DePaul University

Shannon MilliganFaculty Center for Ignatian Pedagogy

Loyola University Chicago

ADVANCED SURVEY DESIGN

Tuesday, December 8, 20152:00 – 3:30pm

Workshop Outline

• Introductions• Conflicting Recommendations for Survey Design?• Cognition and Survey Design• Recommendations to Reduce Cognitive LoadACTIVITY!• Affective Assessment and SurveysACTIVITY!• Survey Design & Distribution Tools• Analyses of your Survey Instrument

Workshop Outcomes

By the end of this workshop, participants will be able to:

• Apply knowledge of the cognitive processes students use to respond to surveys to design effective survey items and instruments.

• Identify non-cognitive variables that can be assessed with surveys

• Use the tools available to them at their respective institutions to design and distribute surveys.

• Identify a variety of methods available for the analysis of survey data.

Conflicting Recommendation

s for Survey Design?

Recommendations for Survey Design that seem to Conflict

Examples:Neutral Point • Always include? • Never use? • Sometimes yes; sometimes no?

Number of Scale Points to Include• 2?• 3?• 4?• 5?• 6?• 7?• 9?

• The more the better?

All of This is in the Literature!

So, What’s Up with the Literature?All of these recommendations may be appropriate depending on the specific context:

• respondent attributes • nature of the items in the survey • length of the survey• Etc.

Generally looking at things like:• Reliability• *Validity • Survey Outcomes • Response Rate (high)

• Use of Response Sets (low)

Survey Design is as Much Art as Science!“There is always a well-known solution to every human problem - neat, plausible, and wrong.” H.L. Hencken

Cognition and Survey Design

Cognitive LoadPaas and Van Merrienboer, 1994

The amount of cognitive effort (or thinking) students need to exert to respond to a survey item.

If cognitive load exceeds the student’s working memory capacity, they will take some sort of shortcut (Paas & Van Merrienboer, 1994), or satisfice (Krosnick, 1991)

• Read questions less carefully (skim)

• Use a response set

• Give same response for all questions, regardless of content

• Overuse neutral or N/A response option

• Skip the question (provide no response)

• Respond randomly

• Decide not to complete the survey

Cognitive to Respond to Surveys (Tourangeau, 1984)

1.Interpretation

2.Retrieval

3.Judgment

4.Response

Recommendations to Reduce

Cognitive Load

Step 1: Interpretation•Use language that is clear and familiar to survey respondents. • Avoid cognitively taxing wording.• Avoid unfamiliar words and phrasing.• Avoid jargon and acronyms

• Ensure that question stems are clear and explicit.• Do not use concepts that are unclear or unfamiliar to respondents.• Avoid complex sentence structures.• Ask about only one concept in each stem; avoid double-barreled

questions

• Use questions that do not make assumptions.• Ask for information in a direct manner by avoiding double negatives • Ensure question stems are succinct, including only as much information as is necessary for respondents to properly interpret what is being requested of them.

Interpretation (Continued)• Include clear instructions that clarify the purpose of the survey instrument, and provide respondents with expected procedures for responding to the survey instrument.

• Ensure that every portion of a survey instrument is visible without the need for additional action by the respondent. • Use radio buttons instead of drop-down boxes to display response

options.• Do not “hide” definitions respondents may need to interpret and

respond to survey items.• Use easy-to-read font size and type.• Use high-contrast font and background colors.

Step 2: Retrieval•Use stems that request information with which respondents have primary experience and avoid asking for second-hand information (i.e., information that the respondent has heard about, but not experienced personally) or hypothetical information.•Group conceptually similar items together.

Step 3: Judgment•Use the smallest number of response options necessary to encompass all meaningful divisions of what you are asking about.•General Guideline: four or five response options, depending on whether or not there will be a neutral option.• Include a neutral option if you reasonably expect participants to have no opinion, but otherwise, they should be avoided •Neutral responses can be difficult to interpret•Offering a neutral option may encourage satisficing

Step 4: Response•Use the smallest number of response options necessary to encompass all meaningful divisions of what you are asking about.•Label the scale options.•May only need to label the most extreme options• Include a neutral option if you reasonably expect participants to have no opinion, but otherwise, they should be avoided •Neutral responses can be difficult to interpret•Offering a neutral option may encourage satisficing

Activity!

Practice Evaluating Survey Items!

Individually:Complete the worksheet

In Groups:Compare ResponsesDid Everyone Identify the same items for Improvement?Are there differences in ways you edited items?

Surveys and the Rise of Affective

Assessment

Growing Emphasis on Non-Cognitive Abilities• Grit• Growth• Social-emotional development• Self-awareness/management/efficacy• General affect• Engagement• Mattering• Climate

• Research shows strong relationships between these variables and overall success

From NPR: “Nonacademic Skills Are Key to Success. But What Should We Call Them?” (May 28, 2015)

The Role of Surveys From NPR: “To Measure What Tests Can’t, Some Schools Turn to Surveys” (December 2, 2015)

From that article: “A growing battery of school leaders, researchers and policymakers think surveys are the best tool available right now to measure important social and emotional goals for schools and students”

Why?• Easy to administer• Easier to collect and analyze than reflection papers (and the like)• Many surveys/survey questions already exist• Faster data sharing = faster decision-making/implementation

(maybe)

Activity!

Non-Cognitive Assessment and You

Individually:•What non-cognitive variables might be of interest to your program?•Are you already assessing any of these variables?

In Groups:•Share what you are assessing and what you might be interested in assessing•What variables might be important to look at on an institutional-level?

Survey Design/Distributio

n Tools

Three Main Tools Used at DePaul

1. Qualtrics

2. Google Forms

3. Survey Monkey

QualtricsAdvantages:•Free to DePaul faculty and staff•Supported by Information Services•Very Flexible and Comprehensive System•Lots of Features

Disadvantage:•Reporting Features aren’t Great•Steeper Learning Curve than Other Systems

Google FormsAdvantages:•Free to Everyone•Data is Collected in Excel Format•Easier to Learn than Qualtrics

Disadvantage:•Not Nearly as many Features as Qualtrics

Survey MonkeyAdvantages:•?Disadvantage:•Most Advanced Features Cost $•To get all the features available in Qualtrics, cost is $780/year•Limited to 10 questions and 100 responses on the free version

Overview of Survey Analysis

Survey Analysis• For when you want to:• Group survey respondents• Group survey items• Make predictions/observe relationships• Analyze “fit” of respondents and items

•Which to use based on:1. Purpose

2. Audience

http://www.edmeasurement.net/5244/SPSS%20survey%20data.pdf

Basic Survey Analysis• Sometimes all you need are:

1. Means

2. Correlations

3. Frequency tables/graphs

Survey Analysis-Group Respondents

• Cluster Analysis• What it does: Creates groups or “clusters” of respondents

based on similar responses to a set of survey questions• Kind of intuitive because: Clustering is part of organizing (e.g.

organization of produce section, medical symptoms)• What it answers: What do members of each cluster have in

common?• Ex. First-generation students tend to have less familiarity with

research opportunities

• Useful for: Marketing, outreach, cohort creation

Survey Analysis-Group Survey Items

• Factor Analysis • What it does: Groups statistically related survey items into a

number of “factors”• Kind of intuitive because : Groupings can be done based on

preconceived ideas or based on the data analysis. Also, largely correlation-based• What it answers: Can items be removed from the survey? Do

the factors relate to constructs that make sense (e.g. identified learning outcomes)?• Useful for: Survey refinement, naming constructs

Example: Do our questions aimed at measuring a certain outcome actually seem to measure that outcome?

Survey Analysis-Make Predictions• Regression • What it does: Determines the relationship between multiple

predictor variables and a dependent variable• Kind of intuitive because : Similar to correlation (determining the

relationship between 2 variables), but with the ability to control for other variables• What it answers: What item(s) are significant predictors of an

outcome? Is there a positive or negative relationship between the predictor variable and outcome variable? How well do our variables explain the observed outcome?• Useful for: Looking at relationships between responses to certain

questions (or factors) and outcomes. Determination of resource allocation. Possible survey refinement

Example: Do students who report greater library use have a higher GPA?

Survey Analysis-Analyze “Fit” of Respondents and Items

• Rasch Analysis and Item Response Theory (IRT)• What it does: Provides information about “ability” of

respondents and “difficulty” of survey items• But they differ in: Rasch only considers respondent ability and

item difficulty. IRT can also account for guessing and greater differences between high/low ability respondents• What it answers: What item(s) are too easy or too difficult? Is

the survey measuring a single variable? Are there trends in responses between respondent groups?• Useful for: Survey refinement, detecting bias in survey items,

determining number of levels needed in a rating scale

Example: Is a survey item written in such a way that it is interpreted differently across student groups?

Contact InformationJen SweetDePaul UniversityOffice for Teaching, Learning & Assessmentjsweet2@depaul.edu

Shannon MilliganLoyola University ChicagoFaculty Center for Ignatian Pedagogysmilligan@luc.edu