Surveys

How do you feel about surveys?

Annoying?Intrusive?Frustrating?Ambiguous?Boring?

Fun?Exciting?Great use of my

time?Too short?Clear?How do you feel about lectures on surveys?

SurveysA survey is an instrument that collects data

that takes the form ofa questionnaire or an interview

Surveys can beCross-sectional – a moment in time, the

“snapshot”Longitudinal – perceptions over time

Trends – follow a topic over time Cohorts /panels – follow a group or small sample

over time

SamplingSurveys are either “sample” or “census”We cant always reach the whole population

– but may instead look at a sample of that population

If we want to generalize to the population we will need to randomly select the sample, or systematically select a sample that looks just

like the population

Sampling ExampleFor example, let’s say we want to know

what special educators think are the most important factors keeping them on the job. What do you think they will say?

Sampling ExampleFor example, let’s say we want to know what special

educators think are the most important factors keeping them on the job. What do you think they will say?We probably cant survey ALL special educators (the

census)We can randomly select teachers from all over the country…We can randomly select teachers from randomly selected

districts…We can stratify the sample and then randomly select (small,

medium, large, for example)Another alternative is to never generalize. If we select a

sample from our district – it is simply the opinion of a few.If we don’t account from sample problems, we introduce

bias

Retention Factors

Made up Example of BiasWe want to know how our parents feel about year

round schooling – there are 200 parents at our school – so we try to survey them all.

We don’t know it, but about 20% (40) of the population is vehemently in favour of it – and will stop at nothing to voice their opinion

The other 80% (160) are fairly ambivalent, but would probably prefer things the way they are now

We send out surveys. We get a 30% (60) return rateResults indicate 66% of people surveyed would

prefer year round schooling, so we make the change. BUT – all of the 40 replied, and only 20 of the 160 replied.

Real Example of Bias34 masters students are sending their written

comprehensive exam by Monday at noon.One student sends it 24 hrs late. The comps coordinator sends an email out to

faculty asking if we should accept the paper.4 reply – absolutely no.7 do not reply.

A later lunch conversation revealed that the 7 didn’t vote because they are newer faculty – and hesitate taking a stand. The majority of them would have accepted the paper.

The student “failed out” of the program

Overview: Steps to Create a Survey1. Define the purpose/delineate broad issues

you will address2. Choose a format and create a survey plan3. Construct the questions4. Pilot test the questions5. Administer the survey6. …Analyze the data

Step 1. Define the Purpose and PopulationThe purpose of your survey comes directly from the

purpose of your program evaluation – it is simply a component of the larger evaluation

Be sure to isolate specific topics you want to address with the surveyFor example – do parents read to kids at home? This

topic area might turn into 6 questions on a survey.The population will be made up of those who have

the information you need. There may be multiple perspectives, so your population may not be homogenousFor example – parents, students, teachers, and admin

all have something to say about year round school

Step 2. Choose a Format/Make a PlanQuestionnaire can be Web, Email, phone,

or in person (individual or focus group)Each have advantages and disadvantages

What are the good and bad? (1 pt per box where you hit)Method

The Good The Bad

Mail

Email

Phone

In-person

Format ComparisonMethod

The Good The Bad

Mail InexpensiveCan be confidentialStandardized items/procedures

Response rate lowCannot probe furtherOnly reading respondentsCan be slow to get responses

Email Same as above, plusFastEasy to give to a lot of people

Same as above, plusNeed emailPossibility of ballot box stuffing

Phone High response rateQuick data collectionCan reach remote areas

Need phone numbersComprehensive admin training neededPeople don’t like it!

In-person

Ability to probe deeplyHigh return rateCan be recorded for laterFlexible format possible

People need to be closeTime-consumingNo-anonymityPossible interviewer biasComprehensive admin training needed

Advantages of web surveysLow costs - only software, no printing, envelopes

or postage needed.No data entry costs.Minimal data entry errors.Easy to correct problems during survey

administration.Quicker and cheaper than other methodsResponse rates are good, maybe somewhere

around 40 - 60%, depending on the topic and population.

Most are familiar with the process (you are now!)

Step 2. Choose a Format/Make a PlanNo matter your delivery mode – you will need

to prepare a cover letter. This explains the survey, and announces the upcoming delivery. It lets respondents know how important their input is.

On a web survey – your cover letter is an email, and repeated on the first page of the survey

Make a plan about when the cover letter goes out, when the survey goes out, and how you will follow-up with non-responders (reminders)

Example Web Survey Response Rate with Follow-ups

Step 3. Construct QuestionsStructured/closed (mc), semi-structured /open ended

What are the advantages/disadvantages of each?Do you require demographics?Design Issues

Keep it simple: avoid complicated designs with lots of colors.

Make the survey one (screen) page in length, unless categories makes sense in which case group items by concept

For drop down boxes, make sure first response category is not the first option (“select” usually works)

Keep the survey as short as possible. Nobody likes a long survey.

Step 4. Pilot testYou get an awful feeling after you send out a

survey, when people let you know they didn’t understand questionsIt is hard to pilot test all the itemsIt is harder to un-administer a bad survey

The pilot sample should not just be your friendsA variety of people that might loosely look like

your sampleIt should also include your friends, because they

will take time and be honest (usually)It should not be your partner – because they

really wont care and will lie to you

Step 5. Administer the SurveyThe “tailored design” approach (Dillman

(2000))Different levels of response depending what

happensPre-notice (cover), survey, reminder, follow-up

with survey again for non-responders, and so on. For web surveys it is simply multiple emails

With multiple emails you need to take out respondents from your email list. Follow up emails can thus be tailored to non-responders

If you do only one survey mailing, you WILL get a low response rate.

User TrackingYou have two choices:1.Ask for name and other identifying information

Multiple responses can be eliminated.Responses can be linked to existing dataNo evidence this affects response ratesImportant to state that only aggregate data will be released.

2.Allow anonymous responses.In theory, anyone can answer the survey.Multiple responses will vary with topic and survey length.Cannot use incentives.Research indicates that promises of anonymity and/or

confidentiality do not affect response rates except for sensitive topics such as sexual behavior.

Finer Points of Web AdministrationThe email should be concise, with hyperlink to

survey visible when participant opens the email.Provide a time estimate up frontAvoid sending emails on Monday – I don’t know why.I would send my first on a Tuesday, and my second

on a Saturday.How many? 2 or 3 reminders are fine, especially if

you allow participant to opt out.Paper based invitation asking for Web participation

does not work as well as an email invitation

Step 6. Analyze the DataYou may need to merge with other data

sourcesFrequencies, descriptives (e.g. mean and

sd), group comparisons (e.g., t-tests and ANOVAs), relationships (e.g., correlations and regression)

And finally – write the report.

SourcesDillman, Don A. (2009) Mail and Internet

Surveys: The Tailored Design Method. New York: John Wiley & Sons.

Stephen Porter and Michael Roy, Wesleyan Univ.

Gay, Mills, & Airasian (2009). Educational Research. New York: Pearson.

Donohue, E. (2004). Survey Techniques and Tactics. Google it.