Survey exercise follow-up

Survey population Sample?

Random Purposive Convenience

Entire population Small, bounded population

All the people who work here

Non-response bias an issue in each case

Sample design Survey element – unit ABOUT WHICH information is

collected; unit of analysis E.g., Person; household; event

Survey universe – theoretical and hypothetical aggregation of elements E.g., Americans; users; visits

Population – specified aggregation of survey elements: Households in the US, defined as one person, or a

collection of related or unrelated persons…as of March, 2002

Visits to any web page in defined domain during March, 2002

Sample Design, cont. Survey population -- Aggregation of elements from

which survey sample is actually selected Households in 50 states with listed telephone numbers

Sampling unit – elements consider for selection at some stage listed residential phone numbers If multi-stage sample:

State; telephone directory; phone numbers Sampling frame – list of sampling units

Telephone directories for specified places Observation unit – element from which info collected

person over 18 who either answers the phone or is fetched by a person under 18 answering the phone

Sample selection Consider possible biases of sample design Mixed methods may give uninterpretable samples

E.g. mail to a group AND hand out in Sproul Plaza – so what population does your sample represent? And people on the mailing list who walk through Sproul

have *2* chances of being sampled Consider practical issues

Confidentiality often precludes getting a list, e.g. from the university

If you can’t quantify units (people, events), you can’t sample proportionately

Non-users or non-affiliated users (e.g., potential students; web users with no particular connection to the site) are hardest to track down

Introducing survey Purpose of introduction

Justify your asking Induce their cooperation Allay possible fears

“this is not a test…” Help them understand domain of survey, questions

Describing purpose of survey Enough to gain cooperation, not enough to bias

answers E.g. “a study of compensation in this field,” not “a study of

whether women’s salaries lag behind men’s in this field” SHORT but not at the expense of clarity.

Getting cooperation

Worthwhile topic, survey Not threatening Not a waste of time Interesting to them

Good design If you are not credible, not worth their

effort Short and easy to answer

Directions Short and clear People skim! The more you ask of them, the more likely

they are to mess up “please rate from 1, most important, to 7, least

important” Give instructions

For skip patterns “Did you use the library today? If not, go to

question 3. If yes,… For what to do if question does not apply

Choice of questions

Think through: what will you do with the answers?

If you don’t know, don’t ask If you do know, word in a way that

helps you E. g. “Have you ever used a library?”

What difference will it make if they answer “no”?

Choice of questions Respondents must be qualified to answer

“What would be the best way to bring peace to the Middle East?”

Respondents must think the topics of the questions are: Not invasion of privacy Not “dumb questions”; worth their time Not a test

Of them Of some idea or proposal they may not like

(unless you mean it) “if we had to cut the budget, should we lay people

off or reduce salaries?”

Time dimension to questions General, usual, most often:

What do you usually do? Specific

What did you do today? What have done in the last month?

NOT: What do you use this site for? (today? Ever?)

Memory What’s important to you may not be to them

“what did you do the last time you used this site?” Time and frequency are hard to remember

“Approximately how many times in the last month…”

Question wording Resolve any possible ambiguity!

Definitions, illustrations, examples Have you ever used a library? What’s a library? “Did you use the library’s homepage?”

Which page is that? What proportion of your income do you pay for housing?

Individual or household income? Housing: rent or mortgage; taxes? Insurance?

You cannot use “etc.” unless the reader can complete the list NO: “borrow library books, ask ref questions, etc.” YES: “borrow library books, ask reference questions, or

otherwise use the library’s resources or staff, in person or remotely.”

Use parallel construction What did you use this site for today?

Searching for information Placing an order Checking a previous orderNOT Searched for information I wanted to check on an earlier order

What did you use this site for today? If yes… What was the question to which “yes” was the

answer?

Answers Closed-ended are easier (which is usually,

but not always, a good thing) Easier for users (recognition, not recall; less

writing) They are less likely to think of different choices

Easier for you to summarize Be as exhaustive as possible with choices

If you leave out an answer that a user might reasonably give, they may or may not think to add it

They may become confused about the question or the survey

Answers, cont. Allow degrees of answers

Yes/ No, always/never are rarely the case E.g., Usually; Sometimes; Rarely

Allow multiple dimensions of answers How satisfied are you with your visit today?

Very: found what I was looking for. Not very: did not find.. What if I found what I needed but with

difficulty? I may have been successful but am dissatisfied with the experience.

Answers, cont. Give clear instructions

Check one? Check the one that best describes…? Check as many as apply?

Define! Lay out clearly

E.g. Ordinal scales all on one line How satisfied are you?

Very satisfied somewhat satisfied not at all OR

Very somewhat not at all

Satisfied satisfiedsatisfied

NOT: How satisfied are you? Very satisfied Somewhat satisfied Not at all satisfied

Ranges and intervals Want answers to be easy to for your reader

to interpret Generally, use either:

Intervals of equal size OR Intervals reflecting meaningful groupings

Age: under 18 18-25 25-40 40-55… OR intervals get larger when you expect fewer

responses How many years have you worked here?

Less than 1 1-3 4-10 10-20 more than 20

Avoid: Years worked here

1 year 10% 2-5 years 15% 6-7 years 20% 8-15 years 50%

Examples Have you checked out a book from

the UC Berkeley Library system? Yes no

If you use Pathfinder, Gladis, or Melvyl to find books or periodicals, where do you use them? Terminal at main library Terminal at my departmental library I don’t use these services

Examples If you could allocate funds for more salary and had

to pick from the following places, how would you prioritize where you would obtain these funds?(1 – first choice to find money through 7 – last place you

would go) tuition Pension plan Taxes Decreased health coverage State bonds Fund raising Less vacation/sick leave

Examples How long has your library offered live online

reference service? How long have you been a Reference

Librarian? (providing reference services comprises at least 40% of your day) Up to 5 years Between 5 and 10 years Between 10 and 15 years Over 15 years

Examples

Which part(s) of this web site interests you the most?

Which part(s) of this web site do you find the most uninteresting?

What parts of this web site do you like the most?

Which parts of this web site do you dislike the most?

Examples 1. Have you ever used the following online

services: [list]

2. If you checked any box in question 1, how important do you think online help would be in each of these activities? (only respond as applicable) [Item] Absolutely unimportant |--|--|--||--|--|--| absolutely important