david geddes, ph.d. senior vice president & partner fleishman-hillard institute for public...

David Geddes, Ph.D.Senior Vice President & PartnerFleishman-Hillard

Institute for Public Relations Measurement Commission

21 tips for conducting 21 tips for conducting evaluation research evaluation research

on a boot laceon a boot lace

2

Today’s objectives

Advice on doing it yourself

Advice on being a good client

3

Where to get help

University departments

Social work, Sociology, Psychology, Political Science, Business, and others

Contacts via board of directors

American Marketing Association

Market Research Association

Market research firms

LinkedIn (www.linkedin.com)

National Opinion Research Center (University of Chicago) http://www.whatisasurvey.info/

4

http://www.italladdsup.gov/community_partners/dc_teachmehow_02.asp

Introduction to evaluation research

Process evaluation examples

Telephone survey with annotations

Intercept and self-administered survey

5

Build a community

Vision

Lutheran Social Services Research Community

Share best practices

Methods

Facebook or other collaboration tool

6

Set your objectives

Program objectives

Specific information needs

1

7

Do your background research2

8

Think about different types of evaluation and measurement

1. Formative evaluation

Helps planning

2. Process evaluation

How well is the program operating?

3. Impact evaluation

Building awareness and understanding?

4. Outcome evaluation

Concrete program or “business” metrics

3

9

Don’t assume you know audience attitudes and opinions

You need formative research

4

10

Set up your process metrics5

11

Measure impacts

Awareness

Knowledge

Attitudes and perceptions

Intended actions

6

12

Identify survey targets

Known respondents

Population =Sample frame Members Donors Clients Employees Civic organization

leaders

Defined population

Sample frame derived from population General public Congregation members Potential donors Potential adoptive

parents

7

13

Pick the right survey method to reach targets

Telephone

Mail

Online

Mall or store intercept

8

14

Telephone surveys

Advantages

Speed, in some cases

Control

Probe and clarify

Cost if you do it yourselfDisadvantages

Cost if you go outside

Hard to reach some socio-economic groups

15

Telephone surveys

Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. $8,000 = $16 cost per interview Review cost parameters

Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. $4,000= $20 cost per interview Review cost parameters

16

Telephone surveys

Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys.

$7,000 = $35 cost per interview

17

Doing your own telephone surveys

Sample list

Staff

Interns, volunteers, students, etc.

Call management system

Data recording system

Discipline

Persistence

18

Going outside for telephone surveys

Smaller local firms

Be flexible on timing

19

Mail surveys

Advantages Lower cost Easy if you have a mailing list Easier to do yourself

Disadvantages Slow Lower control Lower response rates Need a mailing list

20

Doing your own mail surveys

Clear questionnaire layout

Sample list

Staff

Interns, volunteers, students, etc.

Lots of manual paper handling

Discipline

Patience

21

Online surveys

Advantages

Speed, in some cases

Control

Cost if you do it yourself (SurveyMonkey)Disadvantages

Purchase a sample list

Unrepresentative of some socioeconomic groups

22

Online surveys

Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. $3,300 = $6.60 cost per interview Review cost parameters

23

Doing your own Online surveys

SurveyMonkey programmer

Sample list

Staff

Interns, volunteers, students, etc.

24

Mall intercept surveys

Advantages Control Probe and clarify Cost if you do it yourself

Disadvantages Slow Need mall or store cooperation Unrepresentative of some socioeconomic

groups

25

Doing your own mall intercept surveys

Cooperation of mall or Wal-Mart

Staff

Interns, volunteers, students, etc.

Discipline

Patience .. Figure on one interview/hour

26

Select an appropriate sample size

What are possible sources of error?

How far off could we be?

What sample size do we need?

8a

27

Give advance notice

If you know the target respondents

Newsletter

E-mail

Postcard

Benefit: improves response rate

9

28

Remember you are asking for their time

This applies to…

Pre-survey letters or cards

Introduction to the survey

And the questions themselves

Don’t ask meaningless questions

10

29

Write a good introduction to the questionnaire

Introduce yourself

“Research” not “survey”

Survey topic: social services

Not a sales call … not a fundraising call

You want their opinion / input

11

30

Move from general to specific12

31

Lead with an interesting question

Engage the respondent

Ask interesting questions and people will talk

13

32

Make judicious use of open-ended questions

Example:

“Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else?”

2 to 3 per questionnaire maximum

Finish with a “final thoughts” question

14

33

Use good response scales

Building a scale

Clearly distinguish good and bad

1 Agree

2 Disagree

15

34

Use good response scales

Building a scale

Add a polite positive and negative

1 Strongly Agree

2 Agree

3 Disagree

4 Strongly disagree

35

Use good response scales

Building a scale

Add a top positive response

1 Very strongly agree

2 Strongly Agree

3 Agree

4 Disagree

5 Strongly disagree

36

Use good response scales

Building a scale Add a don’t know/no opinion

1 Very strongly agree

2 Strongly Agree

3 Agree

4 Disagree

5 Strongly disagree

6 Don’t know / no opinion

37

Use good response scales

Building a scale Add a mid-point if you really need it ... and

balance the scale 1 Very strongly agree 2 Strongly Agree 3 Agree 4 Neither agree nor disagree 5 Disagree 6 Strongly disagree 7 Very strongly disagree 8 Don’t know / no opinion

38

Make sure you cover all possible responses

Multiple choice questions

16

39

“No opinion” does not equal “Don’t know”17

40

Always include a “refused” option18

41

Be careful with numeric responses

“About how much do you spend on gas each month” $__________

Which of the following best describes the amount you spend on gas each month: Under $25

$26 to $50

$51 to $100

Over $100

19

42

Be careful about “do you floss your teeth” items

Socially desirable responses

Responses that may imply guilt

20

43

Test your questionnaire21

44

Wrap-up

• Begin with your objectives and information needs

• Define the people you want to survey• Select an appropriate survey method• Write good questions• Test• Launch the survey• Analyze the results

45

Exercise