the surrey business school teaching how to sample in research: some things that might help mark nk...

The Surrey Business Schoolwww.surrey.ac.uk/sbs

Teaching how to sample in research:

some things that might help

Mark NK Saunders


A sampling issue?

In February 2011 the UK’s Outdoor Media Centre launched its ‘Hall of Fame’ competition to identify the 100 best advertising posters of all time. Working with the History of Advertising Trust, they generated a list of 500 posters. This was reduced to a shortlist of 228 posters by a committee of media and creative experts together with the editor of weekly magazine - Campaign. These were displayed on a dedicated website www.outdoorhalloffame.co.uk. Creative agencies, media planners, advertisers, media owners and the general public were invited in an article in Campaign to go to the web site, view the advertising campaigns and cast their votes for what they considered to be the best outdoor posters. Each person was able to cast a total of ten votes, the best advertisements being “chosen after more than 10,000 reader votes”.


How might sample selection have affected the results?

12

3

4

5

67

8


Session PlanSampling is not boring –attracting attentionSampling does not always involve balls and urns –introducing non probability samplingSampling does not always require a sampling frameSampling and statistics –misconceptions about probability samplingThe sample selected matters –the link with research question The how many question -the issue of sample size


Attracting attention

“A small test of 50 women found

almost nine out of 10 of them said their skin looked more luminous after using it, while 72 per cent said fine lines appeared less visible.”


Introducing non probability sampling (1)


Introducing non probability sampling (2)

Some implied research questions (students’ suggestions)

How do people manage their careers?

What is the best way to manage a career?

Guildford


Sampling does not always require a sampling frame…

Data collected by… Catching commuter train from

Guildford to London 2nd class ticket 5 weeks – Monday to Friday Interviewing 5 people on average

each journey

Unclear regarding… Time of year How selected participants


The sample selected matters

Characteristics of population from which sample selected

Catching commuter train (commuters?)

Living near Portsmouth – Guildford – London line

Working in London

Working “9 to 5”

What else?


Sample selected matters (2)…Generalisabiity: the way in which the sample is drawn affects the extent

to which the findings can be applied (generalised) or other settings

Sample from: Commuters Living south of London Unlikely to be senior

managers Working “9 to 5” Unlikely to be employed

in manufacturing

Research Question People Living anywhere With a career Doing any type of job


So what have we done so far?

Hopefully sampling is not boring –real issue in research

Sampling does not always involve balls and urns –introduced non probability sampling and shown in relation to this…

…sampling does not always rely on a sampling frame –can’t get a list of passengers easily

The sample selected matters…


Testing understanding…What is the relationship between a population and a sample?

1 2 3 4

0% 0%0%0%

1. The sample is a sub set of the population

2. The sample is the population

3. The sample is always representative of the population

4. The sample is the method by which you select the population

15

www.rwpoll.com


Samples … realities in management research (& the Guildford study)

REALITIES

Don’t always have sampling frame

Sampling frame often restricted

Difficult to show sample truly random

CONSEQUENCES

Inferences (including statistical) confined to actual population from which sample drawn –can’t generalise statistically

May not be able to answer certain questions … rephrase the question!

Population assumed itself to be a random sample from larger population or


Unclear how non-probability sample was drawn

Sampling

Probability Non-probability

Volunteer Haphazard

Snowball Selfselection Convenience

Heterogeneouspurposive

Homogeneouspurposive

Critical casepurposive

Typicalcase

purposiveTheoretical

Purposive

Extremecase

purposive

Quota

Quota

But different non-

probability techniques would have

differing implications


The how many question –what non-probability sample size?

REMEMBER –making generalisations to theory not about a population (except with quota samples)

Sample size depends upon research question and objectives… what need to find out what will be useful what is credible what can be done within available resources

Continue until reach data saturation


Non-probability sample size ...more detail

Author Nature of study Size

Bertaux (1981) Qualitative 15

Kvale and Brinkmann (2009) Interviews 5-25

Bernard (2000); Morse (1994) Ethnographic 35 - 36

Creswell (1998); Morse (1994) Grounded theory 20 - 35

Creswell (1998); Morse (1994) Phenomenological 5 - 25

Guest et al. (2006); Kuzel (1992); Romney, Batchelder and Weller (1986)

Considering a homogenous population

4 – 12

Kuzel (1992); Creswell (2007) Considering a heterogeneous population

12 - 30

Source: Saunders (2012)


Non probability sample selection

Justifying choices Following a recipe

for Quota sampling


Misconceptions about probability sampling –simple random sampling

Selecting a sample using simple random sampling will select the sample evenly from the population

If a truly random process is used to select a sample from a population, then the resulting sample will be just like the population, but smaller –a miniature replica of the population.

The size of the sample should be proportional to the size of the population. -the ratio of sample size to population size needs to be considered when deciding how large the sample should be.

The larger the sample the greater the accuracy


Misconceptions…-simple random sampling selects evenly

Last five UK lottery draws

5 17 32 39 42 40

9 22 31 34 35 43

7 10 12 15 25 42

2 9 27 33 39 40

8 14 15 17 20 47

Look at the pattern of numbers


Misconceptions about probability sampling –miniature replica

Write down your student number.

Then answer the questions on the slide for the last digit of your student number115678 = 8113459 = 9 etc etc

Then, I will show the questions again but this time we will ALL answer them

In comparing the answers, we will be able to see realities of random sampling…. (and of questionnaires!)

(Idea from Prof Sam Warren)


I often receive bad service when I am shopping in England (student number ending 1, 4, 5)

1 2 3 4 5

20%

17%

20%

17%

27%

1. Strongly agree

2. Agree

3. Neither agree or disagree

4. Disagree

5. Strongly disagree


Results –a miniature replica?

Students Number Strongly agree

Agree Neither Disagree Strongly disagree

1,4,5 57 20% 17% 27% 17% 20%2,3,8 63 12% 19% 29% 23% 17%6,7,0 47 19% 18% 26% 19% 18%All (incl. 9)

162 17% 18% 28% 20% 17%

Impact of larger

sample size

Variation in number

responding

Variations in response


Misconception: Sample size is proportional to population(1)

Sample size for 95% certainty with 5% margin of error

0

50

100

150

200

250

300

350

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Total Population

Sam

ple

siz

e

The problem of making statistical inferences from a small probability sample


Probability sample and population size (2)

0

50

100

150

200

250

300

350

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Total Population

Sam

ple

siz

e

population 5% 2%

10,000 370 1936

1,000,000 384 2395

10,000,000 384 2400

Sample size for 95% certainty with 5% margin of error


Misconception: larger the sample size greater the accuracy

0

0.01

0.02

0.03

0.04

0.05

0 2000 4000 6000 8000 10000

The larger the probability sample size, the more confident can be that represents the true population for a given population size.

population 5% 2%

100 79 96

200 132 185

500 217 226


Different probability sampling techniques would have different implications (given knew total population)

Systematic Stratifiedrandom

ClusterSimplerandom

Multi-stage

Sampling

Probability


Probability sampling –following a recipe

How to do it most text books -just follow the recipe…-use practice exercise-justify choices

Key issue is understanding why and implications of what do-discuss case studies...-use multiple choice questions to check http://wps.pearsoned.co.uk/ema_uk_he_saunders_resmethbus_5/111/28550/7309036.cw/index.html


Probability samples … statistical inference Inferential statistics allow

general statements to be made about the population form the sample providing data normally distributed

Betting about the probability of being wrong or inferring invalid conclusions

“It is a sad fact that if one knows nothing about the probability of occurrence for aparticular sample of units for observation, very little of the inferential machinery … can be applied”

(Hays 1994: 227)


But beware of the effect of sample size on practical significance…

The larger the sample size the more likely your results are statistically significant

Statistically significant result - unlikely to have occurred by chance

Practically significant result – practically meaningful in real world

Jacob Cohen


Response ratesMost important aspect is sample is representative of ...

High response rate can help this

Non-responses different from rest of population as did not respond!

Non response due to 4 interrelated problems:

refusal to respond

ineligibility to respond

inability to locate respondent

respondent located but unable to make contact


Bibliography (1)

American Association for Public Opinion Research (2008) Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (5th edition) Lenexa, KA: AAPOR

Baruch Y and Holtom BC (2008) Survey response rate levels and trends in organizational research Human Relations 6.8 1139-60.

Groves, R.M., Dillman, D.A., Eltinge, J.L. and Little, R.J.A. (eds.) (2001) Survey Non-response. New York: John Wiley.

Guest, G., Bunce, A. and Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and validity. Field Methods. Vol. 18, No. 1, pp. 59-82

Lance, C.E. & Vandenberg, R. J. (2009), Statistical and Methodological Myths and Urban Legends. Routledge, London:Taylor & Francis.


Bibliography (2)

Meyer, J.H.F., Shanahan, M.P. & Laugksch, R.C. (2005), “Students’ Conceptions of Research. I: A qualitative and quantitative analysis”, Scandinavian Journal of Educational Research, 49. 3, 225-244.

Patton, M.Q. (2002) Qualitative Research and Evaluation Methods. 3rd edn. Thousand Oaks, CA: Sage, Ch.5

Rogelberg SG and Stanton JM (2007) Introduction: Understanding and dealing with organizational survey non-response Organizational Research Methods 10.2 195-209

Saunders, M.N.K., Lewis, P. and Thornhill, A. (2012) Research Methods for Business Students. 6th edn. London: FT Prentice Hall, Ch.7.

Saunders M.N.K. (2012) ‘Choosing research participants’ in G Symons and C Cassell (eds) The Practice of Qualitative Organizational Research: Core Methods and Current Challenges.London: Sage 37-55

the surrey business school teaching how to sample in research: some things that might help mark nk...

Documents

surrey business school

probability sampling

sampling frame sampling

sampling issue

guildford slide

issue of sample size

sampling frame data

settings sample