page 1 of 17 chapter 4 desiging studies section 4.1 -...

of 17

CHAPTER 4 DESIGING STUDIES

Section 4.1 - Sampling & Surveys (Part 1) pp. 206-223

1. Populations and Samples. Statistics is largely practiced in order to make inferences about

populations of individuals based upon a sample chosen to represent the population. In this section, we

are going to explore how to sample populations.

Definitions: The population in a statistical study is the entire group of individuals about which we want information. A sample is the part of the population from which we actually collect information. We use information from a sample to draw conclusions about the entire population.

Example: Identify the population and sample in each of the following examples:

(a) The student government at a high school surveys 100 of the students at the school to get their

opinion about a change in the bell schedule.

(b) The quality control manager at a bottling company selects a sample of 10 cans from the production

line every hour to see whether the volume of the soda is within acceptable limits.

2. The Idea of a Sample Survey

The first step in a sample survey is to say exactly what population we want to describe. The second step

is to say exactly what we want to measure, that is, to give exact definitions to our variables. The term

sample survey is reserved for studies that use an organized plan to choose a sample to represent a

population. The final step in planning a survey is to decide how to choose a sample from the population.

It should be noted that a survey or sample survey does not only refer to studies where people are asked

questions. Choosing the cans in the example above is a type of sample survey.

3. How to Sample Badly

a. Convenience Samples - A convenience sample is choosing individuals who are easiest to reach.

Example:

Convenience samples often produce unrepresentative data.

Convenience samples are almost guaranteed to be biased.

Definition: The design of a statistical study shows bias if it systematically favors certain outcomes. Note: when asked to identify bias in the design of a statistical study, you are expected to identify the direction of the bias.

of 17

Example:

of 17 b. Voluntary Samples - A voluntary response sample consists of people who choose themselves by

responding to a general appeal. Voluntary response samples show bias because people with strong

opinions (often in the same direction) are most likely to respond.

Example:

Write-in and call-in opinion polls are almost sure to lead to a strong bias.

Another problem is that people often times respond more than once.

4. How to Sample Well

A sample chosen by chance rules out favoritism by the sampler and self-selection by respondents.

Random sampling, the use of chance to select a sample, is central to the principle of statistical sampling.

Definition: A simple random sample (SRS) of size n consists of n individuals from the population chosen in such a way that every set of n individuals has an equal chance to be the sample actually selected.

Example:

An SRS not only gives each individual an equal chance of being chosen but also gives every

possible sample an equal chance of being chosen.

Often times a Table of Random Digits or a Random Number Generator are used to choose SRSs.

How to Choose an SRS Using Technology

Step 1: Label – Give each member of the population a distinct numerical label from 1 to N.

Step 2: Randomize – Use a random number generator to obtain n different integers from 1 to N.

of 17 How to Choose an SRS Using Table D

Step 1: Label - Give each member of the population a numerical label of the same length.

Step 2: Table - Read consecutive groups of digits of the appropriate length from Table D.

The sample contains the individuals whose labels you find.

Always use the shortest labels that will cover your population.

Ignore any group of digits that was not used as a label or that duplicates a label already in the

sample.

Digits can be read in any order but it is recommended to read rows from left to right.

Example. The management company of a local mall plans to survey a random sample of 3 stores to

determine the hours they would like to stay open during the holiday season.

a. Use a random number generator to select an SRS of size 3.

b. Use Table D at line 101 to select an SRS of size 3.

Aeropostale Just Sports

All American Burger Mrs. Fields

Arby’s Nike Factory Store

Barnes & Noble Old Navy

Carter’s for Kids Pac Sun

Destination Tan Panda Express

Famous Footwear Payless Shoes

Forever 21 Star Jewelers

GameStop Vitamin World

Gymboree Zales Diamond Store

Haggar

5. Other Sampling Methods

Unfortunately it is usually very difficult to actually obtain an SRS from the population of interest. It is

often costly in time and money.

Definition: To select stratified random sample, first classify the population into groups of similar individuals, called strata. Then choose a separate SRS in each stratum and combine these SRSs to form the full sample.

Example:

of 17

Choose the strata based upon facts known before the sample is taken.

If the individuals in each stratum are less varied than the population as a whole, a stratified

random sample can produce better information about the population than an SRS of the same

size.

Definition: To take a cluster sample, first divide the population into smaller groups. Ideally, these clusters should mirror the characteristics of the population. Then choose an SRS of the clusters. All individuals in the chosen clusters are included in the sample.

Example:

Cluster samples are often used for practical reasons.

They do not offer the statistical advantage of better information about the population that

stratified samples do.

Team Work - With your team mates, read and discuss the Sampling at a School Assembly Example on

pp. 221-222. Then list the advantages and disadvantages of each plan.

Plan Advantages Disadvantages

SRS

Stratified Random Sample

Cluster Sample

Multistage Samples - Most large scale sample surveys are multistage samples that combine two or more

sampling methods. A good example is on page 222.

of 17

Section 4.1 (Part 2) pp. 224-229

1. Inference for Sampling - The purpose of a sample is to give us information about a larger population.

This process is called inference because we infer information about the population from what we know

about the sample.

Inference from convenience samples or voluntary response samples would be misleading because these

methods are biased. They most likely are not representative of the population of interest.

The first reason to rely on random sampling is to eliminate bias in selecting samples from the list of all

available individuals. Even if we do this, it is unlikely that the results of a random sample are exactly the

same as the entire population. Properly designed samples avoid systematic bias but their results are

rarely exactly correct and we expect results to vary from sample to sample.

The second reason to use random sampling is that the laws of probability allow trustworthy inference

about the population.

It should also be noted at this point that larger random samples give better information about the

population than smaller samples.

Definition: A sampling frame is the list of individuals from which a sample is drawn.

2. Sample Surveys: What Can Go Wrong?

Random sampling eliminates bias in choosing a sample.

Even a large sample will give a result that differs from the truth about the population.

There will be a “sampling variability” that is described by the margin of error that comes with

most poll results.

Good sampling technique includes the art of reducing all sources of error.

of 17

There are two main sources of errors in sample surveys: sampling errors and nonsampling

errors.

Sampling Errors - The margin of error tells us how much sampling variability to expect (based upon

probability laws) and we can control it by choosing the size of our random sample. It does not tell us

about sampling errors -- mistakes made in the process of taking a sample that could lead to false

information about the population.

Bad sampling methods - voluntary response samples, convenience samples

Undercoverage - when some groups of the population are left out of the process of choosing

the sample.

Nonsampling Errors - Nonsampling errors are those that can plague even a census.

Nonresponse - Nonresponse occurs when an individual chosen for the sample cannot be

contacted or refuses to participate.

o Voluntary response sample versus nonresponse

Incorrect Response - Often times respondents “remember things that never happened” or

answer questions in a manner that is politically correct. Good interviewing technique or survey

implementation can help to reduce this.

o The wording of questions is the most important influence on the answers given to a

sample survey.

Example:

of 17


Experiments

A sample survey aims to gather information about a population without disturbing the population in the

process. Sample surveys are a type of observational study.

1. Observational Study versus Experiment

Experiments do not just observe individuals or ask them questions. They actively impose some sort of

treatment in order to measure the response. For example, “Does aspirin reduce the chance of a heart

attack?”

Definition: An observational study observes individuals and measures variables of interest but does not attempt to influence the responses. An experiment deliberately imposes some treatment on individuals to measure their responses.

The goal of an observational study can be to describe some group or situation, to compare

groups, or to examine relationships between variables.

The purpose of an experiment is to determine whether a treatment causes a change in the

response.

*************************************************************************************

When our goal is to understand cause and effect, experiments are the only source of fully convincing

data.

*************************************************************************************

Example - For a long time, scientists have believed that soy foods in Asian diets explain lower rates of

breast cancer, prostate cancer, osteoporosis, and heart disease in places like China and Japan. However,

when experiments were conducted, soy either had no effect or a small effect on the health of the

participants. For example, several studies randomly assigned elderly women to either soy or placebo,

and none of the studies showed that soy was more beneficial for preventing osteoporosis.

So what explains the lower rates of osteoporosis in Asian cultures? Genetics? Other dietary factors?

Other differences between cultures?

In this example, the explanatory variable was whether women got the soy or not, and the response

variable was whether or not they got osteoporosis. In the vocabulary of experimental design, another

name for explanatory variable is treatment.

of 17 In this example, the effect of eating soy was mixed up with the characteristics of women who ate the

soy. These characteristics are confounding variables.

Definition: Confounding occurs when two variables are associated in such a way that their effects on a response variable cannot be distinguished from each other.

Note: If you are asked to identify a possible confounding variable in a given setting, you are expected to

explain how the variable you choose (1) is associated with the explanatory variable, and (2) affects the

response variable.

Example - A common definition of “binge drinking” is 5 or more drinks at one sitting for men and 4 or

more for women. An observational study finds that students who binge drink have lower average GPA

than those who do not. Identify a variable that may be confounded with the effects of binge drinking.

Explain how confounding might occur.

When discussing confounding, it is best to explain what is happening in the context of the problem. Do

not overly rely on statistical vocabulary without additional explanation. Just explain what you see

happening.

2. The Language of Experiments

Definitions: A treatment is a specific condition applied to the individuals in an experiment. If an experiment has several explanatory variables, a treatment is a combination of specific values of these variables.

of 17

The experimental units are the smallest collection of individuals to which treatments are applied. When the units are human beings, they are often called subjects.

Example - A study published in the New England Journal of Medicine compared two medicines to treat

head lice: an oral medication called ivermectin and a topical lotion containing Malathion. Researchers

studied 812 people in 376 households in seven areas around the world. Of the 185 households

randomly assigned to ivermectin, 171 were free from head lice after two weeks compared to only 151 of

the 191 households randomly assigned to Malathion.

Identify the experimental units, explanatory and response variables, and the treatments in this

experiment.

The advantage of experiments over observational studies is that experiments can give good evidence for

causation.

Sometimes, the explanatory variables in an experiment are called factors. Many experiments study the

joint effect of several factors. In such an experiment, each treatment is formed by combining a specific

value (called a level) of each of the factors.

Example - Does adding fertilizer affect the productivity of tomato plants? How about the amount of

water given to the plants? A gardener plants 24 similar tomato plants in identical pots in his greenhouse.

He will add fertilizer to the soil in half of the pots. Also, he will water 8 of the plants with 0.5 gallon of

water per day, 8 of the plaints with 1 gallon, and the remaining 8 with 1.5 gallons. At the end of three

months he will record the weight of tomatoes produced on each plant.

Identify: (a) explanatory and response variables, (b) experimental units, (c) all treatments.

of 17 3. How to Experiment Badly


How to Experiment Well

1. Random Assignment

The remedy for confounding in the SAT Prep course example is to do a comparative experiment in which

some students are taught in the classroom and other similar students take the online course.

Comparison alone is not enough to produce results that can be trusted. If the treatments are given to

groups that differ greatly when the experiment begins, bias will result. The solution is random

assignment.

Definition: In an experiment, random assignment means that experimental units are assigned to treatments at random, that is, using some chance process.

Example - 50 students have agreed to participate in an experiment to compare the online SAT course

with traditional classroom instruction. Describe how you would randomly assign the two instructional

methods.

of 17

2. Three Principles of Experimental Design

Logic of randomized comparative experiments:

Ensures that influences other than the experimental treatments operate equally on all groups.

o Controls for the effects of other variables.

Balances out the effects of lurking variables that we cannot control or do not think of on

treatment groups.

o Random assignment forms groups of experimental units that should be similar.

Since groups are roughly equivalent except for treatments, any differences in average response

must be due either to the treatments or to the play of chance in the random assignment.

Principles of Experimental Design

The basic principles for designing experiments are: 1. Control for other variables that might affect the response: Use a comparative design and ensure that only the systematic difference between the groups is the treatment administered. 2. Random Assignment: Use impersonal chance to assign experimental units to treatments. This helps create roughly equivalent groups of experimental units by balancing the effects of other variables that are not controlled on the treatment groups. 3. Replication: Use enough experimental units in each group so that any differences in the effects of the treatments can be distinguished from chance differences between the groups.

Example - Many students regularly consume caffeine to help them stay alert. Thus, it seems plausible

that caffeine might increase an individual’s pulse rate. One way to investigate this is to have volunteers

measure their pulse rates, drink some cola with caffeine, measure their pulse rate after 10 minutes and

calculate difference. Unfortunately, even if every student’s pulse rate went up, we could not attribute it

to caffeine.

Explain how to use all three principles of experimental design in the caffeine experiment.

3. Completely Randomized Designs

of 17 This diagram details the SAT prep experiment: random assignment, the sizes of the groups and which

treatment they receive, and the response variable.

Definition: In a completely randomized design, the treatments are assigned to all the experimental units completely by chance.

Completely randomized design does not require that each treatment have equal number of

experimental units.

Assignment of treatments must occur completely at random. (The best bet is to choose them

using the “hat method.”)

Some experiments include a control group. The primary purpose of a control group is to provide a

baseline for comparing the treatments of the other treatments. Refer to example on p. 246.

It should be noted that although many experiments include a control group that receives an inactive

treatment, a control group can be given an active treatment. For example, if researchers are only

concerned with comparing two active treatments and do not care to determine if they are different than

no treatment.

of 17 Experiments: What Can Go Wrong?

The logic of a randomized comparative experiment depends on our ability to treat all the subjects the

same in every way except for the actual treatments being compared. Good experiments require careful

attention to details to ensure that all subjects are really treated identically.

Placebo Effect - The placebo effect is the tendency in humans to show a response whenever they think a

treatment is in effect. Well-designed experiments use a control group so that the placebo effect

operates equally on both the treatment group and the control group, thus allowing us to attribute

changes in the response variable to the explanatory variable.

Double-Blind - In a double-blind experiment, neither the subjects nor those who interact with them and

measure the response variable know which treatment a subject received.

Single-Blind - Some experiments cannot be carried out in a double-blind manner. Sometimes the

subjects know what treatment they are receiving. If those who interact with them do not know how the

individuals are treated, the experiment is single-blind.

4. Inference for Experiments - In an experiment, researchers usually hope to see a difference in the

responses so large that it is unlikely to happen just because of chance variation. We can use the laws of

probability, which describe chance behavior, to learn whether the effects are larger than we would

expect to see if only chance were operating. If they are, they are called statistically significant.

Definition: An observed effect so large that it would rarely occur by chance is called statistically significant.

If we observed statistically differences among groups in a randomized comparative experiment, we have

good evidence that the treatments actually caused these differences. A statistically significant

association in data from a well-designed experiment does imply causation.

of 17 Section 4.2 (Part 3) pp. 251-257

1. Block and Randomized Block Design

If our experimental units (subjects) differ in some characteristic that may affect the results of our

experiment, we should separate the groups into blocks based on that characteristic and then randomly

assign the subjects within each block. (Sounds like stratified sampling doesn’t it?)

We use blocks to reduce variability so that we can see the effects of the treatments.

The blocks themselves are not treatments.

Blocks are another form of control.

They control the effects of some outside variables by bringing those variables into the

experiment to form the blocks

Definition: A block is a group of experimental units that are known before the experiment to be similar in some way that is expected to affect the response to the treatments. In a randomized block design, the random assignment of experimental units to treatments is carried out separately within each block.

The idea of blocking is an important additional principle in experimental design.

A wise experimenter will form blocks based on the most important unavoidable sources of

variability (lurking variables) among the experimental units.

Randomization will then average out the effects of the remaining unknown variables.

Our goal is to be able to assess cause and effect relationship between the treatment imposed

and the response variable. Blocking reduces variability so that the differences we see can be

attributed to the treatment we imposed.

*************************************************************************************

Control what you can

Block on what you cannot control

Randomize to create comparable groups

*************************************************************************************

of 17 Example. A cell phone company is considering two different keyboard designs (A and B) for its new line

of cell phones. Researchers would like to conduct an experiment using subjects who are frequent

texters and subjects who are not frequent texters. The subjects will be asked to text several different

messages in 5 minutes. The response variable will be the number of correctly typed words.

a. Explain why a randomized block design might be preferable to a completely randomized design for

this experiment.

b. Outline a randomized block experiment using 100 frequent texters and 200 novice testers.

2. Matched Pairs Design - Matched pairs design is a common form of block design for comparing two

treatments.

The idea is to create blocks by matching pairs of similar experimental units.

Then we use chance to decide which member of the pair gets the first treatment. The other

subject gets the other treatment.

Just like other forms of blocking, matching helps reduce the effect of variation among the

experimental units.

Example: Marathon runners are matched by weight, physical build, and running times. They are asked

to test the durability of a new model of running shoe and compare it to the manufacturer’s old show

design. A coin is tossed to determine which runner in each pair will wear the new design. After the

marathon, the difference in wear pattern is then measured and recorded.

of 17 Sometimes each “pair” in a matched pairs design consists of just one experimental unit that gets both

treatments one after the other. In this case, each experimental unit serves as its own control. The order

of the treatments can influence the response, so we randomize the order for each experimental unit.

Example: A researcher believes that students are able to concentrate better while listening to classical

music. To test the theory, she plans to record the time it takes a student to complete a puzzle maze

while listening to classical music and the time it takes to complete another puzzle of the same difficulty

in a quiet room. Because there is so much variability in problem-solving abilities among students, a

matched pairs design will be used to reduce variability so that any difference recorded can be attributed

to the conditions under which the student completed the puzzle.

Each student will complete a puzzle under each of the conditions. A coin will be flipped to determine

whether they will first complete the puzzle in the quiet room or while listening to music. The difference

in time it takes to complete the puzzles is then recorded for each student.

page 1 of 17 chapter 4 desiging studies section 4.1 -...

Documents