Diana C. MutzUniversity of Pennsylvania

Random sampling + Random assignment =

Population-Based Experiments

Goal to increase methodological innovation and to expand the diversity of methods used across the social sciences

Particularly studies that benefit from combining the internal validity of experiments with the external validity of representative population samples.

NSF-supported infrastructure project that provides social scientists with opportunities for innovative data collection

TESS uses two large-scale data collection instruments

Internet surveys Telephone Interviews

Our goal is to increase methodological innovation and to expand the diversity of methods used across the social sciences.

Experiments as the Gold Standard for establishing causality:

“There are no complex statistical techniques that provide the power and elegance of a simple experimental design."

“College sophomores may not be people.” (C. Hovland).

“A greater effort must be made to conduct research on persons from life stages other than late adolescence.”

“… I suspect that even middle-aged people, separated from family and friends and confronted with test-like materials on novel and artificial topics in an academic laboratory, would often behave like college students do.” (D. Sears).

Allowed extensive pretesting. Investigators can listen in on

pretest interviews, and revise accordingly.

Respondents are sent pre-survey incentives and are paid for their interviews.

Timing can be coordinated with Internet data collection.

Address-based sampling (to include cell-phone-only households) for recruitment.

Free notebook computer and internet access if needed so both internet and non-internet households are included

Payment for study participation Special subpopulations known in advance Multimedia: Pictures/Video/Audio Rapid Turnaround

Decreasing investigator demand for this mode of data collection.

Limitations on experimental stimuli More difficult to recontact the same

respondents (greater attrition) Evidence on reliability of measures

generally better via internet Ability to randomly sample

specialized populations

Any faculty member or graduate student in any social science department anywhere in the world.

Proposals may come from any substantive area in the social sciences as long as they utilize experimental or quasi-experimental designs.

Repeated use is encouraged.Researchers to build on previous findings in subsequent proposals.

Any experiment or quasi-experimental study designed to explore substantive or methodological hypotheses within the social sciences.

Limits of request for sample and number of questions

Must make a clear contribution to knowledge. Particularly studies that would benefit from

combining the internal validity of experiments with the external validity of representative population samples.

Proposals are limited to 5 pages in length. Minimal investment of time in proposal

preparation Encourages fast turnaround from reviewers

Proposals are submitted through an easy online process. With just a few clicks of the mouse, your

proposal enters the review process.

Diverse Associate PIs oversee the review process…

Diversity of topics reduces potential contamination of respondents.Investigators need not share a survey interview.

Capitalizes on economies of scale to provide more data to more investigators than would otherwise be possible for the same cost.

Encourages social scientists to use methodologies they would not otherwise use.

Encourages quicker pace of research progress

Promotes more policy-relevant research

1. Experimental designs on policy-relevant populations

Experimentally investigated what preventative care benefits were most important to consumers

Respondents: 500 with a minor

present in their home 500 without a minor

present in their home

Respondents choose among three health plans, with the possibility of opting into special vaccination coverage.

 Respondents randomly assigned to receive threatening

or non-threatening feedback on a general knowledge

test or a “masculine knowledge test:”

When it is sold in the US, what country does the beer Heineken come from?

What are the true measures of a “two by four”?

Who had back-to-back MVP seasons in the NFL besides Joe Montana?

Which filter on a car typically requires changing every 3000 miles?

What is considered a good maximum benching weight?

Which tool clamps onto an object and will stay clamped without being held in place?

The conventional wisdom about infidelity based on student samples:

Men value sexual fidelity, whereas women value emotional fidelity.

The finding when research is based on a representative sample of adults:

Both men and women show more anger and blame over sexual infidelity, and both men and women show greater hurt feelings over emotional infidelity.

Do OCD screening batteries work less well for African-Americans?

If so, why?

2 by 2 experiment with representative samples of European-Americans (50%) and African-Americans (50%), asked about ethnic and racial identity either before the OCD battery (to increase salience of race), or after the scales were asked.

1) Same racial difference in average scores in representative samples as in college student sample. 2) Increasing the salience of race increases African-Americans’ OCD screening scores.

1. Experimental designs on policy-relevant populations

2. Using advance knowledge of respondent characteristics

1. Experimental designs on policy-relevant populations

2. Using advance knowledge of respondent characteristics

Representative sample of participants drawn from the population of individuals whose family income is less than $30,000/year

Representative sample of people residing in coastal areas

1. Experimental designs on policy-relevant populations

2. Using advance knowledge of respondent characteristics

3. More complex, subtle and engaging interactions with respondents

A person is “laid off” because his or her company had to reduce its staff.

“Think for a moment about the person and then tell me how much government help, if any, that person should receive while looking for a new job.”

One vignette, 96 variations:The person is a [white/black] [male/female] in [his/her] [early 20’s/mid 30’s/early 40’s]. [He/She] is [single/a single parent/married/married and has children] and [is/is not] a dependable worker.

Random samples of Black and White men and women evaluate Black and White, male and female models.

Asked to indicate when model becomes “overweight,” and when “obese”

Results:Female image perceived as overweight

and obese at significantly smaller body sizes than the male image.

Race of image did not matter, but race of perceiver did; White men judged bodies as obese and overweight at smaller body sizes than Black men

Does seeing the aftermath of natural disasters affect willingness to insure against them?

Tornadoes, Hurricanes, Fires

Disentangling direction of causality:

Perceived quality of neighborhood

Level of social organization, interaction

Do the deaths of private security personnel (i.e., “contract soldiers”) affect Americans less than deaths of military soldiers?

Do they elicit less of a response?Do they help insulate support for the war

from the death toll?

1. Experimental designs on policy-relevant populations

2. Using advance knowledge of respondent characteristics

3. More complex, subtle and engaging interactions with respondents

4. Repeated contact over time

Did Sotomayor’s confirmation change Hispanic attitudes toward government?Toward the Supreme Court?

Three groups interviewed after nomination, but before confirmation, then again after confirmation

Nominated, but not confirmed Post-confirmation

Nominated, but not confirmed Post-confirmation

1. Experimental designs on policy-relevant populations

2. Using advance knowledge of respondent characteristics

3. More complex, subtle and engaging interactions with respondents

4. Repeated contact over time5. “Firehouse” studies in response to

unfolding events

Respondents:

Georgia Residents

Half of Respondents, African-American

Half of Respondents, White

1. Experimental designs on policy-relevant populations

2. Using advance knowledge of respondent characteristics

3. More complex, subtle and engaging interactions with respondents

4. Repeated contact over time5. Firehouse studies6. Improving measurement, especially

sensitive issues and behavioral outcomes

Why such large discrepancies between the self-reports of heterosexual men and women?

Social desirability? Memory strategies?

Men - top of head multiplication Women – enumeration of individuals

Ball tossing game to induce feelings of rejection and social ostracism.

Source of ideas and examples for population-based experimental designs

Six thousand respondents answer two questions each.

Here is a list of things some people have done, please tell us how many you have done:

Written a letter to an editor of a newspaper Called into a talk radio program Visited a country outside of the United

States

Bodenhousen, “The role of gender in mental illness stigma”

Corrigan, “Shame, Blame and Contamination: Mental Illness Stigma and the Family”

Use of between subjects design, large samples, and anonymous internet surveys help acquire sensitive data

  2 (masculine vs. general knowledge test) x 2(threatening vs. non-threatening feedback) experimental design.

Rs complete a 10-question test. They are randomly assigned to the masculine-knowledge test or the general-knowledge test.

 After the test, Rs receive feedback on how they did. Rs are randomly assigned to receive threatening feedback or non-threatening feedback.

Dependent measures involve a series of questions about attitudes toward gay men.

 

    

Masculine Knowledge TestWhen it is sold in the US, what country does the

beer Heineken come from? What are the true measures of a “two by four”?Who had back-to-back MVP seasons in the NFL

besides Joe Montana?Which filter on a car typically requires changing

every 3000 miles?What is considered a good maximum benching

weight?Which tool clamps onto an object and will stay

clamped without being held in place?

Proposers may designate the field within which they would like their proposal reviewed.

After reading the proposal, a TESS Associate PI selects two reviewers appropriate to the topic and methods.

The reviewers are sent the anonymous proposal via email and respond within two weeks.

Continuous panel recruitment

Rotating panel with tenure of approximately 3

years Maximum of 1 survey per week; usually 3 surveys

per month involving a broad spectrum of topics and customers

Average 10-15 survey minutes in length per week, with some surveys of 25-35 minutes with incentives

Sample selection controls for prior survey taking to minimize panel conditioning