Lecture 5Lecture 5

Data Coding and Experimental Data Coding and Experimental Research MethodsResearch Methods

22

OverviewOverview

Research Problems: my commentsResearch Problems: my comments

Working with DataWorking with Data Error-CheckingError-Checking Basic Data PreparationBasic Data Preparation

MethodsMethods Introduction to Experimental MethodsIntroduction to Experimental Methods

33

Problems and JustificationsProblems and Justifications

Research Question/Problem: What is the effect Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry?degree for getting a job in the tech industry?

Different types of degrees teach different skill Different types of degrees teach different skill sets. Jobs in the tech industry will look for these sets. Jobs in the tech industry will look for these skill sets, which should lead to different skill sets, which should lead to different outcomes for CS students versus MIMS outcomes for CS students versus MIMS students.students. Not a justification by itself– this is part of the Not a justification by itself– this is part of the context context

of the argumentof the argument. By itself, this just sets up an . By itself, this just sets up an argument that will lead to specific hypotheses.argument that will lead to specific hypotheses.

44

Problems and JustificationsProblems and Justifications(Round 2)(Round 2)

Research Question/Problem: What is the effect Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry?degree for getting a job in the tech industry?

If we can understand how different degrees (CS If we can understand how different degrees (CS versus MIMS) give different job opportunities, it versus MIMS) give different job opportunities, it has implications for future enrollment in various has implications for future enrollment in various academic programs. Furthermore, this research academic programs. Furthermore, this research could help us to begin the process of could help us to begin the process of differentiating between various fields of study.differentiating between various fields of study. Nope, still not a justification. These are Nope, still not a justification. These are implicationsimplications

of the research.of the research.

55

Problems and Justifications Problems and Justifications (Round 3)(Round 3)

Research Question/Problem: What is the effect of obtaining a MIMS Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech degree versus a CS MA degree for getting a job in the tech industry?industry?

Knowing which graduate degree to obtain is a significant problem Knowing which graduate degree to obtain is a significant problem for those who want the best jobs in the tech industry. Once a for those who want the best jobs in the tech industry. Once a student joins a program, he or she makes a large commitment of student joins a program, he or she makes a large commitment of time and money. Degree-granting programs often entice potential time and money. Degree-granting programs often entice potential students to enroll based on the potential job prospects that they will students to enroll based on the potential job prospects that they will have once they receive a degree. Yet, it is not always clear to have once they receive a degree. Yet, it is not always clear to prospective students what skill sets are prospective students what skill sets are offeredoffered by different by different programs, and which skill sets are being programs, and which skill sets are being sought sought by the best by the best employers in the tech industry. This research will increase our employers in the tech industry. This research will increase our understanding of the relationship between degree type (CS versus understanding of the relationship between degree type (CS versus MIMS) and the related skill sets that ultimately lead to technology MIMS) and the related skill sets that ultimately lead to technology job outcomes. job outcomes.

66

Data PreparationData Preparation

Basic Data SetupBasic Data Setup Unique ID for each caseUnique ID for each case Numeric responses whenever possibleNumeric responses whenever possible

Includes categorical, scale, count, etcIncludes categorical, scale, count, etc Variable namesVariable names Variable labelVariable label Value labelValue label Missing CodesMissing Codes

RecodingRecoding SimpleSimple Intermediate (equations, computations)Intermediate (equations, computations)

77

Error Checking ExamplesError Checking Examples

Checking Original Variables for ErrorsChecking Original Variables for Errors FrequenciesFrequencies DescriptivesDescriptives

Checking and Setting “Missing” codesChecking and Setting “Missing” codes

Recoding and Creating New Variables from Recoding and Creating New Variables from Existing VariablesExisting Variables FrequenciesFrequencies Cross-TabulationsCross-Tabulations

88

Two Example Datasets:Two Example Datasets:

Class Data SetClass Data Set

Subset of 1993 General Social Survey Subset of 1993 General Social Survey

Introduction to Experimental Introduction to Experimental Design and MethodsDesign and Methods

1010

Pro’s and Con’s of ExperimentsPro’s and Con’s of Experiments

Pro’sPro’s Gives researcher tight control over independent factorsGives researcher tight control over independent factors Allows researcher to test key relationships with as few Allows researcher to test key relationships with as few

confounding factors as possibleconfounding factors as possible Allows for direct causal testingAllows for direct causal testing

Con’sCon’s Usually a smaller N than surveysUsually a smaller N than surveys Sometimes give up large amounts of external validity in favor of Sometimes give up large amounts of external validity in favor of

construct validity and direct causal analysisconstruct validity and direct causal analysis Require a large amount of planning, training, and time– Require a large amount of planning, training, and time–

sometimes to test relationship between only 2 factors!sometimes to test relationship between only 2 factors!

1111

Active versus Attribute Independent Active versus Attribute Independent VariablesVariables

ActiveActive independent variable(s): independent variable(s): The I.V. is given to the participants, usually for some The I.V. is given to the participants, usually for some

specified time period. It is often manipulated and specified time period. It is often manipulated and controlled by the investigator.controlled by the investigator.

Attribute Attribute independent variable(s):independent variable(s): A predictor, but a defining characteristic of individuals. A predictor, but a defining characteristic of individuals.

Cannot be manipulated. Cannot be manipulated.

1212

True ExperimentsTrue Experiments

True experimentsTrue experiments protect against both time and group threats to protect against both time and group threats to internal validity by internal validity by randomly assigning subjectsrandomly assigning subjects to treatment and to treatment and control groups. The treatment (independent variable) is active.control groups. The treatment (independent variable) is active.

If we cannot randomly assign subjects to different groups, then it is If we cannot randomly assign subjects to different groups, then it is a a quasi-experimentquasi-experiment. . The independent variable is active.The independent variable is active.

If we cannot randomly assign subjects to groups because the If we cannot randomly assign subjects to groups because the groups groups containcontain the attribute of interest, and if we give all groups the the attribute of interest, and if we give all groups the same treatment, then it is an same treatment, then it is an associational non-experimentassociational non-experiment.. The The independent variable is not active.independent variable is not active.

1313

Randomization in Sample and Randomization in Sample and AssignmentAssignment

Random Sample Random Sample System for choosing participants from a System for choosing participants from a

populationpopulation Generally, the larger the sampling population Generally, the larger the sampling population

the better your generalizability becomes.the better your generalizability becomes.

Random AssignmentRandom Assignment Method for assigning participants randomly to Method for assigning participants randomly to

experiment conditionsexperiment conditions

1414

Two Essential Criteria in True Two Essential Criteria in True Randomized Experimental DesignRandomized Experimental Design

(1) Independent Variables must be (1) Independent Variables must be manipulated (usually by experimenter, manipulated (usually by experimenter, sometimes by context)sometimes by context)

(2) Participants must be assigned (2) Participants must be assigned randomly to various conditions or groupsrandomly to various conditions or groups

1515

Pre-test, experimental manipulation Pre-test, experimental manipulation and post-testingand post-testing

Pre-test:Pre-test: allows us to check group equivalence allows us to check group equivalence before the intervention X is introduced.before the intervention X is introduced.

Experimental manipulation:Experimental manipulation: An independent An independent variable (X) that the experimenter manipulates.variable (X) that the experimenter manipulates.

Post-test:Post-test: allows us to check group equivalence allows us to check group equivalence after intervention X has been introduced. after intervention X has been introduced.

1616

Common types of true experimentsCommon types of true experiments

R

O

O

X O

O

R

X

O

O

R

O

O

X O

O

(1)

(2)

X O (3)

O (4)

Pretest-Posttest Control

Post-only Control

Solomon 4-group

1717

Example: Pen StudyExample: Pen Study

Question: Do individuals in Japan and the Question: Do individuals in Japan and the US make differential choices about US make differential choices about ‘unique’ versus ‘less unique’ items when ‘unique’ versus ‘less unique’ items when given a choice?given a choice?

1818

Pen StudyPen Study

Independent VariableIndependent Variable Cultural difference: Japanese students compared to Cultural difference: Japanese students compared to

US studentsUS students

AssignmentAssignment Subjects were not randomly assigned because they Subjects were not randomly assigned because they

already fell into one of the two groups.already fell into one of the two groups.

Dependent Variable:Dependent Variable: Pen layout (3 of one type, 1 of another)Pen layout (3 of one type, 1 of another) Would they choose the ‘common’ pen or the ‘unique’ Would they choose the ‘common’ pen or the ‘unique’

one?one?

1919

Example: Trust-Building StudyExample: Trust-Building Study

Question: Do increased risk-taking Question: Do increased risk-taking behaviors over time increase interpersonal behaviors over time increase interpersonal trust?trust?

2020

Trust-Building Study Trust-Building Study

Independent VariableIndependent Variable Experiment Condition (3 conditions): Experiment Condition (3 conditions):

Fixed partner on every trial, cannot control amount to entrust to partnerFixed partner on every trial, cannot control amount to entrust to partnerFixed partner on every trial, can control amount to entrust to partnerFixed partner on every trial, can control amount to entrust to partnerRandom partner on every trial, can control amount to entrust to partnerRandom partner on every trial, can control amount to entrust to partner

AssignmentAssignment Random assignment of participants to one of the 3 conditions.Random assignment of participants to one of the 3 conditions. Same experiment conducted in Japan and US, and comparisons made Same experiment conducted in Japan and US, and comparisons made

between the two studies.between the two studies.

Dependent VariableDependent Variable Cooperation rate (i.e., whether they returned the coins to the partner or Cooperation rate (i.e., whether they returned the coins to the partner or

not)not)

Validity: RevisitedValidity: Revisited

2222

Experiment Construct ValidityExperiment Construct Validity

How do we know that our independent variable How do we know that our independent variable is reflecting the is reflecting the intended causal constructintended causal construct and and nothing else?nothing else?

ContaminationContamination Demand CharacteristicsDemand Characteristics

AnythingAnything in the experiment that could guide subjects to in the experiment that could guide subjects to expected outcomeexpected outcome

Experimenter ExpectancyExperimenter ExpectancyResearcher behaviorResearcher behavior that guides subjects to expected that guides subjects to expected outcome (self-fulfilling prophecy)outcome (self-fulfilling prophecy)

2323

Solving Expectancy EffectsSolving Expectancy Effects

Naïve experimenterNaïve experimenter Those conducting study are not aware of theory or hypotheses Those conducting study are not aware of theory or hypotheses

in the experimentin the experiment

Blind Blind Researcher is unaware of the experiment condition that he/she Researcher is unaware of the experiment condition that he/she

is administeringis administering

StandardizationStandardization Experimenter follows a script, and only the scriptExperimenter follows a script, and only the script

““Canned” ExperimenterCanned” Experimenter Audio/Video/Print material gives instructionsAudio/Video/Print material gives instructions

2424

Other Issues with Demand Other Issues with Demand CharacteristicsCharacteristics

Evaluation Apprehension:Evaluation Apprehension: Subjects Subjects knowknow that they are being evaluated and this that they are being evaluated and this

changes their behaviorchanges their behavior

SolutionsSolutions Double-blind experimentsDouble-blind experiments Experiments in natural setting (i.e., subjects do not Experiments in natural setting (i.e., subjects do not

know they are in an experiment)know they are in an experiment) Cover storiesCover stories Hidden measurementsHidden measurements ““Faithful Subject”Faithful Subject”

2525

Generalizability (external validity) in Generalizability (external validity) in ExperimentsExperiments

Threats to external validity always involve Threats to external validity always involve an interaction of the treatment group with an interaction of the treatment group with some other factor.some other factor.

Threats usually fall into 3 types:Threats usually fall into 3 types: SettingSetting PopulationPopulation HistoryHistory

2626

Three threats to generalizability in Three threats to generalizability in experimentsexperiments

SettingSetting Physical and social context of the experimentPhysical and social context of the experiment

PopulationPopulation Is there something specific about the sample that Is there something specific about the sample that

interacts with the treatment?interacts with the treatment?

HistoryHistory Is there something about the time that interacts with Is there something about the time that interacts with

the treatment?the treatment?

2727

Why Generalizability is not always Why Generalizability is not always a problema problem

Experiments often are trying to isolate specific Experiments often are trying to isolate specific causes and effects in controlled settings. Thus, causes and effects in controlled settings. Thus, they may not even be claiming to be they may not even be claiming to be generalizable to specific settings.generalizable to specific settings.

Experimental findings can provide theoretical Experimental findings can provide theoretical basis for real-world tests. basis for real-world tests.

It is often a balancing act for research: true It is often a balancing act for research: true causation versus large-scale associational and causation versus large-scale associational and comparative testing.comparative testing.

2828

Considerations before using Considerations before using experimentsexperiments

Cost and EffortCost and Effort Is the effort worth it to test the concepts you are Is the effort worth it to test the concepts you are

interested in?interested in?

Manipulation and ControlManipulation and Control Will you actually be able to manipulate the key Will you actually be able to manipulate the key

concept(s)?concept(s)?

Importance of GeneralizabilityImportance of Generalizability Are you testing theory, or trying to establish a real-Are you testing theory, or trying to establish a real-

world test? world test?