Graziano and RaulinResearch Methods: Chapter 8

Research Methods:Validity and Threats to Validity

Generating Research Hypotheses(Review)

Ideas lead toobservationslibrary research

Statement of problemProblem statements

become research hypotheses when constructs are operationalized

Characteristics of a Good Study Question

“FINER”F Feasible

I Interesting

N Novel

E Ethical

R Relevant

Actually testing three sets of hypothesesThe null hypothesisThe confounding variable hypothesesThe causal hypothesis

Accept causal hypothesis only if youreject null hypothesis (statistical analysis)rule out each potential confounding variable

hypothesis (based on appropriate controls)

Testing Research Hypotheses

Correlation (also called covariation)Relationship found between variables

Time orderCause must occur before result

NonspuriousnessAlternative explanations must

be eliminated from possibilityExperiments are intended to

reduce or rule out alternative explanations and confoundingvariables

Criteria for Nomothetic Causality

Asking “the Question”The PICO format:

P PopulationI Intervention or Interest areaC Comparison intervention or statusO Outcome

Applying the “PICO” format“What is the usefulness or accuracy of the

current 1-10 pain scale assessment in treating a patient’s pain, and what are other options that may prove more useful?”

Does a 10 point pain Visual Analog Scale (____, ____) accurately assess pain in the first day postop abdominal total hysterectomy patient when compared with the Faces Pain Scale (Pasaro, 1997)?

Evaluating Hypotheses

Statistical Validity – carrying out the actual statistical analysis properly

Construct Validity refer most often to a characteristic of an instrument but also to the whole study

External Validity refers to the generalizability of study findings

Internal Validity refers to a characteristic of a study’s design

Types of Validity

Are the statistical tests accurate?

Threatened byUnreliable measuresViolations of statistical assumptions How do we detect these problems???

Strengthened byUsing well validated measuresHaving approximately equal sample sizes in

each group)

Statistical Validity

Is our theory the best explanation for the results?

Threatened byAny alternative explanation for the resultsHOW do we locate these alternative

explanations?Strengthened by

Using well-validated constructs to build the theoretical predictions for the study

Construct Validity

Do the results apply to the broader population?

Threatened byUnrepresentative samplesGeneralizing beyond the limits of the sampleHOW do we know when this problem is

present???Strengthened by

Gathering a representative sample (if possible)Clearly describing sample, so that other

researchers will know the limits of generalization

External Validity

Is the independent variable responsible for the observed changes in the dependent

variable?

Threatened by Confounding variablesHOW do we detect the presence of

confounding variables????Strengthened by

Adding adequate controls to reduce or eliminate confounding

Internal Validity

Confounding and internal validityMany sources for confounding (covered next)With proper controls, confounding can be

virtually eliminated (see Chapter 9)

Confounding and construct validityMake sure that you have considered alternative

theoretical explanations for the anticipated phenomenon

HOW????

Avoiding Confounding

Pretest-Posttest Research Design

Single-group, pretest-posttest design compares pre-treatment and post-treatment scores to determine improvement

Fails to control most sources of confounding

Historical events may occur during the course of the experiment.Remember Pygmalion effect & its story

Maturation of the subjects.Testing and retesting can

influence awareness of variables or behaviorLearn Hawthorne effect & its story.

Instrumentation – measurement methods or procedures may not be equivalent

Sources of Internal Invalidity

Statistical regression of subjects starting out in extreme positions.

Selection biases (we will see several types)

Experimental mortality(a.k.a. attrition) – subjects drop out of the study before it's finished.

Sequence effects – Performance on one measure is related to previous experience with other measures. Outcome depends on the sequence of measures.

Sources of Internal Invalidity

Demoralization subjects incontrol group find out, loseinterest in study, stop trying

Diffusion of treatment (those who get the experimental stimulus spread it to controls)

Rivalry (controls change behavior to try to beat the experimental group)

Equalization of treatment (researcher compensates controls for not getting treatment)

Social threats to validity

Social Psychology Experiment:Conformity to Norms (file)

Participants are not passiveThey try to understand the study to help them

to know what they “should do” (termed subject effects)

Respond to subtle cues about what is expected (termed demand characteristics)

Placebo effect: treatment effect due to expectations that the treatment will work

Subject Effects

Based on the expectations of the researcher

Can affect the outcome of studies if not controlled

May be due to the experimenter providing demand characteristics to the participant

Not the same as scientific fraud (which is deliberate)

Experimenter Effect

Three closely-tied concepts

Validity The accuracy of the study or procedureIncreased by using appropriate control

procedures

The more controls we employ, the higher the level of constraint

Controls may increase some types of validity while, by their unnatural aspect, decreasing other types of validity.

Validity, Control, and Constraint

Risk is balanced by rewardA poorly designed study will provide no useful

information; therefore, any risk would be unacceptable

Informed ConsentVirtually guarantees that you will have

confounding due to selection because some people will refuse to participate

A small price to pay to maintain ethical standards

Ethical Principles

Start by building a research hypothesisTesting the research hypothesis is actually

testing three hypotheses(1) null; (2) confounding-variable; (3) causal

Several types of validityMany potential confounding variablesSubject and experimenter effects can also

affect the outcome of the study

Summary