Talking Theory

Five Perspectives on Theory Construction in Psychology

Voodoo Correlations are Everywhere—Not Only in

NeuroscienceKlaus Fiedler, University of

Heidelberg

Voodoo Correlations in Neuroscience

• Or “Puzzlingly High Correlations,” at least (Vul, Harris, Winkielman, & Pashler, 2009)

• Selecting voxels based on thresholds leads to distorted estimates of correlations

Other Research Biases—Study design

• Sampling stimuli– Generally an intuitive process; researcher

implicitly simulates what is expected to be observed

– Random sampling of stimuli can eliminate some effects, ex. Overconfidence & trivia questions (e.g., Gigerenzer, Hoffrage, & Kleinbölting, 1991)

• Pilot testing of stimuli/tasks– X values are auditioned to correlate with Y

Other Research Biases—Selecting Variables

• Dependent variables– What is chosen to be studied affects theoretical

perceptions• Ex. TV & aggression

• Sampling levels of IVs– Magnitude of ΔY is easily manipulated by ΔX– Effect sizes are useful, but can also be misleading

Other Research Biases—Analysis

• Mediator variables– Trifling mediators (Y’) can be found if they are

correlated enough with Y• Moderator variables– Strong moderators can be overlooked or assumed in

certain paradigms, overestimating actual effects• Publishing– Weak findings will be “optimized”– Analyses will be re-run until they are “successful”– File-drawer problem

Fiedler: Conclusions• Aim for representative designs (Brunswick, 1955)

– When that fails, treat stimuli as random factors, as appropriate, to enhance external validity

• Preplanned comparisons• >2 levels of IV• Convergent validation

– Multiple methods to converge on the existence of an effect– Usually underestimates the true effect size, but can provide

a lower limit• Study boundary conditions

– Can reveal important moderators, enabling conditions– Or rather, make implicit moderators explicit

The Problem of Circularity in Evidence, Argument, and

ExplanationUlrike Hahn, Cardiff University

Question

• How are we to effectively argue in psychology?

Circular Argumentation

• Reassertion: ex. “God exists, because God exists.”– A because A

• Clearly convertible: ex. “Opium induces sleep because it has a soporific quality.”– A because A’

• Self-dependent justification: ex. “God exists because the bible says so, and the Bible is the word of God.”– A because B; B because A

Circular Argumentation• Is it invalid?

– No; on the contrary, it’s always valid• Is it persuasive?

– Sometimes yes, actually– Reassertion and convertible cases are not very persuasive; miss

the point of argumentation: belief change– Self-dependent case is frequently used in hypothesis testing: ex.

“Electrons exist because their signature effects can be seen in cloud chambers”• Modus ponens argument: “if P, then Q”; falsifiable, but not entirely

verifiable– Persuasiveness of argument rests on conditional probabilities

[What’s your P(e|~h)? Eliminating alternative hypotheses]

3 Domains of Circularity

• 1: Circular Explanations– Instead of explaining, merely re-describes the

observed phenomenon– Gigerenzer (2009) makes grand accusations

against much of psychology• Matching bias in Wason selection task is a result of

greater “transparency” – but what does that mean?

Levels of Explanation

Explanation / MechanismExplanation / Mechanism

TheoryTheory

DataData

3 Domains of Circularity• 2: Circular Analysis– Voodoo correlations—practice of pre-selecting data– Is it circular?

• No: not technically an argument• No claim nor evidence, but bias introduced to data• Presents a limit on the “interestingness” of findings

– Is it wrong?• Not really… but it’s not right, either; why bother?• Inferences based on this data are statistically invalid

– Consider model fitting• Model is constructed to accommodate data• Evaluating freedom of model (i.e., minimizing parameters) helps

to guard against false positives

3 Domains of Circularity

• 3: Circular Methods– Refers to Fiedler (2011)– Circular? Not technically, but significant concerns

about “interestingness”

Hahn: Conclusions

• Q: Is psychology a circular science?– A: No; at least, not more than any other science.

• Q: Why write a paper about circularity in psychology when there is none?– A: To be pedantic.

• Be aware at what level you are speaking/thinking about a phenomenon

Much Pain, Little Gain? Paradigm-Specific Models and Methods in

Experimental PsychologyThorstein Meiser, University of

Mannheim

Using Paradigms

• Paradigms can be useful, but they they can also hinder scientific progress

• Wason card selection task was developed to study deductive reasoning, but turns out people don’t reason deductively with it– Inappropriate for studying deductive reasoning,

but fecund paradigm for other ideas

Variables & Constructs• Manifest X & Y variables in a given study map

onto latent constructs ξ & η• Considering a given paradigm as being sui generis

can divorce the real constructs– Example: DRM paradigm purports to be an instance of

false memory, but is also an instance of category learning (e.g., Hintman, 1986)

• Need to bridge gaps in literatures, connect disparate literatures– Like Bobbie Spellman did with category induction &

persuasion (Ranganath, Spellman, & Joy-Gaba, 2010)

Meiser: Conclusions

• We should aspire to whittle down the universe of plausible explanations and constructs, not add to it

• Relying on traditional paradigms (and accounts of paradigms) can obfuscate further knowledge to be learned from them

• Paradigm-independent methodologies, such as signal detection theory, are advocated

Friends and Foes of Theory Construction in Psychological

Science: Vague Dichotomies, Unified Theories of Cognition, and the New

ExperimentalismLeonel Garcia-Marques & Mario B.

Ferreira, University of Lisbon

Dichotomous Theories

• Competing choices– ex. Modal vs. amodal conceptualization, analogical vs.

propositional representation– In reality, two out of an infinite possibility of theories

(e.g., Goodman’s (1946) concept of “grue”)• Textbook experiments—illustration of a particular

theory, leave aside complications, exceptions, etc.• Problem of theoretical mimcry—same result can

be accounted for by modification of alternate theory

Unified Theories of Cognition

• Attempts to explain cognitive architecture at a grand scope– John Anderson’s ACT & ACT-R, SOAR, etc.

• Avoid problems of more specific theories, but suffer their own problems– Task theories—how is the specific instantiation

represented?– Parameter overfitting—what can explain

everything explains nothing

The New Experimentalism• Dichotomies present the possibility of unproductive

theoretical mimics• UTCs are limited by parameter fitting and task theories• Good experiments:

– Lead to productive errors (effetive error-correcting mechanisms)

– Results are only weakly theory-dependent– Tested hypotheses are easily falsifiable– Severe tests of specific hypotheses (low false positive rate)

• The results of good experiments can effectively serve to limit the search space of possible theories

• More novel results must be accommodated by new theories, reducing theory glut

Garcia-Marques & Ferreira: Conclusions

• Generate more useful hypotheses• Have >2 levels of IV• Describe data better; ex. Exploratory analyses,

robust descriptive statistics• Use effect sizes• Test alternate accounts

Why the Cogntive Approach in Psychology Would Profit From a

Functional Approach and Vice VersaJan De Houwer, Ghent University

Extrapolation from Behavior• The cognitive perspective aims to uncover the mental

operations that are responsible for behavior• Behavioral effects are often taken as proxies for mental

events– Ex. Classical conditioning paradigm is taken to mean formation

of association between stimuli in memory• Mental construct is not a necessary component; alternate

causes could explain effects at least part of the time• Could take a tentative view; what’s the harm?

– Researchers could take construct for granted, become conditioned and shut out alternate explanations

– Theories that are invested in particular behavioral effects could be massively disrupted if they are found to be poor proxies

Questions

• Does this entail a difference in how we talk about experiments, or about how we think about them? (Both)

• Is this an argument for a stricter separation between experimentation and theorizing? (Yes)

De Houwer: Conclusions• Don’t skip steps in your explanations

– Before you propose a mental construct as the explanans of a behavioral effect, be sure you know what the environmental explanans is

– This goes double for imaging studies• Five suggestions:

– Use separate functional and mental terms– Define behavioral effects by causal impact of specific

environmental stimuli on specific aspects of behavior– Don’t treat behavioral effects as proxies of mental constructs– First describe empirical findings in functional terms; should be

abstract and general– Research should be directed to advance both mental and

functional explanations