emil experimental methods in linguistics halszka bąk & rafał jończyk faculty of english, adam...
TRANSCRIPT
EMiLExperimental Methods in Linguistics
Halszka Bąk
& Rafał Jończyk
Faculty of English, Adam Mickiewicz University
yellow black
worp vs warp
Research origins
Linguistics: magpie research methods.
Psycholinguistics: psychology.
Neurolinguistics: psychology and neuroscience.
Experimental tasks
Lexical Decision task
CatLafilAim: ortography and semantic processing
Presentation duration: 20ms, 150ms, 200ms, 250ms?
Stimuli placement: centre of the screen? Periphery?
depends on your goals!
?
Semantic Decision task
DoctorHospital?DoctorRose?
The coffee was too hot to… DRINK
The coffee was too hot to…EAT
AIM: Processing of meaning/semantics
Word/Picture naming task
Stroop task
RedBlueGreenAim: investigating interference in the RTs of a task; conflict between word meaning and its colour
Emotional Stroop task: captures attention and RTs to emotional vs non-emotional stimuli
Name the colour:death vs idea
Discrimination task
Which face is more happy?
Failla et al. 2003
Dichotic listening task
Aim:
1. selective attention;
2. hemispheric lateralization of sounds and perception
Free-word recall
Word recognition
Dog leaf
yellow leaflet
punk green
cat skin
dead ideal
PNJA death
nurse hospital
workshop doctor
house stick
dog leaf
yellow leaflet
punk green
cat skin
dead ideal
PNJA death
nurse hospital
workshop doctor
house stick
Task 1
tasksheet, page 2 & 3
Types of data &
methods of interpretation
Quantitative data
Reaction/Response Times (RTs):
measured in miliseconds (ms)
influenced by: priming, emotional states, attention, etc.
Quantitative data
Accuracy rates (ACC) or Error rates (ER):
measured in 1s (correct) and 0s (incorrect)
influenced by: bilingualism, fallacies, cognitivedeficits, etc.
Qualitative data
Facial expression
Prosody (speech contour)
Gestures
Body language
Etc.
Physiological data
Galvanic Skin Response (GSR; also Skin Conductance Response (SCRs))Measures skin conductivity in microSiemens (mS)
influenced by: stimulus modality, intensity, etc.
Physiological data
Heart rate (HR)
measured in beats per minute (BPM)
influenced by: stimuli intensity, participant’s health, etc.
Physiological data
EEG: Event-Related Potentials (ERPs):
measured in microVolts / sec
unrivalled temporal rsolution
Physiological Data
Functional Magnetic Resonance Imaging (fMRI):
blood / water diffusion moleculesmeasured in voxels
great spatial resolution
poor temporal resolution
!!!
Your HYPOTHESES determine what TYPES OF DATA you will COLLECT
Task 2
tasksheet, page 3
Research Questions & Hypotheses
Research question
frames a problem to be addressed & resolved in research
Hypothesis
A prediction about a relationship between
Two (or more) variables
A solid hypothesis is…
...TESTABLE
confirm or falsify the hypothesis
…SUPPORTABLE
provide empirical evidence to confirm/falsify the hypothesis
…GROUNDED
ground your hypothesis in theory/previous research
…RELEVANT
fill a gap in the research, resolve inconsistency, etc.
Special case…
…NULL HYPOTHESIS - Nº
No relationship between the variables
It is always there
fail to reject Nº = cannot draw conclusions from your research
Task 3
tasksheet, page 4
Variables
A variable is…?
…something that changes or varies
e.g. among people (IQ, gender, etc.)
Independent variable
the aspect you manipulate to influence the end results
e.g. instruction, types of stimuli, types of participants
priming, etc.
What is the cause of a given phenomenon?
the cause = independent variable
Dependent variable
the end result (hard data)
e.g. RTs, ACC, ER, ERPs, etc.
e.g. the amount of time you choose to spend doing grammar exercises determines your final score on the PNJA exam in June ;)
summing up
HYPOTHESIS: Positive sentence context makes processing of positive words faster and more accurate.
PREDICTION: faster & more accurate processing of ”+” WORDS in ”+” SENTENCE context.
INDEPENDENT VARIABLE: sentence context, word valence.
DEPENDENT VARIABLE: RTs, ACC/ER
TYPE OF RELATIONSHIP: ”+” SENTENCE causes smaller RTs and greater ACC in responses to ”+”WORDS.
Task 4
tasksheet, page 4
Experimental Design
The simple experiment
1. Define hypothesis
2. Specify and define variables.
3. Participants: experimental vs. control groups.
4. Random assignment of participants to groups
5. Collect dependent variable
6. No significant result – no effect, no interpretation
if you cannot randomly assign participants to your different groups, you cannot do a simple
experiment
2 x 2 design
2 independent variables & 2 levels of each variable
(positive; negative) x (congruent; incongruent)
Gloria accidentally poured boiling water over herself and was BURNT / ROMANTIC*
Their honeymoon in the gorgeous scenery of Paris was very ROMANTIC / BURNT*
Reliability & Validity
Validity
Internal
External
Ecological
Internal validity
A B
Your data reflects the reality under investigation:
independent variables and their manipulation
Threats to internal validity
1. Testing: pre-tests may cause biases
2. Instrumentation: pre- and post-tests must match
3. Regression: biased participant selection
4. Confounding variable:
not properly controlled, manipulated or unnoticed
External validity
Results can be generalized to the wider population
Threats to external validity
1. Participant selection: atypical population sample
2. Setting: the context
3. History: a specific historical context
4. Construct effect: a particular construct exists in one group but not in others
Ecological validity
Experiment conducted in an environment
& in circumstances most similar to or exact to a real-life
situation
Reliability
A reliable experiment can be reliably replicated yielding the same results
Task 5
tasksheet, page 5
Errors & biases
Errors
Murphy’s law: Anything that can go wrong will go wrong
Type 1 error:
You’ve done everything right. Got a significant result. The result is statistically a chance accident.
Type 2 error: You’ve done everything right. DID NOT GET A SIGNIFICANT RESULT. The lack of result is statistically a chance accident.
Demons you can fight:Random errors & biases
Everything done right.
No risk of Type1 or Type 2 error…
Still…the result is WRONG
The result is a chance accident due to a randomly occuring external factor…
This is why you need reliable and replicable experiments!
Solution?
Fight! Increase the number of participants
(random error tends to balance out to zero)!
Biases
2 major offenders, 1 major circumstance
Offender 1
Researcher:
”I’m going to get what I expect & publish it in SCIENCE!”
“It is remarkable how often the first interpretations of new evidence have confirmed the preconceptions of its discoverer."
(John Reader. 2011. Missing Links. Oxford University Press)
Offender 2
Participant A:
”I’ll make you look good/bad!” (obeying demand characteristics)
Participant B:
”I’ll make myself look good!” (social desirability bias)
Non-mitigating Circumstance 1
Day 1:
You’re taking part in an important experiment…
Day 2:
So yeah… this… this thing yeah it’s for my thesis and um…
(Check out Mitchell and Jolley. 2010. Research Design Explained. Seventh edition. Pages 129-143)
Task 6
tasksheet, page 1 & 2
Cause and effect in experimental research
(correlation vs. experimentation)
“Correlation does not imply causation”
Correlation Experiments
Reality observed and measuredmanipulated and measured
Fallaciescum hoc ergo propter hoc(no causality implied in correlated observations)
post hoc ergo propter hoc (when internal validity is poor)
Validity Ecological Internal & external
Task 7
tasksheet, page 6
Original research vs.
Replicating experiments.
Original Replication
Point of origin Theory Existing research
ApproachTheory-based, practically creative
Data-driven, practically repetitive
MethodsSelect from all available and viable
Only choose the one used in the study being replicated
Modifications Compromise validityIncrease validity provided that control group is included!
Stimuli & Participantselection
Participants
Ideal case:
a large randomly selected sample from a homogenous population.
Typical case for MA/BA: a minimally valid, inherently somewhat biased sample
Remedy:
a careful selection of hypothesis, inclusion of relevant control pre- and post-tests.
StimuliA happy case:
stimuli exist, have been validated in a thorough and correct validation procedure, and they are available.
An unhappy reality:
a norming study is an absolute, inevitable MUST.
An unhappier reality still: you must create, validate, and norm your own stimuli.
Remedy:
Meta-analyses of existing research (more later), start early.
The experimenter’s conduct
Beware!
Biases
Errors
Self-presentation
Self-care
Ethical considerations
Human Subject Research(HSR)
Linguistic research is a case of HSR.
Seek permissions to proceed with Internal Review Boards (IRBs) and Ethical Committees.
PhD and senior staff:
strongly advised to take a basic HSR course. E.g. CITI course (http://www.citiprogram.org)
Rule of thumb
Don’t hurt people. Emotionally, psychologically, physically.
Tools & Resources
Meta-analytical tables
ask a single study design question, and search through the top-cited papers in the field to see how others have “done it”.
Example:
Question: How many participants should I use?
Literature: Find top-cited 30-50 papers from the past 10 years on the subject.
Table: Bibliography note, participants number, relevant participant variables.
What to do: abstract > introduction (skim) > methods > participants
Result: Overview of the typical number and type of participant in your discipline.
Existing resources
Affective Sciences databases: SEMAINE, BINED, ILHAIRE, SAVEE, ANEW, IADS, IAPS, etc.
Online corpora: COCA, BNL.
Online services: instant.ly, Monkey Surveys, Google Docs, etc.
Existing experiments for replication: e.g. Englelab (http://www.englelab.gatech.edu).
Linguistlist: http://linguistlist.org/
Courses
Coursera: Free of charge.
MIT courses: Free of charge (Coursera, Edx)
CITI courses: Standard Fee 100$. Requirements: Affiliation with a US Institution (https://www.citiprogram.org/)
Bibliography
Field, Andy. 2009. Discovering statistics using SPSS (and sex and drugs and rock’n’roll). Third edition. London: SAGE.
Mitchell, Mark L. and Janina M. Jolley. 2010. Research Design Explained. Seventh edition. Belmont: Wadsworth CENGAGE Learning.
Further recommended reading
Feynman, Richard. 1974. Cargo Cult Science. Caltech commencement address. (http://neurotheory.columbia.edu/~ken/cargo_cult.html) (Date of Access: 02.04.2014)
Feynman, Richard. 1985. Surely you’re joking, Mr. Feynman! Adventures of a curious character. New York: W. W. Norton.
Feynman, Richard. 1988. What do you care what other people think? Further adventures of a curious character. New York: W. W. Norton.
Sagan, Carl. 1995.The Demon-Haunted World: Science as a Candle in the Dark. New York: Random House.