experimental, quasi experimental, single-case, and internet-based researches in education
DESCRIPTION
TRANSCRIPT
Experimental, Quasi-experimental, Single-Case Research and Internet based experiments And Article CritiqueHatice ÇİLSALAR-Yelda SARIKAYA-ERDEM
Experimental ResearchDefinition: Testing an idea to determine whether it influences an outcome or dependent variable.
Key Characteristics: Random Assignment: Process of assigning individuals at random to groups or to
different groups Control over Extraneous Variables: Controlling influences of selection of
participants, the procedures, the statistics, the design likely to affect the outcome.
Pretest-posttest, covariates, Matching Participants, Homogenous samples, Blocking variables Manipulation of Treatment Conditions: Steps-Identify a treatment variable
and its levels or conditions, manipulate treatment conditions Outcome measures: Dependent variable that is the presumed effect of the
treatment variable. Group comparisons: Obtaining scores for individuals or groups on the
dependent variable and comparing the means and variance between the groups. Threats to validity: History(time passes), Maturation, Selection, Mortality,
Interaction, Testing, et.Cresswel, (2014); Frankel, Wallen & Hyun, (2012)
Experimental Research
A ‘true’ experiment includes several key features:
one or more control groups
one or more experimental groups
random allocation to control and experimental groups
pretest of the groups to ensure parity
post-test of the groups to see the effects on the dependent variable
one or more interventions to the experimental group(s)
isolation, control and manipulation of independent variables
Cohen, Mannion, & Morrison (2007)
Experimental Research
Cresswel, (2014); page:334
Experimental Research
Muijs(2004); page:334
How to Design an Experimental Research
Define your research objectives
Formulate hypotheses: H0 and H1
Set up your research design
Select instruments
Select appropriate levels at which to test your hypotheses
Assign persons to groups randomly
Carry out the experiment meticulously
Analyze the data
Experimental ResearchTrue Experimental Designs: 7Pretest-Posttest Controlled Experimental Group
Design
Two control Groups and One Experimental Group Pretest-Posttest Design
The Posttest Control-Experiment Group Design
Cohen, Mannion, & Morrison (2007)
Experiment group
R(Random Assignment)
O1(Observation)
X (treatment)
O2
Control group
R O3 O4
Experiment R O1 X O2
Control R O3 O4
Control R X O5
Experiment R X O1
Control R O2
Experimental ResearchThe Posttest Two Experimental Group Designs
The Pretest-Posttest Two Experiment Groups Design
Matched Pairs DesignFactorial Design
Cohen, Mannion, & Morrison (2007)
Experiment R X 1 O1
Experiment R X 2 O2
Experiment
R O1 X 1 O2
Experiment
R O3 X 2 O4
Low Receive Health Lecture
Smoking Number
Medium Receive Health Lecture
Smoking Number
High Receive Health Lecture
Smoking Number
Low Receive Standard Lecture
Smoking Number
Medium Receive Standard Lecture
Smoking Number
High Receive Standard Lecture
Smoking Number
Experimental ResearchParametric Design
Repeated Measures Design
Cohen, Mannion, & Morrison (2007)
Poor Readers Token Number of Correct Word
Average Readers
Token Number of Correct Word
Good Readers Token Number of Correct Word
Outstanding Readers
Token Number of Correct Word
Control Number of Correct Word
G1 O X1 O X2 O X3 O
G2 O X2 O X3 O X1 O
G3 O X3 O X1 O X2 O
G4 O X2 O X1 O X3 O
G5 O X3 O X2 O X1 O
Experimental Research
Poor Experimental Designs:
One-shot Case Study
One-Group Pretest-Posttest Design
The Static-Group(Non-Equivalent)
Comparison Design:
The Static-Group(Non-Equivalent)
Pretest-Posttest Design:
Frankel, Wallen, & Hyun, 2012
X O
O
O X O
X O
O X O
O O
Experimental ResearchTrue Experimental Designs:
The Randomized Posttest Only Control Group Design
The Randomized Pretest-Posttest Only Control Group Design
The Randomized Solomon Four Group Design
Random Assignment with MatchingFrankel, Wallen, & Hyun, 2012
Treatment R O X O
Control R O O
Treatment R X O
Control R O
Treatment R O X O
Control R O C O
Treatment R X O
Control R C O
Gall, Gall, &Borg, 2003
Experimental Research
Single Group Designs
The One-shot Case Study
One group pretest-posttest design
Time series designs
Control Group Design with Random Assignment
Pretest-posttest control group design
Posttest only control group design
One-variable multiple condition design
Creswell (2014)
Experimental ResearchBetween Group Designs
True experimental design: (Randomized)Pretest-Posttest design or Posttest only design
Quasi experimental design: (Un-randomized)Pretest-Posttest design or Posttest only design
Factorial design
Within Group/Individual Designs
Repeated measures design: Interrupted(One experiment) or Equivalent (More than one experiment)
Single subject designs: Multiple baseline design or Alternating treatments
Experimental Research
Strengths:
Causality: The best type for testing hypotheses about cause-and-effect relationships
Manipulation of independent variable
Help to see whether the treatment made difference.
Go beyond description and prediction, beyond the identification of relationship-what causes them.
Frankel, Wallen & Hyun, (2012)
Experimental Research
Limitations: Difficult to
Control some variables
Address all threats
Ethical issues: Control group may be disadvantaged by not receiving treatment or vice versa.
Quasi-experimental
“quasi” means, in essence, “sort of.” = quasi-experiment is a “sort of” experiment.
Definition: A quasi-experiment is a study that includes a manipulated independent variable but lacks important controls (e.g., random assignment), or a study that lacks a manipulated independent variable but includes important controls. Includes nonrandom assignment-matching.
More threat to internal validity: maturation selection, mortality, interaction of selection, history, testing, instrumentation, regression- Cresswell (2014)
Quasi-Experimental Research
Muijs(2004); page:334
How to Design an Experimental ResearchDefine your research objectivesFormulate hypotheses: H0 and H1Set up your research designSelect instrumentsSelect appropriate levels at which to test your
hypothesesAssign persons to groups randomly (only
experimental design) Carry out the experiment meticulouslyAnalyze the data
Quasi-experimentalTypes:
A Pre-experimental Design: The one group pretest-posttest O1 X O2
A Pre-experimental Design: The one group posttest only design X O1
A Pre-experimental Design: The posttests only non-equivalent groups design
A Quasi-experimental design: The pretest-posttest non-equivalent groups design
Experimental O1 X O2
Comparison O3 O4
The One Group Time Series
Quasi-experimental
Cresswell (2014)
Single-Case Research- Definition
Key Features:Single - one subjectStandard conditionsRepeated measurementEffectiveness or productivity
Three components:(a) repeated measurement, (b) baseline phase, and (c) treatment phase.
alternative to group designs. Alberto& Troutman, 1995;Best& Khan, 1998,Tekin (2002),
Group designs compare the performance of one sample of individuals (e.g., people who don’t smoke, or rabbits who don’t have smoke blown into their cages) with another (e.g., people who do smoke, or rabbits who do have smoke blown into their cages).
Single-subject designs compare the performance of an individual before and after a specified intervention.
Fre
quen
cy o
f di
srup
tions
Day
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
8
7
6
5
4
3
2
1
0
Baseline Praise
Dependent measure
Ordinate
Abscissa
Measure of timeUnit of time
Condition identifications Independent variable
Data pointsData path
Condition change line
Regardless of the research design, the line graphs used to illustrate the data contain a set of common elements.
A-B Design
Single-Case Research- Types
A-B-A-B Designs: Reversibility-last experimental control or no functional relationships
(Choen, Mannion, & Morrison, 2007; Kennedy, 2005)
Number of fulfilled assignments and without token(A) andtreatment with tokens(B).
Single-Case Research- Types
B-A-B Designs: an intervention already placed
Sometimes an individual’s behavior is so severe that the researcher cannot wait to establish a baseline
Or an intervention already placed so researcher must begin with an intervention. In this case, a B-A-B design is used. The intervention is followed by a baseline followed by the intervention.
Fre
quen
cy o
f di
srup
tions
Day
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
8
7
6
5
4
3
2
1
0
Praise Baseline Praise
B A BSingle-Case Research-
Types
Kennedy, (2005)
Single-Case Research- Types
A-B-C Designs: additional opportunity to analyze how various interventions influence behaviors
Earns Candy
Earns Money
Single-Case ResearchInstructor feedback
Peer feedbackA-B-C Designs:
Single-Case Research
StrengthsResearcher can establish a cause-and-effect relationship between treatment and behavior using only a single participantSee the effect of a treatment on a single participantFlexibility – development of the design depends on participant’s responsesBy using comparative designs, compare and contrast the results of the studies easily
Single-Case Research
LimitationsProblem with generalizations since designs use only one participantMultiple observations can affect participant’s responsesAbsence of statistical controls and reliance on visual inspection of the data
Internet based experiments
Three data collection method through Internet;
Nonreactive data collection
Online Surveys
Web based experiments (Reips,2002)
Internet based experiments
Why?
Speed,
Low cost,
Experimenting around a clock,
A high degree of automation of the
experiment, a wider sample.
Large number of participants
High statistical power
Protection of anonymity
Huge representativeness
Reips (2002)
There is little evidence in the literature that Internet-based surveys achieve higher response rates, as a general rule, than conventional surveys
Cohen,
Internet based experiments
Form of emails to emails-plus-attachments of the questionnaire itself, to emails directing potential respondents to a web site, or simply to web sites.
Although email surveys tend to attract greater response than web-based surveys, web-based surveys have the potential to reach greater numbers of participants
Page layout options should be simple not advanced
Avoid open-ended questions not to distrupt participants attention
Confirming of each item can be difficult for those who have less developed computer skills.
Keep the introduction to the questionnaire short (no more than one screen), informative (e.g. of how to move on) and avoiding giving a long list of instructions.
Keep the response categories close to the question
Internet based experiments
Advantages:
Ease access to a large number of demographically and culturally diverse participants
Specific participant population
Better generalizability of findings to population, more settings or situations
Avoidance of time constrains, organizational problems: scheduling difficulties, as thousands of participants may participate simultaneously
Highly voluntary participation
High participation: High statistical power
Detectability of motivational confounding
Reduction of experimenter effects, demand characteristics
Cost saving of personnel hours, equipment, administration
Greater openness of the research process
Access to the number of nonparticipants
Ease access for participants
Public control of ethical issues
Reips (2002)
Internet based experiments
Disadvantages:
Possible multiple submission: warning about multiple submission, blocking using same IP address, or handing out passwords-one time password, participant pool or online panel, control by collecting personal identification, controlling internal consistency
Self-selection: can be controlled by using the multiple site entry technique.
Dropout: Promising immediate feedback, giving financial incentives, by personalization
Misunderstood instructions: Pretest of materials and providing the participants with the opportunity for giving feedback
The comparative basis is relatively low.
External validity is limited by their dependence on computer
Reips, (2002)
Cohen
Internet based experiments
Dillman et al. (1999) three ways to overcome problem of differential expertise in computer usage:
having the instructions for how to complete the item next to the item itself at the start of the questionnaire
asking the respondents at the beginning about their level of computer expertise, and, if they are more expert, offer omitting instruction part and, if they are less experienced, directing them to instructions
having a ‘floating window’ that accompanies each screen and which can be maximized for further instructions.
Internet based experiments
Reips, (2002)
Internet based experiments 16 Standards:
1. Consider to use web-based software tool to create survey
2. Pretest the instrument for clarity of instructions availability on different platforms
3. Make a decision about advantages out-weigh the disadvantages
4. Check your web survey for configuration errors
5. Consider multiple site entries
6. Run survey both online and offline for comparision
7. If dropout is to be avoided use the warm-up technique
8. Use dropout to determine whether there is motivational confounding
Reips, (2002)
Internet based experiments 16 Standards:
9. Use high-hurdle technique, incentive information
10.Ask filter questions at the beginning of the experiment to encourage serious and complete responses.
11.Check for obvious naming of files, conditions, passwords
12.Use , if needed to avoid multiplication, participant tools or password techniques
13.Perform consistency checks
14.Keep logs
15.Report and analyze drop out rates
16.The experimental materials should be kept available on the Internet, as they will often give a much better impression of what was done than any verbal description could convey.
Reips, (2002)
ReferencesCohen, L., Manion, L., & Morrison, K. (2013). Research methods in education. Routledge.
Creswell, J. W. (2014). Educational research: Planning, conducting and evaluating, quantitative and qualitative. Pearson International Edition.
Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education. McGraw-Hill International Edition.
Gall, M. D., Gall J. P. & Borg, W. R. (2003). Educational research: An introduction. Pearson.
Kennedy, C. H. (2005). Single-case designs for educational research. Financial Times/Prentice Hall.
Reips U. D. (2002). Theory and techniques of web based experimenting. In B. Batinic, U.D. Reips, & M. Bosnjak (Eds.) Online Social Sciences. Seattle Hogrefe & Huber.
Reips, U. D. (2002). Standards for Internet-based experimenting. Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie), 49(4), 243-256.
Tekin, E. (2000). Karşılaştırmalı tek denekli araştırma modelleri. Özel Eğitim Dergisi, 2(4), 1-12.
