sotl: research design nancy gourash bliwise, phd step faculty development workshop emory university...
TRANSCRIPT
![Page 1: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/1.jpg)
SoTL: Research Design
Nancy Gourash Bliwise, PhDSTEP Faculty Development Workshop
Emory UniversityJune, 2014
![Page 2: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/2.jpg)
Research Design in SoTL
![Page 3: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/3.jpg)
Commitment to Outcomes Research
![Page 4: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/4.jpg)
Learning Objectives
Identify the basic elements of four common quasi-experimental designs used in outcomes research
Discuss strategies to “control” factors that might impact validity of conclusions drawn from designs
Be able to identify ethical issues in educational outcomes research and resources for IRB review
![Page 5: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/5.jpg)
Outcome Research is …
Important
Rewarding
Necessary
![Page 6: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/6.jpg)
And often…
Difficult
Time-consuming
Small effect sizes
![Page 7: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/7.jpg)
Type of Research
Descriptive/Qualitative
Question: What Is? descriptive, generative naturalistic,
observational, constructivist
Samples: small, targeted Assessment: interviews,
observations inductive formative, “thick
description”, expansive
Inferential/Quantitative
Question: What Works? hypothesis testing,
confirmatory empirical, statistical,
comparative Samples: large,
representative Assessment: scores, rates
deductive summative, precise,
reliable
![Page 8: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/8.jpg)
Formal Design
Focus of SoTL Actions as educators Student learning and performance
Question guides choice of design
Source Material: Cook & Campbell (1979) – Quasi-Experimentation:
Design & Analysis Issues for Field Settings Trochim (2006) – Research Methods Knowledge Base
Online: http://www.socialresearchmethods.net/kb/ NSF – User-friendly Handbook for Mixed Methods
Designs Online: http://
www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153
![Page 9: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/9.jpg)
Optimal Design -- Descriptive
No single preferred design
Common features Focus on individual and collective experiences of
students Context is described Search for patterns Developmental Multiple assessment methods and assessment
opportunities Subgroup analysis
![Page 10: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/10.jpg)
Optimal Design -- Inferential
![Page 11: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/11.jpg)
Randomized “Clinical” Trial
![Page 12: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/12.jpg)
Real Life
Quasi-experimental designs Non-random assignment to groups
Frequently used (and effective designs Single group repeated measures design
Non-equivalent groups pre-test/post-test design
Cohort design
Single group crossover (ABAB) design
![Page 13: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/13.jpg)
Uninterpretable Designs
One-shot case study
Single-group pre-test/post-test design
Static group comparison
X O
O1 X O2
X O O
![Page 14: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/14.jpg)
Single Group Repeated Measures/Time Series
Add multiple observations to track learning/change
Goal is to study the trajectory of learning Depth of knowledge Understanding of concepts Application Integration/synthesis Testing of ideas
O1 O2 X O3 O4 O5 O6
![Page 15: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/15.jpg)
Non-equivalent Groups Pre-test/Post-test Design
Compare two classes New vs. standard
Willing/interested colleagues Commitment to method
Pre-test/Post-test essential
Targeted measurement
O1 X O2
O1 O2
![Page 16: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/16.jpg)
Cohort Design
What if you are the only one who teaches this? Or colleagues are not “willing”
Compare outcomes to yourself But be very, very careful
Risk – nothing else can change Assumes stable cohort
O1 X
O2
![Page 17: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/17.jpg)
Single Group Cross-over Design
Multiple units/cases
Replicate (ABAB)
Standard assessment strategy
XA1O1O2O3 XB1O4O5O6 XA2O7 O8O9XB2O10O11O12
![Page 18: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/18.jpg)
Less “Control”
Rule out competing explanations of findings
Argument/Logic
Measurement Student demographics Possible pre-test differences Characteristics of “treatment”
Design
Statistical controls
Preventive action
![Page 19: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/19.jpg)
Sound Measurement
Existing standards
Independence of observations, where possible
Conceptual comparison
Multiple dimensions/skills
![Page 20: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/20.jpg)
Your Turn…
Choose design that best “fits” your case
Type of class
Possibility of comparison
Assessment strategy
![Page 21: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/21.jpg)
IRB Review
Educational research Human participants
Type of research Non-research/exempt/expedited
Primary Issues Coercion Informed consent
![Page 22: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/22.jpg)
Everything IRB
www.irb.emory.edu
Social-behavioral studies
Multi-disciplinary team Rebecca Rousselle Carol Corkran
![Page 23: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/23.jpg)
Non-Research
New category
Systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge
PI can make this determination
If it is just for your own evaluation, it is not research
![Page 24: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/24.jpg)
Exempt vs. Expedited
Most surveys, interviews, or observations of public behavior are presumed exempt if the participant cannot be identified (no links from the person to the data) or the responses/data can not harm the participant.
IRB determines this
![Page 25: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/25.jpg)
Coercion
Major issue
Students must be free not to participate
![Page 26: SoTL: Research Design Nancy Gourash Bliwise, PhD STEP Faculty Development Workshop Emory University June, 2014](https://reader035.vdocuments.net/reader035/viewer/2022062515/56649cf95503460f949c9e44/html5/thumbnails/26.jpg)
Informed Consent
Description of the purpose of the study
Who is being studied
What will be required
Risks/benefits
Compensation
Legal review
Contact information
Summary of findings