how data mining contributes to efficacy studies and course redesign answering your questions: why?...
TRANSCRIPT
How Data Mining Contributes to Efficacy
Studies and Course Redesign
Answering Your Questions:Why? What? How?
Claire MassonDoug Paetzell Yun Jin Rho
Rasil Warnakulasooriya
24 Sept 2011
Tracking and Analyzing Student Data3
Your Comments from the pre-workshop survey…
1. Why are statistics useful in commenting on success – Show me some data on redesigned programs– How do I determine student success
2. What should we do after the redesign– How do I use the data more effectively– Show me ways to select the best data from pilot efforts
3. How should data be gathered to evaluate the redesign– What is the best way to collect data– What kind of data can be collected
Tracking and Analyzing Student Data4
How to Collect Data Doug & Claire
1. Setting Expectations– Determine goals for course redesign– Effects of increased rigor
2. Pilot the Program– Small program to full implementation– Pace of redesign– Gradual improvement
3. Review some Basic Statistics– Interchangeable learning aids– Why class size matters
4. Getting Started– Worksheets, Checklists, Templates
Tracking and Analyzing Student Data5
Setting Expectations
What are the specific goals of the course redesign?
1
Tracking and Analyzing Student Data6
First Step: Setting ExpectationsDetermine a Primary Goal
Set Quantifiable Expectations• set a specific goal to frame the redesign.
What is the problem we’re trying to solve?
Guiding questions– What percent do we want to increase student grades?– How many students do we want increase class size without
raising costs?– What qualitative effects do we want to see in our classroom as
a result of increased rigor?
Tracking and Analyzing Student Data7
The Flow of Redesign
Set
Goal
Evaluate
Resources
Design
Course
Select
Measurement
Tools
Implement
Course
Prepare
Data
Interpret
Data
Adjust
Course
Analyze
Data
Incr
ease
Lear
ning E
ffect S
ize b
y 0.5
Compute
r Lab A
vaila
ble? I
RB?
Emporiu
m M
odel
Compare
Fin
al Exa
m S
core
s
Compar
e His
toric
al /
Curren
t Exa
m S
core
s
Run Thru
Sem
este
r / G
ive
Last
Yea
r’s F
inal
Exa
m
Results: Statistically
Valid?
Results: Educationally Valid?
Apply Lessons Learned
Tracking and Analyzing Student Data8
Was this Redesign Successful?
TEACHER SAYS YES: “For probably the first time, all students are engaged in working on homework on a regular basis. The rigor of the homework assignment has increased, and even as we’ve implemented grading with no partial credit, success rates have increased in the course.”-Rebecca Muller, Mathematics Instructor
-Southeastern Louisiana University
Tracking and Analyzing Student Data9
Effects of Increased Rigor“The quality of the class experience has changed. Students come with constructive questions…Class time is more productive.”-Kathleen Almy, Associate Professor
Rock Valley College
“MyStatLab saves me from using class time to explain and re-explain how to solve problems. Because students are more prepared to learn and more proactive in their learning, I can convey more complicated, robust concepts to them. It makes teaching the course more fun to teach.”-Gwen Terwilliger, Ph.D., Professor Emeritus
-University of Toledo
Tracking and Analyzing Student Data10
Pilot the Program
The Gradual Process
2
Tracking and Analyzing Student Data11
The Road to SuccessPilot vs. Full Rollout
1. Importance of deadlines and benchmarks with assignments
— Teachers who monitor student participation have higher retention rates
2. Grade inflation may skew the effects
— Beginning phases of redesign may require remediation for students who were passed along previously
3. A good pilot often requires three phases of implementation to achieve success
Source: www.thencat.org
Tracking and Analyzing Student Data12
The Three Phases
Year 1: Develop and aggregate course material
Year 2:
— 1st Semester: Course development
— 2nd Semester: Campus pilots
— 2nd Semester: Course revision
Year 3:
— 1st Semester: Campus full implementation
— 2nd Semester: Convert course material for full campus
— 2nd Semester: Develop customized best practices plan and rework curriculum
www.thencat.org
Tracking and Analyzing Student Data13
Examples of Gradual ImprovementEffect of Course Redesign on Reducing Copying Over Time
—Decrease in copy rate over the four courses
—Decrease between Year 0 and Year 1 due to studio format redesign
—Decrease from Year 2 to Year 3 due to assigning to ABC grades instead of pass-no pass record
Traditional Year 1
Pilot
Year 2
Full Course
Year 3
Full Course
Palazzo/Lee/Warnakulasooriya/Pritchard “Patterns, Correlates, and Reduction of Homework Copying” Physics Education Research 6, 010104
(2010)
Tracking and Analyzing Student Data14
Examples of Gradual ImprovementEffect of Course Redesign on Improving Pass Rates Over Time
—Jackson State Community College conducted three pilots before full implementation
—Largest increased occurs after initial transition
—Continued improvement results from numerous adjustment
Tracking and Analyzing Student Data15
Review some Basic Statistics
Null Effect / Class Size / p-value
3
Tracking and Analyzing Student Data16
Similar Learning AidsStudy Design Yields Null Effect
Null Effects are NOT Negative.
Comparing one learning model to
another with the same intervention
goal, remediation, often yields
same results: null / no effect.
Tracking and Analyzing Student Data17
Same Content / Different PlatformStudy Design Yields Null Effect
Teacher assigned the same content, so there should be no expectation of improvement.
Mean exam scores with standard error bars for A&P (7 exams) using CC in 2010 vs. MAP in 2010 vs. MAP in 2011.
CourseCompass v Mastering for A&P
Tracking and Analyzing Student Data18
The Meaning Behind Class Size
MyITLab
233 124
p-value <0.05
statistically significant
But educationally significant?
Effect size: 0.44
p-value >0.05
not statistically significant
28 27 28 27 28 27
MyMathLab
67% 64%erro
r ba
rs o
verlap
erro
r ba
rs d
on’t o
verlap
# of students inside bars
Tracking and Analyzing Student Data19
Combining data and confounding factors
Combining data must be thoughtfully considered.1. Is it okay to combine your own sections of one class if the same
material is covered. (Consider student population v night / day classes)?
2. Is it okay to combine your student data with a colleague at a different institution (CC and 4-year research schools), administering different exams, etc.?
3. Is it okay to combine data with other instructors at your school teaching the same course?
Tracking and Analyzing Student Data20
Getting Started
Worksheets, Checklists, Templates, Examples
4
Handouts
Checklists Worksheets Templates
Learn more…
Case Studies: • www.mylab.com• www.masteringX.com
White Papers:• (Math): Making the Grade• (English): Vision in Action• (Sci/Eng): Make Learning Part of the Grade
Peer-Reviewed Journal Articles:• (MATH): Brewer/Becker “Online Homework Effectiveness for Underprepared and Repeating College Algebra Students,” Journal of Computers in Mathematics and Science Teaching 29(4), 353-371 (2010) • (BIO): Rayner: “Evaluation and Student Perception of MasteringBiology as a Learning and Formative Assessment Tool in a First Year Biology Subject ATN Assessment Conference (2008)• (PHYS): Palazzo/Lee/Warnakulasooriya/Pritchard “Patterns, Correlates, and Reduction of Homework Copying” Physics Education Research 6, 010104 (2010)
Thank you