small liberal arts college: 1350 undergraduate students statistician within department of math,...
TRANSCRIPT
![Page 1: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/1.jpg)
But…does it work? Do students truly learn
the material better?
![Page 2: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/2.jpg)
Nathan Tintle, Dordt College
Small liberal arts college: 1350 undergraduate students
Statistician within Department of Math, Stat and CS
Class size: Stat 131 (30 students), 5-6 sections per year
3 hours per week in computer or tech-enabled classroom
![Page 3: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/3.jpg)
What we know about randomization approaches
What we don’t
What it means
Overview
![Page 4: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/4.jpg)
Tintle et al. flavor (2013 version)◦ Unit 1. Inference (Single proportion)◦ Unit 2. Comparing two groups
Means, proportions, paired data Descriptives, simulation/randomization, asymptotic
◦ Unit 3. Other data contexts Multiple means, multiple proportions, two
quantitative variables Descriptives, simulation/randomization, asymptotic
Our approach
![Page 5: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/5.jpg)
Qualitative
◦ Momentum: Attendance at conference sessions, workshops Publishers agreeing to publish the books Class testers/inquiries People doing this in their classrooms (clients,
colleagues) Repeat users
Appealing “in principle” and based on testimonials to date
What we know about it
![Page 6: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/6.jpg)
Quantitative assessment
Tintle et al. (2011, 2012)◦ Compare early version of curriculum (2009) to
traditional curriculum at same institution as well as national sample
◦ 40 question CAOS test◦ Results
Better student learning outcomes in some areas (design and inference); little evidence of declines
What we know about it
![Page 7: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/7.jpg)
Post-test answer
National sample
Hope -2007 Hope-2009
Small p-value 68% 86% 96%
What we know about it
Sample sizes: Hope ~200 per group; National Sample 760
P<0.001 between cohorts
Pre-test: 50-60% correct
Example #1. Proportion of students correctly identifying that researcherswant small p-value’s if they hope to show statistical significance
![Page 8: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/8.jpg)
What we know about it 2012-13 results
14 instructors, 7 institutionsTotal combined sample size of
783
![Page 9: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/9.jpg)
Instructor (Inst, Class size)
Pre-test Post-test Change Sample size
1 (LA, Med) 70% 97% 27% 33
2 (LA, Med) 73% 95% 22% 26
3 (Univ, Med) 23% 95% 72% 40
4 (LA, Med) 70% 96% 26% 127
5 (LA, Sm) 28% 92% 64% 11
6 (Univ, Med) 37% 96% 59% 49
7 (Univ, Sm) 39% 73% 34% 23
8 (LA, Med) 60% 97% 37% 35
9 (LA, Med) 29% 96% 67% 95
10 (HS, Med) 24% 74% 50% 38
11 (Univ, Large) 68% 97% 29% 101
12 (LA, Med) 63% 93% 30% 92
13 (LA, Med) 28% 95% 68% 18
14 (LA, Med) 56% 97% 41% 78
What we know about it
![Page 10: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/10.jpg)
Institutional diversity in student background (pre-test)
Post-test performance very good for most (over 90%)
A couple of exceptions◦ Both first time instructors with curriculum who will
use it again this year
What we know about it
![Page 11: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/11.jpg)
Example 1 (continued).
First quiz, 2.5 weeks into course; Simulation for a single proportion
119 people played RPS, 11.8% picked scissors
Evidence that scissors are picked less than 1/3 of time in long run?
What we know about it
![Page 12: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/12.jpg)
The following graph shows the 1000 different “could have been” sample proportions choosing scissors for samples of 119 people assuming scissors is chosen 1/3 of the time in the long run.
What we know about it
![Page 13: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/13.jpg)
Would you consider the results of this study to be convincing evidence that scissors are chosen less often in the long run than expected?
What we know about it
No, the p-value is going to be large 8%
No, the p-value is going to be small 2%
Yes, the p-value is going to be small 77%
Yes, the p-value is going to be large 9%
No, the distribution is centered at 1/3. 4%
![Page 14: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/14.jpg)
Suppose the study had only involved 50 people but with the same sample proportion picking scissors. How would the p-value change?
What we know about it
It would not change, the sample proportion was the same
22%
It would be smaller 11%
It would be larger 66%
Not enough information 1%
Single instructor (me), on 92 students, across 4 sections and 2 semesters
![Page 15: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/15.jpg)
Example #2. Moving beyond a specific item to sets of related items and retention
Tintle et al. 2012 (SERJ)+JSE
◦ Improvement in Data collection and Design, Tests of significance, Probability (Simulation) on post-test
◦ Data collection and Design and Tests of significance improvements were retained significantly better than in consensus curriculum
What we know about it
![Page 16: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/16.jpg)
Retention significantly better (p=0.02)
What we know about it
Pre-test Post-test 4-Months Later50
55
60
65
70
75
Retention of knowledge about tests of significance (6 items from CAOS)
Randomization
Consensus
![Page 17: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/17.jpg)
Example #3. How are weak students doing?
What we know about it
Pretest Posttest25
30
35
40
45
Performance on CAOS for lowest 1/3 of students (2007 vs. 2009)
Consensus
Randomization
![Page 18: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/18.jpg)
2012-2013
What we know about it
Group Pre-test Post-test Change
Lowest (n=210; 13 or less)
38% 55% 17%
Middle(n=329;14-17)
52% 60% 8%
Highest(n=250; 18+)
66% 69% 3%
All changes are highly significant using paired t-tests (p<0.001)
**Among those who completed course; anecdotally we’reseeing lower drop out rate now than with consensus curriculum
![Page 19: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/19.jpg)
Example #4. Understand new data contexts?
Old AP Statistics question
10 randomly selectedlaptop batteries; testedand measured hoursthey lasted
What we know about it
![Page 20: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/20.jpg)
To investigate whether the shape of the sample data distribution was simply due to chance or if it actually provides evidence that the population distribution of battery lifetimes is skewed to the right, the engineers at the company decided to take 100 random samples of lifetimes, each of size 10, sampled from a perfectly symmetric normally, distributed population with a mean of 2.6 hours and standard deviation of 0.29 hours. For each of those 100 samples, the statistic sample mean divided by the sample median was calculated. A dotplot of the 100 simulated skewness ratios is shown below.
What we know about it
![Page 21: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/21.jpg)
What is the explanation for why the engineers carried out the process above?
What we know about it
This process allows them to determine the percentage of the time the sample distribution would be skewed to the right
3%
This process allows them to compare their observed skewness ratio to what could have happened by chance if the population distribution was really symmetric/normally distributed.
64%
This process allows them to determine how many times they need to replicate the experiment for valid results
10%
This process allows them to compare their observed skewness ratio to what could have happened by chance if the population distribution was really right skewed.
23%
![Page 22: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/22.jpg)
Analysis of all (free-response) class tests is ongoing
Integrate observed statistic and simulated values to draw a conclusion?
What we know about it
![Page 23: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/23.jpg)
Summary
◦ Preliminary and current versions showed improved performance in understanding of tests of significance, design and probability (simulation) post-course, and improved retention in these areas
◦ These results appear stable across lower-performing students with older and newer versions of the curriculum
◦ Some evidence of student ability to apply the framework of inference (3-S) to novel situations
What we know about it
![Page 24: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/24.jpg)
Summary
◦ Some instructor differences, but also preliminary validation of “transferability” of findings across different institutions/instructors; new instructors?
◦ **Note: Some evidence of weaker performance in descriptive stats in this earlier curriculum; substantial changes to descriptive statistics approach to combat this.
What we know about it
![Page 25: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/25.jpg)
What’s making the change◦Content?◦Pedagogy?◦Repetition?
How much randomization before you see a change?
Are there differences student performance based on curricula? Are they important?
What don’t we know
![Page 26: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/26.jpg)
What are the developmental learning trajectories for inference (Do they understand what we mean by ‘simulation’)? Other topics?
Low performing students; promising---ACT, GPA
Does improved performance transfer across institutions/instructors? What kind of instructor training/support is needed to be successful?
Using CAOS (or adapted CAOS) questions, but do we still all agree these are the “right” questions? Is knowing what a small p-value means enough? What level of understanding are they attaining?
Why do students in both curriculums tend to do poorly on descriptive statistics questions? Or areas where we see little difference in curricula?
What we don’t know
![Page 27: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/27.jpg)
Preliminary indications continue to be positive
You can cite similar or improved performance on nationally standardized/accepted/normed tests for the approach
Tag line for peers and clients:◦ We are improving some areas (the important ones?) and doing no harm
elsewhere
Still lots of room for better understanding and continued improvement of approach
Student engagement (talk yesterday)
Next steps: Larger, more comprehensive assessment effort coordinated between users of randomization-based curriculum and those that don’t. If you are interested let me know.
What it means
![Page 28: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/28.jpg)
Author team (Beth Chance, George Cobb, Allan Rossman, Soma Roy, Todd Swanson and Jill VanderStoep)
Class testers
NSF funding
Acknowledgements
![Page 29: Small liberal arts college: 1350 undergraduate students Statistician within Department of Math, Stat and CS Class size: Stat 131 (30 students),](https://reader038.vdocuments.net/reader038/viewer/2022110116/551b14e05503462e578b5cbe/html5/thumbnails/29.jpg)
Tintle NL, VanderStoep J, Holmes V-L, Quisenberry B and Swanson T “Development and assessment of a preliminary randomization-based introductory statistics curriculum” Journal of Statistics Education 19(1), 2011
Tintle NL, Topliff K, VanderSteop J, Holmes V-L, Swanson T “Retention of statistical concepts in a preliminary randomization-based introductory statistics curriculum” Statistics Education Research Journal, 2012.
References