diagnosing student understanding of data analysis techniques
TRANSCRIPT
Physics Education Research The University of Edinburgh
Diagnosing Student Understanding of Data Analysis Techniques
Simon Bates Ross Galloway Helen Maynard-Casely
Hilary Singer Kate Slaughter
Physics Education Research Group, University of Edinburgh, UK
AAPT Summer Meeting 2011, Omaha NE, Aug 2011
Physics Education Research The University of Edinburgh
• Why investigate data handling skills?
• Development of the diagnostic
• Student performance on the diagnostic
• Development of supporting resources
• Summary
Outline
Physics Education Research The University of Edinburgh
Labs: the implicit curriculum • In practical work, we expect students
to acquire data analysis skills in parallel to practical abilities.
• Frequently, these generic skills are not explicitly taught and not effectively assessed.
Physics Education Research The University of Edinburgh
Higher Education Academy Student Employability Profiles
Physics Education Research The University of Edinburgh
Typical question (from UBC pre-prototype)
Physics Education Research The University of Edinburgh
A!
Physics Education Research The University of Edinburgh
B!
Physics Education Research The University of Edinburgh
C!
Physics Education Research The University of Edinburgh
D!
Physics Education Research The University of Edinburgh
A!
B!
C!D!
Physics Education Research The University of Edinburgh
No statistically significant difference between 1st, 2nd and 4th year classes.
Response profiles from Edinburgh Physics classes
Physics Education Research The University of Edinburgh
Where we went from here…
• Build on pre-prototype UBC instrument.
• Expand number of questions, increase scope.
• Ensure generic nature so wide applicability: physics, chemistry, engineering, other STEM, etc.
• Ensure test is robust and reliable.
Physics Education Research The University of Edinburgh
Diagnostic test development process
Beichner R. J., (1994) Am. J. Physics 62, 750.
Physics Education Research The University of Edinburgh
23 item MCQ test to address generic data analysis skills. Wide applicability to all STEM disciplines.
Topics covered include: o precision and accuracy
o approximations
o lines of best fit
o errors and uncertainties
o derivation of quantitative relationships from graphical data
Data Handling Diagnostic – prototype test instrument
Physics Education Research The University of Edinburgh
Trial phase deployment
• Prototype test taken by > 1200 students
• Test deployed in 10 institutions across UK and Ireland (7 Physics & 4 Chemistry depts.)
• Test deployed to all undergraduate years (1st – 5th)
Physics Education Research The University of Edinburgh
Validity and reliability evaluation Validity • test covers appropriate range of skills • test does actually evaluate what you intended Reliability • degree of consistency between scores when
test repeated • test has ability to discriminate student ability
Employ statistical tests and analysis, for individual items and for whole test.
e.g. BEMA: Phys. Rev. Spec. Topics PER 2 010105 (2006)
DHD found to perform satisfactorily
Physics Education Research The University of Edinburgh
Student performance from the trial deployment
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
‘Phase space’ of class response profiles
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
‘Phase space’ of class response profiles
PG
Physics Education Research The University of Edinburgh
‘Phase space’ of class response profiles
PG
Faculty
Physics Education Research The University of Edinburgh
χ2 consistency test p-values: 1st 2nd 3rd 4th 5th PG
0.038 0.971 0.985 0.934 0.701 0.238
Physics Education Research The University of Edinburgh
Data handling skill development
• 1st and 2nd year → explicit instruction → improvement
• Later years → implicit development → no improvement
• Postgrad TA’s not more proficient than undergrads they are instructing
• Faculty significantly more proficient
Broadly consistent with findings at University of British Columbia Day & Bonn, PRST-PER 7, 010114, (2011)
Physics Education Research The University of Edinburgh
• Augment test with stand-alone interactive learning resources
– diagnose and address misconceptions.
Where do we go from here?
Physics Education Research The University of Edinburgh
Learning resources
• Stand-alone and web-based, to support rather than supplant other instruction.
• Accessible direct from questions and by category.
• Feature informal explanations, step-by-step guides, animations, practice exercises and self-test questions.
Physics Education Research The University of Edinburgh
Physics Education Research The University of Edinburgh
Example: mean, standard deviation and standard error
Physics Education Research The University of Edinburgh
• Random samples drawn from Gaussian distribution.
• mean = 10 std.dev. = 5
• Track evolution of mean, standard deviation and standard error as number of measurements N increases.
Physics Education Research The University of Edinburgh
• Data handling skills are important and valued but frequently fall into the ‘implicit curriculum’.
• We have developed a valid and reliable Data Handling Diagnostic.
• We have found that development of these skills and overall student proficiency is not what it might be.
• Part of the problem may be that lab Teaching Assistants also have serious gaps in their knowledge.
• Explicit instruction in data handling skills seems to be required.
Summary
Physics Education Research The University of Edinburgh
Acknowledgements: We gratefully acknowledge project grant support from the HEA Physical
Sciences Centre and the assistance of our partner institutions in the pilot phase. We also gratefully acknowledge Doug Bonn and James Day of the University of British Columbia for initial ideas and for allowing re-use of three of their items.
Poster at PERC
EdPER group website bit.ly/EdPER
Talk slides on Slideshare EdPER_talks