loughborough dhd seminar

Physics Education ResearchThe University of Edinburgh

Diagnosing students’ data handling skills

Simon Bates Ross Galloway

Helen Maynard-Casely

Hilary Singer Kate Slaughter


• Diagnostic tests: what, why and how?• Existing diagnostic tests• Why investigate data handling skills?• Development of the diagnostic• Initial results• Development of supporting resources• Summary

Outline


Diagnostic tests: what?

• Standardised, statistically validated test.

• Often in multiple choice question (MCQ) format.

• Targeted to address key learning outcomes.

• Often used to identify misconceptions.


Diagnostic tests: why?

Ausubel’s Dictum: “Ascertain what the student knows and teach accordingly”

“Ascertain what the student misunderstands and teach accordingly”


Diagnostic tests: how?

Often administered using ‘pre / post’ scheme.

Pre-test

Respond

Instruct

Post-test


Existing diagnostic tests

• Scientific reasoning (Lawson)• Student attitudes (CLASS) • Graphs (TUG-K) • Forces and statics (FCI) • Electricity and magnetism (BEMA, homegrown) • Quantum mechanics (QMCS, QPCS, homegrown)• Light and spectroscopy Concept Inventory (Bader)• Data handling (UBC/homegrown)• …


Force Concept Inventory… a mature, established diagnostic test.


Labs: the implicit curriculum• In practical work, we expect students

to acquire data analysis skills in parallel to practical abilities.

• Frequently, these generic skills are not explicitly taught and not effectively assessed.


HEA Student Employability Profiles


0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0y-

axi

s

x -ax is

Typical question (from UBC pre-prototype)


0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0y-

axi

s

x -ax is

A


0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0y-

axi

s

x -ax is

B


0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0y-

axi

s

x -ax is

C


0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0y-

axi

s

x -ax is

D


0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0

y-a

xis

x -ax is

A

0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0

y-a

xis

x -ax is B

0 5 1 0 1 5 2 0-5

0

5

1 0

1 5

2 0

y-a

xis

x -ax is

C0 5 1 0 1 5 2 0

-5

0

5

1 0

1 5

2 0

y-a

xis

x -ax is D


No statistically significant difference between 1st, 2nd and 4th year classes.

Response profiles from Edinburgh Physics classes


Where we went from here…

• Build on pre-prototype UBC instrument.

• Expand number of questions, increase scope.

• Ensure generic nature so wide applicability: physics, chemistry, engineering, other STEM, etc.

• Ensure test is robust and reliable.


Diagnostic test development process

Beichner R. J., (1994) Am. J. Physics 62, 750.


23 item MCQ test to address generic data analysis skills. Wide applicability to all STEM disciplines.

Topics covered include:

o precision and accuracy

o approximations

o lines of best fit

o errors and uncertainties

o derivation of quantitative relationships from graphical data

Data Handling Diagnostic – prototype test instrument


Trial phase deployment

• Prototype test taken by > 1200 students

• Test deployed in 10 institutions across UK and Ireland (7 Physics & 4 Chemistry depts.)

• Test deployed to all undergraduate years (1st – 5th)


Validity and reliability evaluationValidity• test covers appropriate range of skills• test does actually evaluate what you intendedReliability• degree of consistency between scores when

test repeated • test has ability to discriminate student ability

Employ statistical tests and analysis, for individual items and for whole test.

e.g. BEMA: Phys. Rev. Spec. Topics PER 2 010105 (2006)

DHD found to perform satisfactorily


Student performance from the trial deployment

0

20

40

60

80

100

120

140

160

180

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Test Score

# o

f S

tud

ents

Histogram of student scoresHistogram of student scores


0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Question #

Me

an

% C

orr

ec

t

EasyEasy MediumMedium HardHard

Question correct response rates (all students)Question correct response rates (all students)


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Question #

% R

espo

nses

A B C D Blank

Question response profilesQuestion response profiles


Question 18 Question 18 and response and response

profileprofile


0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Question #

% C

orr

ec

t

Question correct response rates broken down by classQuestion correct response rates broken down by class


Question correct response rates broken down by classQuestion correct response rates broken down by class

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Question #

% C

orr

ec

t


45%

50%

55%

60%

65%

70%

1st 2nd 3rd 4th 5th PhD

Year of Study

Me

an

Te

st

Sc

ore

Edinburgh Physics year-by-year mean test scoresEdinburgh Physics year-by-year mean test scores


Edinburgh Physics year-by-year question correct response Edinburgh Physics year-by-year question correct response ratesrates

2 consistency test p-values:1st 2nd 3rd 4th 5th PhD

0.038 0.971 0.985 0.934 0.701 0.238

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Question #

% C

orr

ect

1st year 2nd year 3rd year 4th year 5th year PhD


• Augment test with stand-alone interactive learning resources

– diagnose and address misconceptions.

Where do we go from here?


Learning resources

• Stand-alone and web-based, to support rather than supplant other instruction.

• Accessible direct from questions and by category.

• Feature informal explanations, step-by-step guides,

animations, practice exercises and self-test questions.


Example: mean, standard deviation and standard error


• Random samples drawn from Gaussian distribution.

• mean = 10 std.dev. = 5

• Track evolution of mean, standard deviation and standard error as number of measurements N increases.


• Diagnostic tests have been shown to effectively identify learning challenges.

• Data handling skills are important and valued but frequently fall into the ‘implicit curriculum’.

• We have developed a valid, reliable and effective Data Handling Diagnostic.

• We have found that development of these skills and overall student proficiency is not what it might be.

• The diagnostic instrument and its supporting learning resources are ready for use.

Summary


http://bit.ly/EdPER

Acknowledgements:We gratefully acknowledge project grant support from the HEA Physical

Sciences Centre and the assistance of our partner institutions in the pilot phase. We also gratefully acknowledge Doug Bonn and James Day of the University of British Columbia for initial ideas and for allowing re-use of three of their items.

[email protected] [email protected]

loughborough dhd seminar

Documents