cambridge exam research malta, 2009 © pollitt & ahmed, 2009 the importance of being valid 1 the...
TRANSCRIPT
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 1
The Importance of being Valid
Alastair Pollitt & Ayesha Ahmed
Cambridge Exam [email protected]
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 2
Defining ‘Validity’
There is no true meaning of’ Validity’
Definitions of ‘Validity’ are opinions about what the word should mean
How do we decide what ‘Validity’ should mean?
We need an inclusive concept of ‘Validity’ that is useful to everyone involved in the process of assessment, and not too far from what they normally mean by ‘Valid’.
A useful definition of ‘Validity’
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 3
The real world meanings of “valid”
1 offer 1022 ticket 323 licence 254 certificate 255 passport 196 law 157 voucher 14
. . . . . . . . . . . . . . .
18 reason 6
Relative Rank Concordance frequency
Source: Collins Cobuild Corpora (British + American)
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 4
What we mean by Validity
We have worked for many years with professionals who:
• Write questions; Review questions; Write mark schemes; Mark tests; Design tests; etc
The key issue for them, and us, is what the students’ minds are/were doing during the test –
in other words,
the Evidence that gives us about achievement.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 5
What we mean by Validity
This is the essence of Validity – the interaction between a test question and the student’s mind.
If this goes wrong then Validity is impossible.
But how do we get it right?
By considering further the concept of Evidence.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 6
Evidence
It is not enough to be Logically Valid.
A “widely used definition of assessment”:
The process of seeking and interpreting evidence for use by …
(Assessment Reform Group, 2002:
quoted by Stobart, 2009)
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 7
Importance Statement
A negotiated consensual statement of what is really important in the study of that subject during that phase of education.
Available to teachers, students, parents, examiners, employers, selectors – to everyone with an interest in the subject.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 8
The Importance of Design & Technology
Design and technology prepares pupils to participate in tomorrow's rapidly changing technologies. They learn to think and intervene creatively to improve quality of life. The subject calls for pupils to become autonomous and creative problem solvers, as individuals and members of a team. They must look for needs, wants and opportunities and respond to them by developing a range of ideas and making products and systems.
They combine practical skills with an understanding of aesthetics, social and environmental issues, function and industrial practices. As they do so, they reflect on and evaluate present and past design and technology, its uses and effects. Through design and technology, all pupils can become discriminating and informed users of products, and become innovators.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 9
The Importance of the Importance Statement
1 To ensure that teachers emphasise what is important
2 To ensure that test designers, question and mark scheme writers test what is important
3 To ensure that markers give credit for what is important
4 To ensure that test results are interpreted in terms of what is important
5 The source of Validity
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 10
A model of Validity
Validity is about measuring and interpreting what is important in education.
It is not merely about measuring and interpreting what is measured.
Our model describes where Validity comes from, and how it is maximised.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 11
The Bucket Brigade model
Suppose: Y is a house on fire.
Suppose: X is a well full of water.
How do you put out the fire?
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 12
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 13
A typical assessment professional
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 14
The Entropy law of validity
Version 1
During the assessment process, validity always decreases
Version 2
Validity can only be lost, it can never be recovered.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 15
How to maintain validity - 1
Writing the Importance Statement
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 16
How to maintain validity - 2
Specifying the test
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 17
How to maintain validity - 3
Writing the questions and mark schemes
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 18
How to maintain validity - 4
The students
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 19
How to maintain validity - 5
Marking
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 20
How to maintain validity - 6
Grading, etc
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 21
How to maintain validity - 7
Interpreting results
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 22
How to maintain validity
1 Writing the Importance Statement
2 Specifying the test
3 Writing the questions and mark schemes
4 Communicating with students
5 Marking
6 Grading
7 Interpreting
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 23
Summary
The meaning of test results depends crucially on a proper development of the test, so that it elicits evidence of what is important.
Without meaning, Validity is an empty concept.
The source of Validity must be a clear and shared concept of Importance.
Cambridge Exam Research Malta, 2009
© Pollitt & Ahmed, 2009 The Importance of being Valid 24
8The importance of being Valid
E
Q
i
M
Idea of a Question
Evaluation
Elicitation
IS
A
Importance Statement
TeachingT
Assessment
Evidence