![Page 1: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/1.jpg)
Heriot Watt Heriot Watt UniversityUniversity
Breakout SessionBreakout Session
MCQ with ConfidenceMCQ with Confidence
Dr Phil DaviesDr Phil DaviesDivision of Computing & Mathematical SciencesDivision of Computing & Mathematical Sciences
Department of ComputingDepartment of ComputingFATFAT
University of GlamorganUniversity of Glamorgan
![Page 2: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/2.jpg)
E-Assessment can be viewed E-Assessment can be viewed from different perspectivesfrom different perspectives
• Commercial / Large Scale Implementations– Question Mark’s Perception– MOODLE – Triads (25 different types of Multiple Choice Type
Questions!)– TOIA (FREE!) Tool for On-line Interoperable Assessment– Blackboard Assessment (version!)– Standards (IMS QTI Specification)
• Try it and see
![Page 3: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/3.jpg)
How easy to do MCQ?How easy to do MCQ?Cost?Cost?
• Question• <normally>• Key = • Distracter One, Distracter Two, ……………..
• Note– Key(s) must always be correct– How many students would you expect to get it right for
it to be a GOOD question?– Distracter One / Two / Three equal # of selections– Throw away BAD questions?
![Page 4: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/4.jpg)
TOO HARD .. ?TOO HARD .. ?• Too time consuming• Often needs to be an iterative process• Does it matter if a question is NOT
perfect?• Seeking perfection not feasible• I suggest that assessment MUST be
included within a module, not only at end > make it embedded within TEACHING & LEARNING experience
• Formative / Summative / Both
![Page 5: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/5.jpg)
My Entry into CAAMy Entry into CAA
• 1st Attempt TOTAL FAILURE – 1998 OLAL• 5 minutes all finished apart from one student• Guessing
– Why should the student care if incorrect?– How to make it worthwhile – slow students down – make
them think• Negative marking used +2 and -1 (actually
becomes -3 –> double whammy!)• Allow TWO PASSES of the test• By concentrating on first pass
– will get reward on second pass– Formative
• IMPORTANT – WORTH MARKS
![Page 6: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/6.jpg)
![Page 7: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/7.jpg)
Own Background = Own Background = Objective Testing via MCQObjective Testing via MCQ
• Four Tests MCQ .. Two passes 50/50 (became graduated)
• Cumulative Tests 5%, 10%, 15%, 20%
• Amateur .. No Research Pedigree (CAA Conference)
• Improved Class results > not easier – more directed
• Weaker students improved dramatically• Assessed the learning outcomes of subject• “END JUSTIFIES THE MEANS”
![Page 8: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/8.jpg)
What did the students say What did the students say about it?about it?
• Excellent way of gaining knowledge• 2nd Test Allowed us to learn• Good assistant to learning• Encourages Revision• Very good, promotes learning of a
subject early on• Better than sex!!!!
![Page 9: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/9.jpg)
What decides the QUALITY What decides the QUALITY of a question?of a question?
• In the 2004 World cup qualifier between England and Greece, who scored the all important equalizing goal?
• David Beckham• Victoria Beckham• Brooklyn Beckham• Romeo Beckham
![Page 10: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/10.jpg)
Rewards?Rewards?• Know it• On the tip of my tongue• If I see the answers I’ll know it• Deduction• Guess• Know it … oh no I’ve got it wrong ..
misinformation
![Page 11: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/11.jpg)
TRIED ONE CONFIDENCE TESTONLY SHOWS ANSWERS ONCE CONFIDENCE SELECTED
RELATES TO MARKS +4, +2, +1, -2, -1, 0
![Page 12: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/12.jpg)
Proportion of Questions Proportion of Questions CorrectCorrect
• High Confidence => 84% correct• Fairly Confident => 58% correct• Not confident => 47% correct
• 72% of all questions answered with high confidence
![Page 13: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/13.jpg)
Student thoughts on Student thoughts on ConfidenceConfidence
• “Eliminates guesswork”• “I was waiting for a leggy blond to
bring on my prize at the end”• “separated the lucky students from
me”• “at least I didn’t lose marks for
selecting no confidence”• “are you testing my confidence or
knowledge”
![Page 14: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/14.jpg)
““Bite The Bullet”Bite The Bullet”• Use Confidence Testing for all four tests
(10%, 15%, 20% & 25%)• Real SHOCK to students• Weighting of Passes (50/50, 60/40, 70/30
& 80/20)• How to judge if the students had ‘really’
learned? • Select five questions that were answered
‘poorest’ in test one & pass through to test two
• Select five questions that were answered ‘poorest’ in test two & pass through to test three ( + original five) …. Etc.
![Page 15: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/15.jpg)
![Page 16: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/16.jpg)
Identifying Quality of a Identifying Quality of a QuestionQuestion
• Negative on first pass e.g. > -180• Difference between first and second
pass• Too big an improvement may just
signify a very poor question e.g. >500?
• Create heuristic that identifies degree of difficulty of a question
![Page 17: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/17.jpg)
Passed through to Test FourPassed through to Test Four• 13 of the ‘hardest’ question
• 6 new questions based upon additional work between tests 3-4
• 21 questions (cross section of questions and topics from previous tests)
![Page 18: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/18.jpg)
Improvement of Results Improvement of Results Through TestsThrough Tests
Questions Through Tests
-500
-400
-300
-200
-100
0
100
200
300
400
1 2 3 4 5 6 7 8
Question 5
Question 6
Question 10
Question 11
Question 14
Question 15
Question 16
Question 19
Question 20
Question 21
Question 26
Question 27
Question 40
![Page 19: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/19.jpg)
Improvement Pass One of Improvement Pass One of QuestionsQuestions
Pass One Question Improvements
-500
-400
-300
-200
-100
0
100
200
300
400
1 2 3 4
Question 5
Question 6
Question 10
Question 11
Question 14
Question 15
Question 16
Question 19
Question 20
Question 21
Question 26
Question 27
Question 40
![Page 20: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/20.jpg)
Improvement for Question 5Improvement for Question 5• Stronger Students improved quickly
through the tests• Became more confident in answering
the questions• Weaker Students improved less
quickly than stronger through the test progress, but DID improve
![Page 21: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/21.jpg)
Is it just Confidence or Knowledge Is it just Confidence or Knowledge i.e. questions right or wrong?i.e. questions right or wrong?
Category Average # of Questions Correct Pass 1
Average # of Questions Correct Pass 2
Average Total # of Questions Correct
80> (1) 131.0 158.0 289.0
75-79 (2) 123.5 150.5 274.0
70-74 (3) 117.6 148.6 266.2
65-69 (4) 113.8 150.7 264.5
60-64 (5) 110.3 145.0 255.3
55-59 (6) 107.6 143.4 251.0
50-54 (7) 100.4 136.2 236.6
45-49 (8) 92.5 132.6 225.1
40-44 (9) 94.0 126.4 220.4
35-39 (10) 87.7 121.6 209.3
30-34 (11) 80.5 117.3 197.8
25-29 (12) 76.7 117.3 194.0
![Page 22: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/22.jpg)
Quality of a multiple choice Quality of a multiple choice question?question?
• In the 2004 World cup qualifier between England and Greece, who scored the all important equalizing goal?
• Teddy Sheringham• Michael Owen• Robbie Fowler• QUALITY OF QUESTION NOT
DISTRACTERS
![Page 23: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/23.jpg)
Developmental v Developmental v AssessmentAssessment
• Need not be at alternative ends of the scale
• Student motivation / attendance increased considerably
• Testing knowledge (not done at moment for higher order skills)
• Good students have best results i.e. number of questions correct
![Page 24: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/24.jpg)
COMMENTS?COMMENTS?• HMCQ• Confidence• Two Pass• Formative & Summative• Graduated Assessment• Continual Assessment
![Page 25: Heriot Watt University Breakout Session MCQ with Confidence](https://reader035.vdocuments.net/reader035/viewer/2022062410/568158ec550346895dc62c17/html5/thumbnails/25.jpg)