stem 2012 automatically assessing free-form diagrams in e-assessment systems pete thomas kevin waugh...
TRANSCRIPT
![Page 1: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/1.jpg)
STEM 2012
Automatically assessing free-form diagrams in E-assessment systems
Pete Thomas
Kevin Waugh and Neil Smith
Open University, UK
![Page 2: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/2.jpg)
STEM 2012
Aim
• To extend the capabilities of e-assessment systems (online tests) to include questions that require students to draw free-form diagrams in answers.
• In particular:– Minimal constraints on what might be drawn–Not multiple choice– Support for a variety of diagram types– Automatic marking – Moodle VLE (via OpenMark)
![Page 3: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/3.jpg)
STEM 2012
Example
Student draws diagramin response to question
![Page 4: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/4.jpg)
STEM 2012
What is a diagram?• Free-form
–Not free-hand–Uses a drawing tool
• Specialised for the domain by the teacher• As unconstrained as the teacher wishes
• Graph based–Shows relationships between objects
![Page 5: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/5.jpg)
STEM 2012
Biological Flow Diagram
Illustrates the flow of hormones between glands.
![Page 6: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/6.jpg)
STEM 2012
Entity-Relationship Diagram
Shows the relationships (links) between entities (boxes)
![Page 7: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/7.jpg)
STEM 2012
Use case diagram (UML)
Shows the actions that an actor Book Borrower takes part in when using a library
![Page 8: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/8.jpg)
STEM 2012
Spray Diagram
Illustrates how UK farms have diversified in recent times.
![Page 9: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/9.jpg)
STEM 2012
Spatial Relationships
It is the relative positions of boxes that determines whether a relationship exists. Adult and Child are inside Person – inside indicates subtype relationship..
![Page 10: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/10.jpg)
STEM 2012
Sequences
time
The messages, represented by arrows, are sequenced in time.
![Page 11: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/11.jpg)
STEM 2012
Question in VLE test
This is what the user sees in their browser
![Page 12: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/12.jpg)
STEM 2012
Run the Drawing Tool
Up pops the drawing tool – an applet
![Page 13: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/13.jpg)
STEM 2012
Automatic marker provides feedback
Feedback from the
automatic marker
![Page 14: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/14.jpg)
STEM 2012
Get Answer
![Page 15: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/15.jpg)
STEM 2012
Automatic Marking
• The student’s diagram is compared with the model solution(s) to see how well the objects and the relationships match.
• Similarity measures of how well each element on the student diagram matches with an element on the model solution.– A mixture of NLP and heuristics
• A marking scheme is applied.– Designed by the question setter from a template
![Page 16: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/16.jpg)
STEM 2012
Does it work? – Experiment 1394 diagrams from a real exam - marked out of 7.
Difference between auto and human markers
0 0.5 1.0 1.5 2.0 3.5
Number 389 4 1 0 0 0
% 98.7 1.0 0.25 0 0 0
Cumulative % 98.7 99.7 100 100 100 100
Main corpus
Diagrams only have entity types and relationships
Published in 2011
![Page 17: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/17.jpg)
STEM 2012
Comparison Human – Auto(Distribution of marks)
![Page 18: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/18.jpg)
STEM 2012
Support Tools
• An authoring tool for teacher to design the question, the model solution(s) and the mark scheme.
• A specification tool to define a type of diagram. – Used to specialise both the drawing tool and the
automatic marker
• A marking tool for marking sets of diagrams off-line.
![Page 19: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/19.jpg)
STEM 2012
Authoring Tool for Questions,Model Solutions and Marking Schemes
Question(text)
Model Solution(diagram)
![Page 20: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/20.jpg)
STEM 2012
Specification Tool
A graphical user interface for defining a new type of diagram
![Page 21: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/21.jpg)
STEM 2012
Summary
• A new question type for online tests• Free-form diagrams (graph based)• Automatic marking • Implemented in Moodle• Set of support tools
![Page 22: STEM 2012 Automatically assessing free-form diagrams in E-assessment systems Pete Thomas Kevin Waugh and Neil Smith Open University, UK](https://reader035.vdocuments.net/reader035/viewer/2022070306/551615e8550346a2308b551c/html5/thumbnails/22.jpg)
STEM 2012
Thank You
• We always want more examples of types of diagram. Send me your suggestions.
• There is a website where you can try this technology out. Please contact me for details.