seminario emadrid
TRANSCRIPT
Open Standards and Specifications for Outcome-
based Assessment
Israel GutiérrezUniversidad Carlos III de Madrid
Madrid, April 12th 2010
Computer-based assessmentReduce effort and costs: permit continuous assessmentAssessment re-use: interoperability needed
Variety of assessment formatsProprietary: MoodleXML, HotPotatoes, Blackboard Ad hoc: OpenMark IMS QTI
Open Standards and Specifications for Assessment
Outcome-based Assessment Universities developing an
outcome-based learning strategy Adaptation to European
Space for Higher Education Improve course structure for
accreditation (ABET)
Creation of new coursesLearning outcomesAssessment as a key process
Outline
Analysis of assessment standards and specifications
Usage of assessment formats in Europe
Interoperability issues
Next generation assessment scenarios
Best practices
Methodology
Analysis of Assessment Standards and Specifications
Metamodels (UML diagrams) from MDAUnderstand better studied
formatsAbstraction from syntax
(XML binding)Focus on content
Studied Assessment Formats
IMS QTI
OpenMarkSuML
FML
General purposeAssessment purpose
QAML
Qualitative ComparisonKey features
Multiple response allowanceCorrect response indicationMetadata capabilitiesHybrid question management
Question types (from FREMA)Constructed: essay, short answer, FIB, etc.Constrained: MCQ, MRQ, TF, matching, etc.Miscellaneous: simulations, practical
sessions
Assessment Formats Usage in Europethe current picture – methodology and sample description
• Data collection• Personal interview• In-depth report
• Qualitative analysis• Sample characteristics
• Higher education• Multicultural• Multilingual
Series10
2
4
6
8
10
12
14
Assessment Scenarios
Evaluation Scenarios
Tools & Services
Pilots
Sample size
Project results: WP6
Assessment Formats Study: Resultsthe current picture – key conclusions
Yes22%
No78%
Re-use of assessement resources
• Scarce re-use of assessment resources• Scarce acceptance of IMS QTI specification• High technological diversity
– LMS, ad-hoc tools, etc.• Diversity of assessment methodologies
– formative/summative, various types of activities, face-to-face/blended/elearning settings, etc.
Interoperability Issues: IMS QTIScarce use of the specification
Incomplete support in LMS i.e. adaptive material
Flexibility handicaps
QTI profiles
Versions instability: 1.2.1 or 2.1?Versions are not backwards compatible
Assessment Interoperability
Moodle
.LRN Clix
QTIexport QTIMoodle
QTIlrn QTIclix
QTI 1.2.1
QTI 1.2.1
QTI 1.2.1
Next Generation Assessment Scenarios • Hints
• Adaptation• Instructional
design
Assessment Resources Interoperability ProposalAssessment resources
Objective: Gather and develop best practices about interoperable content as support to higher education and learning outcome oriented
Results• ICOPER Reference Model• Gathering of recommendations• Open ICOPER Content Space• One of the most successful eContent+ projects since 2004
Open ICOPER Content Space• Federated repository of educational resources
Outcome-based Assessment AppUse cases
Search for assessment resources in a repository Filter by intended learning outcomes Filter by assessment methods
Import/re-use of assessment resourcesPublish/share assessment resources
Online demo: http://mozart.gast.it.uc3m.es:9000/
Outcome-based Assessment AppUse cases (cont.)
Annotate assessment resources Peer review Teacher reflection Student feedback
Update learner’s profile
StakeholdersFaculty membersCourse developers
Online demo: http://mozart.gast.it.uc3m.es:9000/
Outcome-based Assessment App
Learning outcome-based search
Learning outcome profile update
Implementation for dotLRN based on
ICOPER LOM profile
Outcome-based Assessment App: EvaluationEngineering evaluation
ImplementersTool developers
End user evaluationFaculty membersCourse developers
Epistemological evaluation ResearchersStandardization bodies
ICOPER Assessment appAssessment resources should include information
about Intended learning outcomesRecommended assessment methodsAnnotations about resources
Usage of integrated environment for outcome-based assessment
