wp5: validation anne de roeck diane evans the open university, uk
TRANSCRIPT
WP5: Validation
Anne De RoeckDiane Evans
The Open University, UK
WP5
• Who?• Objectives• Challenges & Risks• Scenario Based Evaluation• Why now?• Progress• Plan
Who?
• All Partners• Open University (WP5 Lead)
– Distance: 200,000 students – 90% on-line– Research and Teaching
• OU Centre for Research in Computing– CRC = Computing + KMi + IET UserLab– NLP: Anne De Roeck; Donia Scott, …– E-learning: Josie Taylor; Patrick McAndrew;
Diane Evans, …
WP5 Objectives
• Aim:– to validate if eLearning systems (LMS) are
improved by adding new functionalities based on language technology tools and ontology-based (multilingual) retrieval.
• Objectives – to assess the extent to which the integration of
these new functionalities affects the effectiveness of the eLearning process.
Challenges
• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development
Challenges
• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development
Challenges
• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development
• Challenge 1: Methodology:– Scenario based validation– Functionality & user context centred– User questionnaires & interviews
Challenges
• Focus:– Language Technology Tools– Ontology retrieval– Improve effectiveness!
• Challenge 2: Search & Retrieval Scenarios– Embed functionality (teachers & learners?)– Establish base-line– Refine scenarios
Challenges
• Focus:– Multi-lingual experience
• Challenge 3: Multilingual Scenarios– Multiple communities– Multi-lingual scenario– Chart the territory!
Risks
• Content!• Assessment??• Performance of Components?• Interface vs Functionality• Cross-cultural issues
Risks
• Content!• Assessment?• Performance of Components?• Interface vs Functionality• Cross-cultural issues
• Technical deliverable vs user requirements
Methodology
• User Scenarios– ‘a story focused on a user or group of users,
which provides information on the nature of the users, the goals they wish to achieve and the context in which the activities will take place’.
Methodology
• User Scenarios– Step beyond Software Development Use Cases– Qualitative evaluation of functionalities …– …in context of user roles– Accessible to all stakeholders – Developed from high level descriptions into
scenario instantiations during implementation– Used in learning applications (eg mobile
learning) & Software Development
Why now?
• Functionalities & System development• LT4eL = New technology• Co-evolve system & validation
– Process of scenario instantiation:– Explore/refine ideas on possible usage– Clarify user requirements– Establish common ground between pedagogists,
developers, NLPers, users– Concurrent with design & development
Progress
• Aim:– Prepare & support user scenario validation– Clarify objectives– Mitigate risk
• Background literature search and survey: – Keyword extraction & validation methods– Ontologies and deployment in search– Scenario-based validation
Progress
• Contribution WP1:– Identifying and supplying English Content within
the domain of e-learning in educational establishments.
• Contribution WP2/3:– running early version of keyword extractor with
English content and feeding back to WP2 leader. – liasing with Malta on English content– identifying qualitative validation tests for Keyword
Extractor application.
Progress
• Start WP5:– Monitor content (ongoing: high risk!)– Draft validation plan for ILIAS + new
functionality– Delivered high level user scenario (starting
point of instantiation)– Preliminary Questionnaire (establish qualitative
base-line)
Progress/Plan
• Questionnaire:– slide presentation supporting 'focus group' – questionnaire (in Word format) covering search
motivation and behaviours – 81 individual student responses from 6
partners
Progress/Plan
• Questionnaire Findings on:– Reasons to search in non Native language– Difficulties encountered when searching– Selecting search criteria – Handling search results– Other issues
Plan
• Process Questionnaire• Challenge2
– Refine search & retrieval scenarios (Workshop – March)
– Integration (WP4)
• Challenge 3– Multi-lingual dimension to functionalities– Uncharted territory !
Plan
0. Preliminary Questionnaire (M12) - Partners
1. Pilot Validation (M16-18) – OU2. Main Validation (M19-23) – Partners (ILIAS Community part)3. Final Validation (M26-28)
• Reports M24, M30