Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Redesigning the computer assisted language exams for federal government employees: a didactic, methodological and technological challenge
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
0. Context
1.Didactic challenge
2.Methodological challenge
3.Technological challenge
4.Results & future developments
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
0. Context
ı SELOR: Belgian governmental selection bureauin a trilingual country, multilingualism is crucial for state servants-> language testing is crucial part of assessment & selection procedures
ı Since the 90s:‖ ATLAS =
electronic language testing system for Dutch and French.‖ Yearly thousands of candidates‖ Wide variety of governmental jobs (policeman as well as diplomats)
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ATLAS = state-of-the art at its creation,but needed a complete overhaul in three domains
1. didactical component:ı strongly focused on language knowledgeı weak integration of skills-based view on language competence
(Common European Framework of Reference didn’t exist at that time)
2. Methodological componentı Level structure without psychometric underpinning:
4 levelsı No evaluation of the reliability and validity of the ATLAS tests
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
3. Technological component
ı ATLAS operated on the SELOR network, not onlineı Closed, non-adaptable and non-updatable systemı Off-line accompanying training module on cd-romı No itembanking ı No integration into Selor admin
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Constraints:‖ Legal constraints:
e.g. vocabulary & grammar should be tested separatelye.g. 4 levels (1 to 4) should be distinguished
‖ Practical constraints:
e.g. each examen takes maximum 120 minutese.g. SELOR wanted us to reuse the existing items as much as possiblee.g. whole operation had to be realised within one year
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Research team: partners
Didactic componentFrench: Piet Desmet (K.U.Leuven Campus Kortrijk)Dutch: Guy Deville (FUNDP Namur)
Methodological componentSara Gysen (K.U.Leuven)
Technological componentBert Wylin (Televic Education)
CoordinationPiet Desmet & Sara Gysen
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
0. Context
1.Didactic challenge
2.Methodological challenge
3.Technological challenge
4.Results & future developments
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
I. Didactic challenge
ı I.I. Construct definition
ı I.2. Item revision and new item writing
ı 1.3. Metadata
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
1.1. Construct definition
ı From 9 modules to 4 components:‖ 2 knowledge-oriented: vocabulary & grammar
2 skills-oriented: listening & reading
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
I.2. Item Revision & New item writingı Revision of existing items:
‖ Uniformity(e.g. same type MCQ; only one gap in all cloze exercises)
‖ Transparancy for test candidates(e.g. dichotomous rating for all items)
ı New item writing‖ As authentic as possible: real audiofragments, scanned
articles, letters etc‖ A spectrum of different item types, not only multiple
choice, in order to test as direct as possible the different tasks specified in the construct
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı New items writing
‖ New items were developed for the new categories within the listening and reading component
‖ As authentic as possible: real audiofragments, scanned articles, letters etc: same look and feel, same distribution of images as in real-life tasks
‖ A spectrum of different item types, not only multiple choice, in order to test as direct as possible the different tasks specified in the construct
‖ Standard choice of technical item type for each part of the construct
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
1.3. Metadata
ı Item tags and features of 3 types:‖ Content metadata (automatic and manual)‖ Psychometric metadata (cf. 2)‖ Dynamic metadata (evolving through use of system)
ı Important for itembanking:‖ Control of item selection in examen versions‖ Monitoring of item quality
(cf. psychometric data and dynamic metadata)
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Metadata for each item of the database:‖ Content metadata
ш Identification numberш Question formatш Excluded when other item presentш Linked to other itemш Date of creationш Date of adaptationш Adapted for candidates with special needsш Ratingш In training environmentш Inactiveш Assets (multimedia)ш Length audio/videoш Length text ш Example item
‖ Dynamic metadataш Popularity of itemш Average answer time
‖ Psychometric metadataш Logit-valueш P-value
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
0. Context
1.Didactic challenge
2.Methodological challenge
3.Technological challenge
4.Results & future developments
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
2. Methodological challenge
ı 2.I. Screening and calibration of existing database
ı 2.2. Development of an IRT-based item database
ı 2.3. Standard setting & selection rules
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
2.1. Screening of the existing database
ı Screening based on test data from 1995-2000
‖ Elimination of items based onш their p-value (percentage correct answers provided by the test candidates)
lower than 0.10 (extremely difficult)higher than 0.95 (extremely easy)
ш and their occurence in test versions:at least 100 times in test version
ш Results: 218 French items and 849 Dutch items were eliminated
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
2.2. Development of an IRT-based database
ı Submitting the items to a psychometric analysis based onItem Respons Theory (IRT) which allows to place items on a scale
that orders items in function of their intrinsic difficulty level (logit value)
&that orders examinees in terms of their ability
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Example of a measurement scale in IRT model
ı Probabilistic model :Person B has great potential to answer items c & b correctly but far less to answer items d or f correctly, which will normally be solved correctly by person A.The chance that person B will be able to answer item e is almost non existing
Meetschaal
Kandidaten
vaardigheid
Items
moeilijkheidsgraad
Item g
Item e
Item a
Item b
Item c
Item d
Item f
Item h
Item i
Item j
Beheersing in termen van kansen
Meetschaal
Kandidaten
vaardigheid
Items
moeilijkheidsgraad
Item g
Item e
Item a
Item b
Item c
Item d
Item f
Item h
Item i
Item j
Item g
Item e
Item a
Item b
Item c
Item d
Item f
Item h
Item i
Item j
Item e
Item a
Item b
Item c
Item d
Item f
Item h
Item i
Item j
Item a
Item b
Item c
Item d
Item f
Item h
Item i
Item j
Beheersing in termen van kansen
A
B
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Eight different scales:one per target language and per component
e.g. Logit distribution of candidates and items (French) for component 4 - Reading
Metadataset your own metadata
Metadataset your own metadata
Use system metadata
Use system metadata
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
1. Test management by candidate him/herself:- Candidate decides when to start up the next component
(but fixed time limit of 120min for the whole exam) - Possibility to review within component- Overview screen and ‘brainteaser’ tag- No restriction on playing audio and video input,
but limited time allocation is mentioned in instruction- Possibility of not answering an itemBut restrictions:- Time limit: fixed per component (time interval on screen)- Fixed order of components: C1 C2 C3 C4
2. “Resuming” possible in case of problems
3. Equal share of components in overall score
2.3. Standard setting and selection rules
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Exam version1 General introduction to test No time
allocation
2 Vocabulary component
Instruction + 1 example item
20 test items 15 min
3 Grammar component
Instruction + 5 example items
20 test items 15 min
4 Listening component Instruction + 2 example items
20 test items 45 min
5 Reading component Instruction + 3 example items
20 test items 45 min
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
0. Context
1.Didactic challenge
2.Methodological challenge
3.Technological challenge
4.Results & future developments
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
3. Technological challenge
ı 3.I. E-testing: Edumatic technology
ı 3.2. Exam & preparatory learning environment
ı 3.3. Itembanking & Selor Test Administration System
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
3.1. E-testing:Edumatic based environment
ı Edumatic is an authoring system for exercises and testsfor both online and offline assessments(online server based,with export button to offline SCORM-compliant)
ı xml-based data in a flash user interface(ims-qti and scorm compliant)
ı 20+ question typessupports multimedia in all question types
Visit our booth!
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
3.2. Exam & preparatory learning environmentı Customization of Edumatic environment
‖ Selor-skin
‖ Single Sign-On with Selor Admin(SSO: login once, get access to multiple applications)
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Exam configuration‖ Secure browser
Full classroom control system (AvidaNet Exam)
‖ Resume
‖ Strict time allocation
‖ Sequencing
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
ı Online learning environment:available at http://www.selor.be
‖ Preparatory learning environment
‖ Login for free via “My Selor”
‖ Components 1 & 2 (vocabulary & grammar) : access to the entire database+ 35.000 items
‖ Components 3 & 4 (listening & reading):only model items
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
• Stats: user stats
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
• Stats: user stats
• Stats: package stats
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
0. Context
1.Didactic challenge
2.Methodological challenge
3.Technological challenge
4.Results & future developments
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
4. Results & further developmentsı Fully operational
‖ Exam version: in use since October 2007many thousands of candidates every year
‖ Online learning environment: online since September 2007ı Edumatic now official electronic test environment
‖ For all Selor domains (including social skills, law, informatics, mathematics, accountancy)
ı Development of decentralized exam facilities ı Added question types
‖ Including voice recording (with scoring by “human”)‖ Including open questions (with scoring by “human”)‖ Including open questions with automatic feedback
(not used in real exams)
Bert WylineAssessment Scotland 2010
Redesigning the computer assisted language examens for federal government employees
Please visit
http://www.edumaticonline.com
Edumatic Exam
Edumatic Mobile