The future of online diagnosis and feedback

J Charles Alderson, Lancaster University

Former Scientific Coordinator of the DIALANG Project

www.dialang.org

Overview

• Online?

• Diagnosis?

• Feedback?

• DIALANG diagnosis and feedback

• Future possibilities?

• Research needed

Online?

• Stand-alone programs

• Interactive, via the Internet, e.g. DIALANG

• Virtual Learning Environments (VLEs)

Online?

• Stand-alone programs: fixed menu of tasks and “feedback”– Correct/ incorrect; Well done!; Try again!– Total score or level

• Interactive, via the Internet– Adjusts test/ exercise depending on student

proficiency level (self-assessment and Vocabulary Size Placement Test) and preferred language of instructions.

– Feedback categorised according to construct

Online?

• VLEs– Downloadable documents and tasks– Synchronous and asynchronous discussions– Tutor comments and feedback to forums– Private tutor comments to reflective journals– Emails from tutor to individuals and groups– Feedback in reaction to individual posts or queries – Tailored to individual?

• Can we learn from VLEs?

Diagnosis?

• Strengths and weaknesses, plus “remediation”• NOT Placement: into “levels”• NOT Achievement: what you have learned in

general• NOT Aptitude: phonetic coding, grammatical

sensitivity, inductive language learning ability• Very rarely truly diagnostic• Poorly understood; never problematised; not

theorised; under-researched; poor-quality tests

Diagnosis

• Based on variables known to predict development or weakness

• Known to be capable of remediation

• In-depth

• One-to-one exploration

• Feedback crucial part of diagnosis

Feedback

• Any communication or procedure to inform a learner of the accuracy of a response

• But also “other information such as learning guidance, motivational messages, critical comparisons, and learning focus”

• ”Any message or display that a computer presents to the learner after a response”

• Aims to improve performance & learning (directly)• But also to motivate, make learning more interesting,

encourage different learning, raise awareness, …(Huhta, 2006)

Automatically Generated Feedback?

• Item Right/ Wrong

• Percent or proportion correct

• Some verbal description

• Allow multiple attempts (“Try again”)

• Immediate vs. delayed feedback

• “Encouragement”

• Total score/ Band level

Diagnosis in DIALANG

• Macro-level = CEF Level by Skill/Language Component

(test-based and self-assessed)= Vocabulary Size

• Micro-level= Subskill level performance= Item level performance

Diagnosis and feedback in DIALANG

Reading sub-skills

• Inferencing (including lexical inferencing)

• Identifying main idea / Distinguishing from supporting detail

• Reading intensively for specific detail

Listening sub-skills

• Identifying main idea / distinguishing from suppporting detail

• Inferencing (including lexical inferencing)

• Listening intensively for specific detail

Writing sub-skills

• Knowledge of accuracy (grammar /vocabulary /spelling)

• Knowledge of register / appropriacy

• Knowledge of textual organisation (cohesion/coherence, text grammar / paragraph organisation)

Vocabulary sub-skills

• Meaning – denotation, connotation, appropriateness

• Semantic Relations– synonymy, antonymy, hyponymy, polysemy

• Word Formation – compounding, affixation

• Combination– collocation, idiomaticity

Grammar• Morphology - Adjectives and Adverbs – comparison• Morphology - Adjectives and Adverbs – inflection• Morphology - Nouns - definite/indefinite articles• Morphology - Nouns - proper/common• Morphology - Numerals – inflection• Morphology - Others• Morphology - Pronouns – context• Morphology - Verbs - active/passive voice• Morphology - Verbs – inflection, tense, mood, person• Syntax - Organisation/realisation of parts of speech –

word order statements, questions• Syntax – Punctuation• Syntax - Simple Sentences vs complex sentences -

coordination

Immediate or delayed item feedback

VSPT feedback

VSPT Feedback in the Prototype

901-1000 A very high score, typical of a native speaker, or a person with near-native proficiency.

601-900 People who score at this level are typically advanced learners, with a very substantial vocabulary. Learners at this level are usually fully functional, and have little difficulty with reading, though they may be less good at listening.

401-600 People who score at this level typically have a good basic vocabulary, but may have difficulty handling material that is intended for native speakers.

201-400 People who score at this level have a limited vocabulary which may be sufficient for ordinary day-to-day purposes, but probably doesn't extend to more specialist knowledge of the language.

101-200 This level indicates a very basic knowledge of the language, probably good enough for tourist purposes or "getting by", but not for managing easily in many situations.

0-100 This level indicates a person who knows a few words, but lacks any systematic knowledge of the basic vocabulary of the language.

Explanatory feedback

Advisory feedback

How to make progress

Ironies

• DIALANG gets up to 5,000 hits per day in peak season

• Used for placement, NOT diagnosis• Freely available• No EU funding to sustain or develop• Post-2004 EU-enlargement governments

have not been persuaded to add their languages to the interface or as a target language

Problems with automatic feedback

• Feedback not tailored to individual performance/ profile of item responses, or background/ age/ mother tongue/ cognitive style…..

• Not clear that item categorisation is true and indicative of development.

• Nothing in CEFR on language-specific lexis or structure.• No relation to theory or empirical research into second

language development.• Don't know how the diagnosis and feedback is received

or acted upon. Yet usefulness is key to validation.• Take it or leave it: no follow-up from the system.

Student and tutor reactions to feedback (Huhta, 2006)

• Overall test results the most popular part of feedback– The fact that the meaning of A1 etc was defined was

appreciated

• Almost 60% preferred immediate review, 30% post-test review, 10% no preference– Why like: want to know immediately, remember

better if see the result right away, want to learn (during test)

– Why don’t like: disturbs test-taking, can be depressing

Student and tutor reactions to feedback

• According to users, self-assessment is important part of (a system like) DIALANG

• as ‘popular’ as item review (joint second)

• Explanatory feedback the least used / useful

• Extended level descriptions about CEFR less used/ useful than item review

• Advisory Feedback quite popular but not among all users

Student and tutor reactions to feedback

• Users clearly like to have feedback about their self-assessment– younger learners may need to be motivated first

• Good to have choice of two types of item review: immediate and post-test

• Advice appears useful for beginners in particularBut challenges:– “I don’t bother to read the advice because I know I’ll do things

my way anyway”– “... you don’t remember them any more when you study”

• Value and meaningfulness of diagnosis?• Need longitudinal studies• (Long-term) effectiveness of feedback?

Can diagnosis and feedback be enhanced in a DIALANG-type

environment?

i.e. be “automatically” generated, yet with tailor-made help, hints, clues,

feedback and explanations?

Ideas for improved items and feedback

• The following slides refer to several experimental DIALANG items available at

• http://www.lancs.ac.uk/fss/projects/linguistics/experimental/start.htm

• Readers are advised to visit that site, and take each item type before reading each of the next five slides

Item 2: Following instructions on a plan or map

• Feedback could include a map on which the route to the correct room(s) is displayed.

• If the learner had clicked on a room that was in a totally wrong part of the floor, the feedback could remind him / her of the point in the instructions which the learner had apparently missed (e.g. "You were asked to turn right after the lift, not to the left.").

Item 3: Video input

• Learners could be offered a chance of viewing again either the whole clip, or the section which gives the answer to the question he / she answered incorrectly.

• If the learner still cannot answer the question, a written transcript could be made available, possibly with key points highlighted.

• In the next step, access to a dictionary could be provided for the learner, and finally, the correct answer(s) could be disclosed.

Item 4: Drag and drop boxes, with clues

• Feedback can also include the number of attempts which the learner needed to arrive at the correct solution and the number of times the learner listened to the audio clip. The learner could also be shown visually how part of the construction is to be completed or where there are missing pieces in the construction.

Item 14: Drag and drop matching: Prompts another try

• Learners could be allowed access to a dictionary or to hints (e.g. questions, definitions) after a certain number of unsuccessful attempts. The number of attempts could also be reported to the learners.

Item 13: Self-assessment in speaking

• Learners could be asked to record their answer on tape or on the computer itself and then listen to their answer before proceeding to the samples. Hearing one's own response, rather than imagining what one might say, may make the comparison easier and result in more accurate self-assessment.

Need for research into diagnosis

• What to diagnose?

• What components of the skills are essential to further development?

• How does language develop across the CEFR levels?

Applied linguistic research

• Psycholinguistics of FL learning• Grammatical and lexical errors, vocabulary

size and depth• Proficiency in other languages, language

awareness• Influence of mother tongue, age, learning

history, aptitude, motivation, cognitive style, personality?

Learner corpora

• Specially designed, not ad hoc• Learners’ CEFR level pre-assessed• Performance on common, calibrated tasks• A variety of mother tongues, ages, etc• A variety of target languages• SLA-type research into features predictive of

CEFR level• Development of diagnostic tools using such

indicators

Source of insights

• Teachers’ ‘Diagnostic Competence’ (Edelenbos and Kubanek-German, 2004) – "the ability to interpret foreign language

growth in individual children"

• What do teachers do in class when giving feedback?

• What strengths and weaknesses do they concentrate on and what evidence has diagnostic potential?

Advantages of understanding teachers

• “Since teachers are usually the ones who work most closely with learners, it makes sense to look at how they go about assessing their learners' strengths and weaknesses, and to explore what we can learn from them about diagnosis.” (Alderson, 2005b)

• “Procedures for dealing with children’s first language reading problems were evolved by observing teachers at work, challenging, discussing and consulting in an effort to link teacher and pupil behaviours with theory about the reading process.” (Clay, 1979)

• Our understanding can be enhanced by studying how teachers diagnose and feedback to learners both in classrooms and in VLEs: an ideal data-gathering instrument.

Advantages of online diagnosis

• Provision of individualised assessment, immediate feedback, advice and even follow-up.

• Learner could input details of relevant personal characteristics like first language, age, learning strategies, motivation, learning history, and so on, and be presented with a diagnostic assessment tailored to such backgrounds.

Source of insights

• We need a diagnostic framework that can interface with the CEFR, or a future revised version, that can help us to explore how learners develop from one CEFR level to the next and how we can best diagnose problems in such development.

Advantages of online diagnosis

However:

• “Before that day comes, we will need to have developed a much better understanding of foreign language development. If we can then incorporate such understandings into assessment and make them useful to learners through the provision of meaningful and useful feedback and follow-up, then diagnosis will truly have become the interface between learning and assessment." (Alderson, 2005a:268).

What do I teach?

• What do I concentrate on and why?• How do I give feedback and what sort?• What do I diagnose? What can I

diagnose? • Can I give more meaningful feedback?• Can this be incorporated into online

diagnosis and feedback? • Can learners choose their own feedback?

Can they explore what is available?

What do I teach?

• Which language aspects do I notice/ comment on, at which level?

• Which language aspects are most amenable/ resistant to feedback?

• What sort of feedback do my students

a) demand? b) pay attention to? c) find ultimately helpful?

Diagnosis

• Based on variables known to predict development or weakness

• Known to be capable of remediation

• In-depth

• One-to-one exploration

• Feedback crucial part of diagnosis

EmbeddingBennett foresees three phases of innovation in e-

learning:

1. Doing in e-form what we can already do in other media.

2. Enhancing e-learning through experimentation and technological advances

3. Seamless embedding of assessment within learning activities (Bennett, 1998)

Then, diagnostic assessment procedures would be “indistinguishable from learning procedures, and we would indeed have developed tests with learning validity.” (Alderson, 2005a)

The future of online diagnosis and feedback

????

ReferencesAlderson, J. C. (2005a) Diagnosing foreign language proficiency: The interface

between learning and assessment. London: Continuum.Alderson, J. C. (2005b) The challenge of (diagnostic) testing: do we know what

we are measuring? Plenary presentation at LTRC, Ottawa, JulyBennett, R.E. (1998) Reinventing assessment: speculations on the future of

large-scale educational testing. Princeton, New Jersey: Educational Testing Service.

Clay, M. M. (1979) The early detection of reading difficulties: A diagnostic survey with recovery procedures. Auckland, N.Z.: Heinemann.

Edelenbos, P. and Kubanek-German, A. (2004) Teacher assessment: the concept of 'diagnostic competence'. Language Testing, 21(3), 259-283.

Huhta, A. (2006) Users’ reactions to innovative computerised feedback – the case of DIALANG. Paper presented at EALTA Conference, Krakow, May.