Learning From Assessment: Evaluating the Benefits of DALI (Diagnostic Assessment Learning Interface)

Hershbinder Mann & Guinevere Glasfurd-Brown,

University of Essex

2006

Background and Context• The DALI Project is a Teaching and Learning Innovation Fund (TALIF) pilot study at the University of Essex.

• Development of the student portal and online formative assessment resources as part of the SPRInTA Project.

• Online formative testing in three large Departments, primarily to address specific objectives:

• large class teaching: poor performance, high drop-out rate, lack of engagement with case study materials

• threshold testing

• improving feedback (and opportunities for feedback) to students (Nat Student Survey; SSS)

Issues that emerged from past practice

• Student experience• Aims of the learner? (motivation; hoe might we

improve engagement?)• Is there a gap between what students expect

from online formative assessment and what they get/experience?

• How can we encourage a dialogue between staff and students on assessment?

• How might this inform the guidance we provide to staff?

Background and Context

‘Learners have different priorities, preferences and approaches to learning, and different requirements for support. The learning environment needs to reflect these differences. Understanding how different learners experience the tasks, resources and services offered to them is an important precursor to developing effectively personalised systems.’

- Understanding My Learning, JISC

DALI Project

• DALI tool (issues) – enables students to revisit and query assessment data and identify areas of weakness (related to specific learning outcomes at topic level)

• Survey of students• Assessment Think Tank – student focus

group• iLAB session which brought staff and

students together to discuss assessment.

DALIDiagnostic Assessment Learning Interface

DALI is an add-on to Perception. It focuses on the provision of assessment feedback and provides students with information aimed at giving them greater control over their own learning.

The interface:

• allows a student to select the score level they wish to achieve;

• highlights the last topic-based feedback provided to the learner;

• provides statistics associated with any selected topic; and

• displays topic descriptions as learning objectives.

Using DALI

Aims• To identify the factors that affect the uptake of computer-aided assessment among students

• Strengths and weaknesses from a student perspective

• The importance of feedback

• Good / bad feedback

• What should feedback do?

• How can this be achieved?

• The role of DALI

• Guidelines for users

Data Collection

• Participation from three University Departments

• Accountancy and Financial Management, Electronic Systems Engineering, and Biological Sciences.

• Online student survey

• Student focus group

• Staff and student iLab meeting

DALI Project Survey - Results

• Identifying my strengths and weaknesses in assessment is important to me.

• It is important for me to be able to track my own progress.

• The online practice tests are a useful learning tool.

DALI Project Survey - Results

• The tests help me learn more effectively.

• I would take more tests if available.

• I pay little attention to answering the questions correctly.

DALI Project Survey - Results

• I use the tests mainly for the feedback.

• The feedback is what I expected.

• The feedback on the questions could be improved.

DALI Project Survey - Results

• The survey also invited respondents to add additional comments relating to feedback and to online assessment more generally.

• These comments were categorised and placed in one of the following groups:– Feedback, Knowledge/Learning, Revision/Planning,

Accessibility, Technology, Convenience, Question Format, Self-Test, Time, Motivation, Memory, Relevance, Other.

DALI Project Survey - Results

Q18. Positive Comments (Grouped by Category)

0

2

4

6

8

10

12

14

Category

DALI Project Survey – Results

Q19. Negative Comments (Grouped by Category)

0

1

2

3

4

5

6

7

8

9

Category

Think Tank

• As a supplement to the survey, a focus group was assembled. Participants were asked about their experiences of online assessment.

• Questions concentrated on likes/dislikes, reasons for doing the tests, and views on the feedback received.

Think Tank ResultsComments Made in Think Tank (Grouped by Category)

0

5

10

15

20

25

30

35

Category

Focus Group Findings

• The focus group corroborated the survey findings.

• Positives– Students perceive the benefits of using online

assessment in relation to learning, comprehension, and revision, and were pleased to get instant feedback.

• Negatives– Feedback is vague, access is difficult, questions are

not engaging enough, and the questions are not relevant.

iLab Session – Key ResultsStudent comments:

• Revision

• Personal strengths and weaknesses

• QMP is not used to its full potential

• Feedback could be enhanced to make it more relevant to the individual

Staff comments:

• Time, workload, and support

• Developing assessments and question banks is time-consuming as is writing effective feedback

• The ability to extend and update tests is very useful

Conclusions• Project findings have been used to inform a ‘good practice’ staff handbook on online formative assessment.

• The project encountered difficulties with the DALI interface and this is under investigation.

• The project was instructive in highlighting the importance students place upon effective feedback and the potential to personalise their learning.

• The University is developing an ‘intelligent’ assessment environment which has the ability to generate personalised learning profiles.