The Cost of Authoring with a Knowledge Layer

Judy Kay and Lichao Li

School of Information Technologies

The University of Sydney, Australia

The Problem and Motivation• Programming is cognitively demanding, even at the most introduct

ory level;

• We want to help our students become more competent programmers.

• It has been found that student self-assessment can improve students’ metacognition, which can enhance learning;

• Learner reflection is “a generic term for those intellectual and effective activities in which individuals engage to explore their experiences in order to lead to a new understanding and appreciation”;

– There is strong evidence suggesting that more efficient and effective learners pay more attention to their own learning experiences by reflecting on the state of their knowledge and the learning process;

– Learner reflection is especially important for cognitively demanding learning topics, such as learning programming;

Our Solution: ASSESSAssess is a programming education system that facilitates student self-assessment, encourages learner reflection and provides adapted learner feedback.1. Exercises for self-assessment take the form of tasks. Each task has

a programming problem that students need to answer. The system allows students to provide solutions to these problems and self-assess their solutions against a set of marking criteria provided by task authors.

2. More importantly, it provides students with example solutions to assess. These example solutions have been pre-assessed by authors. The student is meant to evaluate them using pre-defined marking criteria and compare her assessment to the task author’s assessment of the same example solution. The comparison gives the student feedback on: how the teacher marked the example, the difference between the teacher’s assessment and the student’s assessment and why the teacher marked it that way.

3. This information is used to update the system’s belief of students’ learning progress, which can be viewed in the student’s user profile.

Problems of AssessThe original version of Assess allowed students to write code and self-assess their solutions, as well as read example solutions and assess them. However, as the system lacked any representation of domain knowledge, its nature can be best described as ad-hoc.

ASSESS with a Knowledge LayerIn the new version of Assess, all the ad-hoc elements were replaced by a systematic knowledge layer that defines the learning objectives, user model elements and a domain ontology.

To accommodate this new layer, the process of exercise authoring was changed considerably. We present the new task authoring process and report a preliminary experiment that aims to evaluate the usability of the new authoring interfaces and show its results.

Exercise (Task) Authoring

1. Provide learning objectives to the system

2. For each task in the system:• Create a task statement and

associate learning concepts with the task;

• Edit automatically generated marking criteria and;

• Create, edit and assess example solutions.

Authoring Interfaces

Learning objectives need to be set up in the system. They are concepts to be taught/learned, defined by teachers according to the learning outcome requirements of a subject. They are what the system attempts to teach and are used to specify teaching/learning goals of individual exercises.

Authoring Interfaces – Stage I

In the first stage, an author provides the problem statement and, optionally, a skeleton answer to help the student. They also need to indicate the difficulty of the problem and specify whether to force the student to save and assess their own answer before viewing the example solutions. More importantly, the author must relate the task to relevant learning objectives. Each chosen learning objective represents a teaching/learning goal of the task. A task must have at least one goal associated with it to denote what it aims to teach.

Authoring Interfaces – Stage II

The second stage in task authoring is the editing of marking schemes. A default set of marking schemes is automatically generated based on the teaching goals for the task. As a result, when students assess their own solutions and our example solutions with them, they should concentrate on the task’s learning goals.

Authoring Interfaces – Stage III

The author can edit the example solution’s marking schemes. All example solutions of a task share a same set of marking schemes that were created in Stage I and II. However, it is possible for each solution to have additional marking schemes. This means that each task has core teaching goals but each example solution can have additional elements.

Stage Three involves creating, editing and assessing example solutions. A new example can be created at this stage.

Authoring Interfaces – Stage III (2)

Authors can assess the solution with the complete set of marking schemes and provide an explanation for the assessment. When a student assesses this example solution, her assessment is compared with the author’s assessment. This discrepancy indicates how well she understands the learning concepts associated with the marking schemes, and is recorded in her individual learner model to provide adaptive learning feedback.

Evaluation – Goals

• How quickly and accurately a novice tutor can create a task, in particular;– Can find appropriate learning

concepts for a task;– Can edit task marking schemes

and;– Can create, edit and assess

example solutions;

• How effective and usable are the interfaces;

Evaluation – Setup• Think-aloud Experiments

• There were five participants;

• They have seen Assess from a student’s perspective, so they know how the system works. This allowed participants to get some insight into Assess, but did not bias the experiment by letting them see what they would need to do;

• We supplied the participants with materials they need to put into the system, as we were not trying to determine how people tackle the more intellectually demanding task of choosing a task and providing solutions;

• They need to create a task with the given material.

Evaluation – Results

1 2 3 4 5

Sta

ge

1

Sta

ge

2

Sta

ge

3

Over

all0

5

10

15

20

Time (mins)

User

Stages

Task Creation Process Timing

Stage 1

Stage 2

Stage 3

Overall

0

1

2

3

4

5

6

Effort

1 2 3 4 5

User

Perceived Intellectual Effort in Task Creation

Stage 1

Stage 2

Stage 3

Overall

The level of intellectual effort required to create a task in Assess with a knowledge layer is acceptable

The time required to create a task in the new Assess system is around 15 minutes for a novice author

Discussion• At first, one might imagine that the addition of a

knowledge layer to a conventional learning tool might require more effort from task authors.

• From our experience, it seems that the knowledge layer may actually reduce the work of defining a new task and improve the quality of the task as the learning objectives are explicit so helping authors concentrate on what they want students to learn.

• The marking criteria in the new Assess system are automatically generated and they correspond to the teaching/learning objectives of the task. This means that authors do not need to create the questions that test the learning objective for the task and avoids the risk of forgetting to include them.

• Furthermore, the knowledge layer makes it feasible to provide learners and authors with learner models that capture learning progress, supporting reflection.

ConclusionWe have presented Assess, a student self-assessment tool, and explained the exercise authoring process of the system with a consistent knowledge layer. We also described a user trial, with results showing that:– The intellectual effort required to enter

a new task to Assess is modest and the interfaces that allow the creation of a new task are relatively usable;

– The entire authoring process in Assess takes about 15 minutes, which again shows the effort and time involved, introduced by the knowledge layer, are minimum.