Visualizing Student Feedback:

Margus Niitsoo, Kaspar KruupUniversity of Tartu

a case for mixed quant-qual approach

This talk

● Quantitative feedback – Why?

● Visualization? What for?

● Feedback for curriculum level?

My perspective● as a student

– graduated just 2 years ago

● as a lecturer– 5 years teaching experience

● as the curriculum manager– past 2 years

● as an engineer– MSc and PhD in Computer Science

Qualitative feedback● “Stricter deadlines would have

disciplined us to study more!”

Qualitative feedback● “Stricter deadlines would have

disciplined us to study more!”

● “It was hard to hear the lecturer”

Qualitative feedback● “Stricter deadlines would have

disciplined us to study more!”

● “It was hard to hear the lecturer”

● “The course sucked!”

Qualitative feedback“The lecturer was one of the best I have seen in my studies”

vs

“The lecturer was hard to follow, monotonous, inept and boring”

Which is (more) true?

Quantitative feedback

The averaged student rating for the course was:

4.2

Quantitative feedback

The averaged student rating for the course was:

4.2 / 5

Quantitative feedback

The averaged student rating for the course was:

4.2 / 51 2 3 4 5

Quantitative feedback

The averaged student rating for the course was:

4.2 / 51 2 3 4 5

Respondents: 14/100

University of Ontario

University of Amsterdam

All in all, I give this class the following grade:(1=very bad, 5=excellent)

An often unanswerable question

Does the score of 4.2 mean that I am above average?

Add the averages (UT):Institute avg | Faculty avg | Uni. avg

Next problem

Does the score of 4.2 mean that I am above average?

By how much?

Easy!

The institute average was4.0 with SD 0.3

Easy?

What (the hell) doesSD mean?

The average for courses was 4.0 with SD 0.3

Standard deviationCourse avg: 4.2Avg. for courses: 4.0SD for courses: 0.3

Standard deviationCourse avg: 4.2Avg. for courses: 4.0SD for courses: 0.3

Quick calculation:(4.2-4.0)/0.3 = 0.66

Standard deviationCourse avg: 4.2Avg. for courses: 4.0SD for courses: 0.3

Quick calculation:(4.2-4.0)/0.3 = 0.66

Translation:Better than ~70%

We are bad with numbers!● Analysis of numbers is slow ● It is hard (requires effort)● We are prone to mistakes!

We are bad with numbers!

Also: a picture is worth a thousand words!

(We can represent more with less)

● Analysis of numbers is slow ● It is hard (requires effort)● We are prone to mistakes!

The UT solution

The UT solution

(University of Amsterdam)

The UT solution

The UT solution

The UT solution

The UT solution

Pros● Good quick overview

– With details also available, if needed

● Better information– Ranking and distribution implicit

● Emphasizes important– Comparison with mean and other courses

● Increased interest in feedback– More visually attractive and accessible than numbers

Some research:● On-line survey among students

– 107 respondents– 4 tasks of feedback analysis + opinion survey– 3 groups for each task

● Tables with numbers, Simplified graph, Full graph

● Main findings– Graphs took less time to process in analysis tasks– Most preferred graphs over numbers

(but some had a strong preference for tables with numbers so both needed)

– Workload question result often misinterpreted (since the “best” response for that question is 0 instead of 2)

(this is joint work with Kaspar)

Curriculum management● Course planning

– What to teach when– Who teaches what

● Quality assurance– Identify bright and problem spots– Spread good practices– Aid lecturers with problem spots

Good overview essential!● Course planning

– What to teach when– Who teaches what

● Quality assurance– Identify bright and problem spots– Spread good practices– Aid lecturers with problem spots

Standard overview (UT)

Problems (as before)● Absolute values

– Is 3.2 good or bad?

● Just one value– Leads to oversimplification

● Mostly numbers– Hard to get an overview

Proposed solution

(Inspired by heatmaps used in Bioinformatics)

Summary● Quantitative feedback has a use

– Finding the bright spots and problem areas

● Presentation of data is important– Although sadly this is often neglected

Further work● Qualitative research into the

perceptions of the academic staff of the new system

– Currently in progress with a larger team

● Good review of systems currently used throughout the world

– Does your uni. have something better?

Thank you!

Any questions and comments would be

welcome!

(If you have any later, try Margus.Niitsoo@ut.ee)