Goal-oriented visualizations of activity tracking: a case study with engineering

students

José Luis Santos, Sten Govaerts, Katrien Verbert, Erik Duval

1

http://www.slideshare.net/jlsantoso

2

Motivation

1. Means: Information visualization

2. Goals1. Awareness2. Self-reflection3. Sense-making

but even more important…

to engage students on these processes.

3

LAMS DOKEOS

KHAN ACADEMYMOODLE

Related work

4

1. Upton et al. Narcissus: Group and Individual Models to Support Small Group Work. UMAP’09 http://goo.gl/ZzNbF2. Derick et al. GLASS: A first look through a Learning Analytics System. LAK’12 http://goo.gl/eY4w23. Mazza et al. Exploring usage analysis in learning systems. Workshop at AIED’05 http://goo.gl/ycrFQ

5

1. Santos et al. Visualizing PLE usage. EFEPLE’11 workshop. http://goo.gl/PWbBF2. Govaerts et al. The student activity meter for awareness and reflection. ACM SIGCHI 2012. http://goo.gl/SGfta3. Duval et al. Learning dashboard and learnscapes. Workshop at ACM SIGCHI 2012. http://goo.gl/j7NeJ 4. Duval. Attention please! Learning analytics for visualization and recommendation. LAK’11. http://goo.gl/Kfwii

7

Design based research methodology

Design Implementation Deployment

Observation & Evaluation

AnalysisRequirements

One… two… three… and… four!

1. Paper prototype– Professors and teacher assistants.

2. Digital prototype– Professors and teacher assistants.

3. First release– Students

4. Second release– Students

8

1st and 2nd iterations

9

-Generic (CAM)-Usability-Think-aloud protocol.-15-30 minutes- 1st -> 7 participants- 2nd -> 5 participants

Results

10

• Size of tables• Redundant information• Personalization• Context centralization vs diversification• Motion chart

11

Scenario

Awareness and reflection

Goal-oriented

How do students spend their time during the lab sessions?

Why and what do we track?

• Individual and group spent time • Type of activity

(development, writing, …)• Application• Program code• Websites

12

+

How do we visualize this data?

13

http://role-sandbox.eu/spaces/dashboardkulor

http://goo.gl/qzP9H

http://sourceforge.net/apps/mediawiki/role-project

14

15

16

17

18

19

20

21

First evaluation with real data

• 36 students• Questionnaire after seeing a demo

– From USE questionnaire• Perceived usefulness • Privacy concerns

22

23

Conclusions

24

CAM Dashboard will help me to understa

nd how I use my tools

CAM Dashboard is useful

CAM Dashboard will help me to

be more effective

CAM Dashboard is easy to

use

Using CAM Dashboard is

effortless

I am sa

tisfied with

CAM Dashboard

CAM Dashboard is fun to

use

I like to

see what o

ther members d

o during th

e course

I feel confident u

sing th

e tracking syste

m in the P&O lab

during th

e course

I would also

feel confident usin

g the tra

cking system

outside of th

e P&O lab during th

e course

• Useful• Helpful• Lab tracking• Usability?

Second evaluation with real data

• 10 students• 4 lab sessions• 15 minutes• SUS• Perceived usefulness • Tracking/privacy concerns.• Questionnaire

25

26

Conclusions

• SUS score: 72 • Useful?• Detection of

certain patterns

27

Conclusions

28

Final conclusions

• Not included in this paper• 14 students• SUS: 66 • Useful for:

– Teachers (6) – Students (5)– Both (3)

• Satisfaction:– 4.4 (over 7)

Although they consider it useful, understandable, usable…

29

…students did not use the tool frequently

30

WHY?

One student said: “We had other priorities”

31

How can we engage students in this process?

32

33

Thank you! Any question?

@jlsantoso@erikduval@katrien_v

http://jlsantoso.blogspot.com

http://erikduval.wordpress.com34