• Divergent opinions about the usefulness and understandability of the

course report: activity of students by step and responses in the quizzes

and tests.

• Students´ satisfaction and comments by step were among the most

useful information for the presented monitor goals as in [2].

How educators value data analytics about their MOOCs

Introduction Results

Research focus

Methods Conclusions and future work

Konstantinos Michos, Davinia Hernández-Leo & Manel Jiménez

{kostas.michos, davinia.hernandez-leo, manel.jimenez}@upf.edu

ICT Department, Communication Department, Universitat Pompeu Fabra, Barcelona

References 1. Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U.: Supporting action research with learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 220-229). ACM (2013).

2. Stephens-Martinez, K., Hearst, M. A., & Fox, A.: Monitoring MOOCs: Which information sources do instructors value?. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 79-88). ACM (2014).

• Data analytics provided to educators in MOOCs: e.g. students´ profiles,

students´ online behavior, students´ satisfaction with the course.

• Supporting the role of educators, their decisions and their actions

based on the data analytics [1].

• Learning design & Data anaytics

.

• Survey study with closed and open-ended questions.

• 4 Educators who developed two editions of three different MOOCs in

UPF.

RQ1: Which information sources and visualizations (from FutureLearn

reports) are most useful for MOOC educators?

RQ2: What information sources (from FutureLearn reports) help MOOC

educators to identify problems and potential improvements in a redesign

of the course?

• Problem: How do educators value the different sources of data and

how do they connect them with their course re-design?

• FutureLearn case – Summary Reports about: a) pre-course survey

b) course report c) satisfaction surveys.

• Questions about the usefulness and understandability of the different

provided data analytics from their own MOOCs. Connection with

monitoring goals as in [2]: 1. Problems with the activities-tasks.

2. Struggling students and what they are struggling with.

3. Difficulty of the grading activities.

4. Appropriateness of course difficulty level for students.

5. Engaging content for students.

6. Most difficult part of the course.

7. Improving the presentation of a topic.

8. Least interesting content for students.

22

22

28

22

28

22

25

19

41

28

41

47

59

44

34

38

34

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

a. Previous experiences of students

b. Enrollement cummulative growth

c. Activity by step

d. Comments by step

e. Quizzes and tests

f. Students' satisfaction

Percent of educators’ responses (n = 32)

Info

rmat

ion

so

urc

es

Usefulness of the information sources for all the monitoring goals

Neutral Agree Strongly agree Not applicable

29

29

25

29

29

25

25

13

4

13

8

4

17

33

25

38

29

33

21

13

42

29

58

29

29

33

38

63

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

1. Problems with the activities-tasks

2. With what students struggle

3. Difficulty of the grading activities

4. Appropriatness of course difficulty

5. Engaging content for students

6. Most difficult part of the course

7. Improving the presentation of a topic

8. Least interesting content

Percent of educators’ responses (n= 24)

Mo

nit

ori

ng

goal

s

Usefulness of the information sources for each of the monitoring goals

Neutral Agree Strongly agree Not applicable

• Εducators redesign actions based on their availble data: simplification

of the content. Interventions in specific parts of the course.

Redesigned quizzes and tests based on the levels of shown difficulty.

• Final decissions were determined after discussion between the groups

of educators and the FutureLearn platform.

• Proposal for interaction analysis and words in sentiment analysis.

• Preliminary results indicated that educators perceived specific data

analytics particularly useful, with some divergences in their opinions.

• Difficulty to connect the information sources with the monitoring goals.

• Extend the study with more educators.

• How topics, learning design and audience

characteristics influence the educators.

• Monitoring goals defined by educators

before their course.

• Study in detail the educators’ actions

based on the presented data analytics.

• Take in to account educators’ divergent

needs in the design of MOOC dashboards.

During the course: monitoring, interventions, dynamic changes

in the course design, prompts in the forums.

After the course: reflection on what happened to improve re-

runs of the course. Decisions in design-time based on past

cohorts of MOOC students