An ‘open research’ approach

Thieme Hennis – thieme@hennis.nl

Pieter de Vries - Pieter.deVries@tudelft.nl

TU Delft Online Learning – presentation LAK’15

• Established: 1842 • 15 bachelor programs• 33 Masters programs• 8 faculties• 18,000 students

(3,000 from abroad)• >2000 PhD-students• #66 ranking THIS

Environment

Infrastructures

& mobility

Health

Delft Research Initiatives

Energy

• Moral obligation• growing demand

in higher education worldwide 2012-2025:

80 million!• Quality

• improve our materials, teaching methods

• Reputation• to be there with

the other top universities

• Innovation• digital and online

education inevitable

In 2013…

Next GenerationInfrastructures 2

Drinking Water Treatment

FunctionalProgramming

Delft DesignApproach

Technology forBioproducts

Solving Complex Problems

ResponsibleInnovation

Treatment of Urban Sewage

Introduction toWater & Climate

Introduction toSolar Energy

AeronauticalEngineering

Credit Risk Management

Economics of Information Security (ProfEd)

Transport Phenomena: mass, heat and momentum

Topology for Condensed Matter: Untying Quantum Knots

Framing. Communicating your message in the world of politics

Take your Data Analysis to the MAX

This year 5 more NEW MOOCs..

“Registered” MOOC students

Total ~ 300.000Nov’14

ProfEd, Highschool initiative,

blended/flipped, online MSc education

Pre-University Calculus

Economics of Cybersecurity

TU Delft Online Learning

• Open education• Open courseware (since 2007 – still growing)• 15 MOOCs (since 2013) – 6 reruns, blended MOOCs

• Online distance education – MSc (since 2013)• Professional education

• 2 studios, 1 mobile studio• 6 instructional designers• Marketing and business development people• Media center (simulations, presentation quality, videos)

• TU Delft Online Learning research

Why Open & Online

education?• MIT does it..

• Altruistic reasons

• New revenue models• Globalization

• Scaling up education

• Flexible education (profEd / LLL)

• Innovationonline <> campus

capacity building

research

Questions questions questions

• Can open education improve campus education?

• How to assess and certify people online?

• How to create more inclusive courses?

• How to facilitate interaction between students?

• Is peer-grading a valid approach? And how to do that?

• How to create personalized learning experiences?

• Etc.

Level Topics of interest and stakeholders

Strategic University role into the future, new students, reputation, business models, ROIDeans, DelftX, MT

Organizational Efficient workflows and procedures, support structure, documentationCourse team, instructional designers, NMC, DelftX, marketing

Course Course design, pedagogy, use of media, methods and tools, etc.Course team

Technology/edX Design of tools for learning, teaching, and data analysis/visualizationedX consortium, NMC

Students Learner demographics, background, performance, engagement, etc.Students

Level Topics of interest and stakeholders

Strategic University role into the future, new students, reputation, business models, ROIDeans, DelftX, MT

Organizational Efficient workflows and procedures, support structure, documentationCourse team, instructional designers, NMC, DelftX, marketing

Course Course design, pedagogy, use of media, methods and tools, etc.Course team

Technology/edX Design of tools for learning, teaching, and data analysis/visualizationedX consortium, NMC

Students Learner demographics, background, performance, engagement, etc.Students

How to answer these questions?*

Don’t: teaching is an art, not a science

Hire researchers

Machine learning

Open up the research process

* limits: few resources and little expertise

How TUD answers these

questions:

Don’t: teaching is an art, not a science

Hire researchers

Machine learning

Open up the research process

Hence…

Great opportunities• … to research MOOCs • … and use MOOCs for research

But..• Limited capacity• No/little history in educational research at

university

Our approach→ connect with relevant researchers worldwide to answer our questions

The currency: access to data +

research instruments

Open research…

Open research…

Facilitating research activities

1. …with relevant parties globally and locally

2. …on a broad range of relevant topics

3. …by giving them access to our data and research

instruments

4. …and managing the collaboration workflow

5. …and publishing the results together

1a. Who are these ‘relevant parties’?

• MOOC Teachers / course teams

• MSc. students / PhD’s

• Researchers from partner institutes

• Researchers globally with expertise on a specific,

relevant topic

1b. Our current ‘MOOC research

network’

• Stanford University - Psychology• UC St. Barbara - Psychology• University of South Australia - Pedagogy, coll. learning• Edinburgh University - Communities of Inquiry• Harvey Mudd College - Gender, diversity• TU München - Gender, diversity• Memphis university - Linguistics• UVA - Methodologies

• TU Delft - Privacy/ethics- Course-relevant topics (researching course teams)

2a. The ‘broad range’ of relevant

topics

• Learner demographics, pedagogy, MOOC forums - UNISA

• Does self-affirmation of personal values increase engagement of at-risk students

(motivation research) – Stanford, UC St. Barbara

• (How) does communication in MOOC forums influence social centrality and

performance? – Simon Fraser Univ.

• How well does the Community of Practice framework fit learning in a MOOC context?

What effects of social and teaching presence on social networks and collaboration? –

Edinburgh univ.

• Why do female students have lower performance in our MOOCs? How to design

socially inclusive courses? – Harvey Mudd College & TU München

• Are concerns over privacy rising? And what implications does it have for online

learning? – TU Delft master student

• Etc..

2b. How to choose research

topics?

Research topics

Literature, developments

Partner interest

Teachers / support staff

Own research

3a. Provide access to data &

instruments

Data sources Research & evaluation

edX subscription data Number of participants, dropouts, location, age, gender, schooling

edX student data, learning analytics Progress, tests results, quizzes, exams, etc.Forum participationVideo clicks, navigation

External data, other media (i.e. Facebook groups, discussions)

Social networks, content and discourse analysis

Surveys (pre, mid, post) Information about demographics, intention,expectations, satisfaction, media use, etc.Interventions embedded in surveys

Interviews / self-assessment: teachers, NMC,DelftX

Experiences with workflow and organizationQuestions and expectations for evaluation

3b. Would you be willing...

We are doing a lot of exciting research at Delft University of

Technology. Would you be willing to receive information about

participation in one of our future research projects? This could be

a survey, interview, or something entirely different (building a

weather balloon?). Let us know by clicking yes below, and we may

send an email to you when we need you.

★ Yes (1)

★ No (2)

4a. Collaboration workflow

4b. Process and involvement

Step Me Partner TUD

Research idea! (or problem) x x x

Intake x x

Provide overview of data & instruments x

Signing MoU, Eth. clearance, abstract + meth. x

Preparing instruments x x x

Deploying instruments x x

Collecting and sharing data x

Analysis x x x

Feedback (papers, guidelines, code, data) x x x

4c. Integration… some…

• Evaluation feedback loop happens, but locally (iteration of MOOC, blended experiments)

• First steps into Research to Practice feedback loop• Teacher training• On-boarding days• Self-assessment tool teachers• Talking with course design

teams (example: assessment)

5a. Selection of publications

• 6 Working papers: 5 course reports + 1 comparative analysis paper

• Understanding Social Learning Behaviors (SEFI’14 - Skrypnyk, Hennis, De Vries, 2014)• Learners’ Social Centrality and Performance (EDM’15 – Dowell, Joksimovic, Skrypnyk,

Hennis, De Vries, 2015)• An exploratory study in the concerns for information privacy (MSc thesis - Hassing,

2015)

• Who is the Learner in the DelftX Engineering MOOCs? (SEFI’15 - Hennis, Skrypnyk, De

Vries, 2015)

• Diversity in Engineering MOOCs, a first Appraisal (SEFI’15 – Ihsen, Yves, Hennis, De

Vries 2015)

• Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy

(EMOOCS’15 - Skrypnyk, De Vries, Hennis, 2015)

5b. Open data?

• More difficult…

• Privacy concerns

• Semi-open data? (only edX

partners?)

Potential of ‘open research’

• Being highly flexible in what

you research

• Being able to scale the

research efforts

• Research network and learn

from their expertise

• Interdisciplinary research

Current work

• Streamline the process

• Awareness of research potential

• Building capacity for cleaning and preparing data

• What about data standards? MOOCDb – xAPI – IMS Caliper

etc.

• Enabling research across courses / MOOC providers?

• More focus on design based research: involvement of

teachers

• Involvement partners: from hub to community

Additional ideas:

• Tender for MOOC research

• Promote TU Delft research areas

• Citizen science / crowdsource research models