streamlined orchestration: investigating the impact of organised orchestration on teaching

Streamlined OrchestrationInvestigating the Impact of Organised Orchestration on Teaching

Lighton Phiri

Supervisors:Prof. Dr. Christoph Meinel and A/Prof. Hussein Suleman

August 21, 2015

Introduction

■ Exploring technology-driven orchestration◻ Understand whether and/or how organisation of learning activities

influences educators’ effectiveness■ Thesis goals

◻ Investigate the impact of an orchestration workbench on teaching◻ Investigate successful use of an orchestration workbench

2

Motivation

Supporting educators considered one the of most effective ways of improving the quality of education

However, contemporary orchestration is:1) Challenging

◻ Heterogenous activities—instructional and non-instructional◻ Timing constraints—typical lecture session lasts ~45 minutes

2) Ad hoc◻ Specialised tools used for managing activities◻ Generic tools appropriated to achieve desired goal

3

Inspiration

■ Inspiration drawn from success of organisation in other domains◻ Developer IDEs for tool and service integration◻ Scientific workbenches for managing workflows

4

Streamlining Orchestration (1)

Orchestration workbench concept■ Organisation of session activities■ Facilitate sequencing of activities

5

■ Single unified view

■ Multi-tool and service intergration

Streamlining Orchestration (2)

Thesis statement:Streamlined orchestration—attainable through explicit organisation of enactment activities using an orchestration workbench—could potentially make educators more effective.

Research questions:1) Does an orchestration workbench enable educators to

become more effective?2) What is the impact of an orchestration workbench on

educators’ teaching experience?

6

Evaluation Plan (1)

‘5+3’ Conceptual Orchestration Framework■ Management

◻ Coordination of learning activities

◻ Regulation of activities to guide learning activities

■ Flexibility◻ Flexibility of tool

■ Assessment◻ Learning experience

7

Evaluation Plan (2)

8

# Evaluation Technique

1. Case studies

2. Experiment in the large

3. Targeted group

4. Survey of existing tools

5. Captive audience

6. General random trial

7. Survey of teaching staff

8. UCT Lecture recording analysis

Expert Reviews

9

Expert Reviews

■ Expert review sessions held with potential EdTech early adopters◻ Individuals integrating technology

in the classroom◻ Understand potential study

environment; recruit participants■ Eight teaching staff interviewed■ Main outcomes

◻ Scale of technology integration◻ Varying learning models

10

Department #

Architecture & Planning 2

CILT 1

Chemical Engineering 1

Computer Science 3

Mechanical Engineering 1

8

Flipped Classroom Experiment

11

Orchestrating a Flipped Classroom (1)

■ Computer Architecture second year course◻ 175 registered students◻ Three core-activities conducted during lecture sessions

■ Study aimed at assessing feasibility and potential◻ Extent of prototype tool use◻ Tool effect on learners’ learning experience

12

Activity Orchestration Orchestration Level

Demonstration Live demonstration Class

Discussion Interactive discussions Class/Individual

Timed quiz Timed quiz session Individual

Orchestrating a Flipped Classroom (2)

■ Prototype orchestration tool◻ Standalone Web-based tool◻ Orchestration of three core activities

■ Prototype utilised between September–October 2014◻ Across a total of 12 lecture sessions

■ Review sessions held with lecturer to determine changes■ Direct observations conducted during lecture sessions■ Video analysis on lecture recordings and screencasts■ Learner survey on last day of class

13

Orchestrating a Flipped Classroom (3)

■ Nine orchestration tools used during duration of course ◻ Most tools used to render content◻ Specialised tools once-off tasks

■ On average prototype orchestration tool used most of the time◻ Used 66.72% on average◻ Context switching occurred an

average of two times with noticeable time during switchover

14

Tool Freq. Duration

Prototype 10 00:30:31

VideoGlide 8 00:07:56

Firefox 1 00:21:29

Impress 1 00:38:26

Evince 1 00:00:50

QtSpim 1 00:12:47

Robotic Arm 1 00:01:24

TextEditor 1 00:02:07

VirtualBox 1 00:00:58

Orchestrating a Flipped Classroom (4)

■ Tool helped organise activities

■ Static sequencing (activity listing) found helpful

■ Timer useful to Avg performers

15

■ ”I did not really notice the tool”

■ ”I found the classroom experience fun”

Orchestrating a Flipped Classroom (5)

Feasibility of organised orchestration■ Facilitated neutral flow of activities■ Results from learner survey indicate tool was impact

neutral

Potential to facilitate improved learning outcomes■ Learner survey suggests orchestration workbench has

the potential to positively impact learning experience

16

Future Directions

17

Further explore the correletion between the use of an orchestration workbench and positive teaching and learning experience

■ Gather additional evidence to support core claim■ Evaluate approach in authentic educational settings

Year #2+ Plan

18

Planned Activities

19

Milestone Milestone description Timescale

1. Experiment #2 M1021 case study September 2015

2. Experiment #3 Controlled experiment October 2015

3. Experiment #4 Large-scale experiment January 2016

4. Experiment #5 CSC1010H case study April 2016

5. Experiment N : TBD

6. : : :

7. Thesis draft Draft manuscript February 2017

8. Thesis manuscript Final manuscript April 2017

Bibliography

[1] Jeremy Roschelle et al. “Classroom Orchestration: Synthesis”. Computers & Education, 6a9:523–526, 2013.

[2] Mike Sharples. “Shared orchestration within and beyond the classroom”. Computers & Education, 69:504–506, 2013.

[3] Pierre Dillenbourg. “Design for Classroom Orchestration”. Computers & Education, 69:485–492, 2013.

[4] Pierre Dillenbourg and Patrick Jermann. “Technology for classroom orchestration”. New Science of Learning. 525–552, 2011.

[5] Luis P. Prieto et al. “Orchestrating technology enhanced learning: a literature review and a conceptual framework”. International Journal of Technology Enhanced Learning, 3(6):583, 2011.

[6] Juan A. Munoz-Cristobal et al. “Supporting Teacher Orchestration in Ubiquitous Learning Environments: A Study in Primary Education”. TLT, 83–97, 2014.

20