11 ways of looking at technology adoption in theclassroom (invited talk @ itd genoa) (05.10.2016)

11 ways of looking at technology adoption in the

classroomLuis P. Prieto

including joint work with M. J. Rodrguez-Triana, A. Holzer and D. Gillet

and many, many others

ITD-CNR, Genova 5.10.2016

A few words about me

B.Eng. Telecommunications(Univ. Valladolid)

Work in telecom industry Software developer Technical lead, EU projects

Long-standing interest: how people use technology

or not!

A growing concern: educational technology adoption

Research

(classroom)Practice

learning learningANALYTICS

etc.

Observationalstudy - routines

Orchestrationframework

LD deployment& enactment

Teacher PD workshops

Communities of teacher LDers

Literaturereviews

Paper-basedtechnologies

Eyetracking & orchestration load

Multimodal teaching analytics

Tools for everydayteacher reflection

Examining the effects of social media in co-located classrooms

Part IIn-depth view of recent research on adoption of a particular technology in the classroom

Examining the effects of social media in co-located classroomsA case study based on SpeakUpMara Jess Rodrguez-Triana, A. Holzer, L. P. Prieto & D. Gillet11th European Conference on Technology Enhanced LearningSeptember 14th, Lyon (France)

7 /26

Motivation

Motivation SpeakUp Methodology ConclusionsStudy

Social media

8


/26

Social media

9

Motivation SpeakUp Methodology Conclusions

Increase participation

Increase interaction

Distraction

Multitasking

How to use social media effectively in the classroom?

Study

/26

10

SpeakUp


/26

speakup.info room name: SpeakUp @ ITDjoin room: 85524

http://speakup.info

13

Research methodology

Motivation ConclusionsStudySpeakUp Methodology

/26

Research question

Does SpeakUp favor situations that lead to learning? Topics:Active participation (i.e., engagement)Attention (i.e., remaining on-task)Social interaction (on relevant content)Teaching style

Case study:Authentic scenario (university)

14


/26

Informants & data sources

15


/26

Data analyses

16


Mixed methods & triangulationQuantitative:descriptive statisticsexploratory computational analysesQualitative:manual message coding (e.g., relevant/not relevant for

learning)manual video coding (e.g., actor, action & target)

/26

17

Case Study

Motivation ConclusionsSpeakUp Methodology Study

/26

Authentic scenario

18

Motivation SpeakUp Methodology Study Conclusions

/26

1st Lecture of a Communication course at EPFL (Switzerland)

90 minutes 3 teachers 145

students SpeakUp

Topic 1: Engagement

High proportion of students joined the room (147 users out of 145 students!)

19


/26

SpeakUp was engaging for the students!

( = 5 in a 1-5 Likert scale)

Topic 1: Engagement

Face-to-face vs SpeakUp-mediated participation

20


Teachers(77 min)

Students(11 min)

Teachers(51

actions)

Students(3481

actions)

/26

Topic 1: Engagement

Participation profiles (K-means clustering)

21


77%

19%

4%

/26

Topic 2: Student attention

22


/26

SpeakUp had no clear impact on the student attention


Topic 2: Student attention

Students perception vs usage

23

What students said :) What students did :(


/26

Topic 3: Social interaction

24


/26

SpeakUp promoted interaction between teachers & students


SpeakUppromoted interaction among students

( = 4.5 in a 1-5 Likert scale)


Face-to-face vs SpeakUp-mediated interaction

25


/26


SpeakUp-mediated interaction15 isolated studentsinteraction degree:

Face-to-face interaction: Typical interaction degree < 8

26

Peer 1 Peer 2 Peer 3

Peer 4 Student Peer 5

Peer 6 Peer 7 Peer 8


/26

Topic 4: Teaching style

High levels of relevant activity correspond to activities guided by the teachers

Further and long term analysis are needed

27


/26

Topic 4: Teaching style

28


/26

Managing two simultaneous

channels is demanding and specially difficult if teaching alone

SpeakUpprovided us with awareness of a students back channel

29

Conclusions


/26

Discussion - Results

Did SpeakUp favor situations that lead to learning? T1: Students were engaged with SpeakUp but quite unevenly

T2: Students were not always on task on SpeakUp and inconsistent with their subjective perception of distraction

T3: SpeakUp facilitated social interaction, especially between peers (larger social network)

T4: Positive effect of teacher prompts on message relevance but SpeakUp usage can be too demanding for just one teacher

Anonymity as a double-edged sword: participation vs distraction

30


/26

Discussion - Limitations

This is only one session in a longer (course-long) analysis: novelty effects?

Link to learning still indirect (situations that are known to lead to learning)

Rich analysis approach... but time consuming! (how can we scale this up?)

31


/26

Future work

Finish the whole course analysis, incl. link between relevancy and scores

Scaffold teachers in guiding SpeakUpusageBest patterns of usage for relevant engagementAutomated support: SpeakUp bots?

32


/26

Contact: [email protected] , [email protected]

SpeakUp: www.speakup.info

mailto:[email protected]:[email protected]://www.speakup.info/

Part IILightning view of previous research on adoption of edutech





Literaturereviews







Observational classroom studies: the discovery of routines Context: public primary school RQ: How do teachers adopt a

CSCL tool in their practice? Results: Importance of

improvised but recurrentteaching actions (routines)

So What? Can these routines be useful for

teacher PD? What is the impact of these

routines in the success of lesson?





Literaturereviews







Conceptual framework fororchestration of TEL: 5+3 Aspects Context: Orchestration as a

worthy Ph.D. topic RQ: What do TEL researchers

mean when they talk aboutorchestration?

Results: Literature review and synthesis 5+3 Aspects

So What? Orchestration is about making

edutech that plays well in authentic classroom conditions

Technologies for classroom shouldsupport at all these aspects





Literaturereviews







Technologies for orchestration of learning designs in blended CSCL Context:

A lot of LD authoring tools, but nobodyuses them

maybe because its difficult to deployin Moodle and other LEs?

RQ: Can we support multi-LD, multi-LE deployment and enactment?

Results: Software architecture, prototype Evaluated in authentic settings,

workshops So What?

Motivated teachers could deploy & enactcomplex CSCL designs

but not many teachers want that levelof complexity!





Literaturereviews







Workshop approach for teacherprofessional development Context: Technology alone

does not drive usage, PD isneeded

RQ: Can we design PD actionsthat take into account thisidea of orchestration?

Results: Approach based onsuccessful routines, GLUE!-PS, conceptual+technological

So What? Workshops were well-received!





Literaturereviews







Supporting communities of teachers as learning designers Context:

A lot of LD authoring tools, not interoperable, not deployable

Communities of (LD) practice as key for itsadoption?

RQ: Can we support teachers (conceptuallyand technologically) to do LD with others, using whatever tools they want?

Results: ILDE: unified environment that integrates many

LD tools (powered by GLUE!-PS) PD package (workshops) for secondary, adult

and higher education So What?

. (?)











Literaturereviews

Three TEL literature reviews

Augmented paper in education: Mostly exploratory works, not many evaluated in

authentic conditions Development is still hard Generic UIs (e.g., tablets) have applicability

advantages Awareness and reflection in blended learning Learning dashboards:

Still many proposals only use single platform logs Growing number of studies in authentic setting, but

few longitudinal Learning is seldom measured (!!)

So What? Prototype syndrome as endemic in TEL Lack of evidence of learning advantages!






Literaturereviews






Paper-based technologies for theclassroom Context: Primary schools, university RQ: Can we apply augmented paper

beyond spatial concepts/skills? Results:

Several prototypes in Math, Chinese Software framework for easier

development of browser-based apps

So What? Still yet another prototype Is the engagement and learning we see

sustainable after novelty wears off?






Literaturereviews






Studies on orchestration load using mobile eye-tracking Context: Face-to-face classrooms RQ: Can we measure teacher load

beyond just asking how did it go? Results:

Method combining eye-tracking and other sensors + video coding + machine learning models

Initial insights about orchestration load patterns (e.g., whole-class activities are higher load)

So What? Initial validation gave encouraging

results if you record multiple sessions Manual coding of videos still expensive!






Literaturereviews






Multimodal teaching analytics: reversing learning design? Context: Face-to-face classrooms RQ: Can we get the enacted orchestration graph

(as opposed to the LD) from the sensors data? Results:

Machine learning models to predict teachingactivity and social plane of interaction

Reasonable accuracy (66-90%), with relativelysimple models

Applicable for the teacher in the study only So What?

Teacher can use this data for their PD reflectionbased on everyday evidence (~ fitness tracker)

Researchers can look at why enactments of thesame LD get different results

Could be used to kickstart learning designformalization? at scale!






Literaturereviews






Tools for evidence-based teacherreflection from everyday data Context: Secondary schools RQ: Can we help teachers in reflecting for

PD, by gathering data from everydaypractice?

Results: Two approaches:

1. High-tech (eye-tracking, beacons) visualizations (classroom mirror)

2. Low-tech (very simple questionnaire forstudents + teacher predictions) simple dashboard (Prolearning)

Low-tech shows improvement in studentexperience and teacher behavior changes

So What? Fancy tech does not necessarily mean better

adoption!






Literaturereviews






Conclusion

Research

(classroom)Practice

Conclusion: 5 tips for adoption

1. Get out of the building and inside the classroom(for long periods, if possible)

2. Demonstrate advantages first (especially learninggains!), before asking for adoption

3. Minimum Viable Prototypes (MVPs) overspecifications

4. If it cannot be used every day, it probably will notbe used after researchers go away

5. Think about scale from the outset

References (I)

Social media in the classroom (SpeakUp) Rodrguez-Triana, M. J., Holzer, A., Prieto, L. P., & Gillet, D. (2016, September).

Examining the Effects of Social Media in Co-located Classrooms: A Case Study Based on SpeakUp. In European Conference on Technology Enhanced Learning (pp. 247-262). Springer International Publishing.

Observational study & routines Prieto, L. P., Villagr-Sobrino, S., Jorrn-Abelln, I. M., Martnez-Mons, A., &

Dimitriadis, Y. (2011). Recurrent routines: Analyzing and supportingorchestration in technology-enhanced primary classrooms. Computers & Education, 57(1), 1214-1227.

Orchestration framework (5+3 Aspects) Prieto, L. P., Holenko Dlab, M., Gutirrez, I., Abdulwahed, M., & Balid, W. (2011).

Orchestrating technology enhanced learning: a literature review and a conceptual framework. International Journal of Technology Enhanced Learning, 3(6), 583-598.

Prieto, L. P., Dimitriadis, Y., Asensio-Prez, J. I., & Looi, C. K. (2015). Orchestration in learning technology research: evaluation of a conceptual framework. Research in Learning Technology, 23.

References (II) LD deployment and enactment (GLUE!-PS)

Prieto, L. P., Asensio-Prez, J. I., Dimitriadis, Y., Gmez-Snchez, E., & Muoz-Cristbal, J. A. (2011, September). GLUE!-PS: a multi-language architecture and data model to deploy TEL designs to multiple learning environments. In European Conference on Technology Enhanced Learning (pp. 285-298). Springer Berlin Heidelberg.

Prieto, L. P., Asensio-Prez, J. I., Muoz-Cristbal, J. A., Jorrn-Abelln, I. M., Dimitriadis, Y., & Gmez-Snchez, E. (2014). Supporting orchestration of CSCL scenarios in web-based Distributed Learning Environments. Computers & education, 73, 9-25.

Teacher PD workshop approach Prieto, L. P., Dimitriadis, Y., Asensio-Perez, J. I., Villagr-Sobrino, S., & Jorrn-Abellan, I. (2013).

Fostering CSCL adoption: an approach to professional development focused on orchestration. In Proceedings of the International Conference of Computer-Supported Collaborative Learning(CSCL2013). volume 1.

Communities of teacher learning designers (METIS ILDE) Hernndez-Leo, D., Chacn, J., Prieto, L. P., Asensio-Prez, J. I., & Derntl, M. (2013, September).

Towards an integrated learning design environment. In European Conference on TechnologyEnhanced Learning (pp. 448-453). Springer Berlin Heidelberg.

Hernndez-Leo, D., Asensio-Prez, J. I., Derntl, M., Prieto, L. P., & Chacn, J. (2014, September). ILDE: community environment for conceptualizing, authoring and deploying learning activities. In European Conference on Technology Enhanced Learning (pp. 490-493). Springer International Publishing.

References (III)

TEL literature reviews Prieto, L. P., Wen, Y., Caballero, D., & Dillenbourg, P. (2014). Review of

Augmented Paper Systems in Education: An Orchestration Perspective. Educational Technology & Society, 17(4), 169-185.

Rodriguez Triana, M. J., Prieto Santos, L. P., Vozniuk, A., Shirvani Boroujeni, M., Schwendimann, B. A., Holzer, A. C., & Gillet, D. (in press). Monitoring, Awareness and Reflection in Blended Technology Enhanced Learning: a Systematic Review. Accepted at the International Journal of Technology-Enhanced Learning.

Schwendimann, B. A., Rodrguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016, April). Understanding learning at a glance: An overview of learning dashboard studies. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 532-533). ACM.

Paper-based technologies for the classroom Caballero, D., Wen, Y., Prieto, L. P., & Dillenbourg, P. (2014, November). Single

locus of control in a tangible paper-based tabletop application: an exploratory study. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (pp. 351-356). ACM.

Web-based apps framework: http://chili-epfl.github.io/paprika/

http://chili-epfl.github.io/paprika/

References (IV)

Eye-tracking studies and orchestration load Prieto, L. P., Sharma, K., Wen, Y., & Dillenbourg, P. (2015). The burden of

facilitating collaboration: towards estimation of teacher orchestration load using eye-tracking measures. In Proceedings of the 11th international conference on computer-supported collaborative learning (CSCL 2015) (pp. 212-219). Sweden: Gothenburg.

Prieto, L. P., Sharma, K., & Dillenbourg, P. (2015). Studying teacher orchestration load in technology-enhanced classrooms. In Design for Teaching and Learning in a Networked World (pp. 268-281). Springer International Publishing.

Multimodal teaching analytics and reversing LD Prieto, L. P., Sharma, K., Dillenbourg, P., & Jess, M. (2016, April).

Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 148-157). ACM.

Tools for teacher reflection [Still unpublished]

Questions (again)?

Thank you! For further info, email me: [email protected](or just catch me this week in the corridor)

mailto:[email protected]

11 ways of looking at technology adoption in the classroomA few words about meA growing concern: educational technology adoptionSlide Number 4Part IExamining the effects of social media in co-located classroomsA case study based on SpeakUpSlide Number 7Social mediaSocial mediaSlide Number 10Slide Number 12Slide Number 13Research questionInformants & data sourcesData analysesSlide Number 17Authentic scenarioTopic 1: EngagementTopic 1: EngagementTopic 1: EngagementTopic 2: Student attentionTopic 2: Student attentionTopic 3: Social interactionTopic 3: Social interactionTopic 3: Social interactionTopic 4: Teaching styleTopic 4: Teaching styleSlide Number 29Discussion - ResultsDiscussion - LimitationsFuture workSlide Number 33Part IISlide Number 35Observational classroom studies: the discovery of routinesSlide Number 37Conceptual framework for orchestration of TEL: 5+3 AspectsSlide Number 39Technologies for orchestration of learning designs in blended CSCLSlide Number 41Workshop approach for teacher professional developmentSlide Number 43Supporting communities of teachers as learning designersSlide Number 45Three TEL literature reviewsSlide Number 47Paper-based technologies for the classroomSlide Number 49Studies on orchestration load using mobile eye-trackingSlide Number 51Multimodal teaching analytics: reversing learning design?Slide Number 53Tools for evidence-based teacher reflection from everyday dataSlide Number 55ConclusionConclusion: 5 tips for adoptionReferences (I)References (II)References (III)References (IV)Questions (again)?Thank you!