game over? new approaches to teaching engineering courses
TRANSCRIPT
Gamifying Personal Information Management
Game Over ?New Approaches to Teaching Engineering Courses
Joaquim Jorge
Sponsored by
Joaquim Jorge
Instituto Superior TcnicoUniversidade de LisboaVisualization and Multimodal Interfaces@ INESC-ID Lisboahttp://web.ist.utl.pt/jorgej
Research Interests: Calligraphic Interaction, Multimodal Interfaces, Graphical Modeling
About ACMACM, the Association for Computing Machinery is the worlds largest educational and scientific computing society, uniting educators, researchers and professionals to inspire dialogue, share resources and address the fields challenges.
ACM strengthens the computing professions collective voice through strong leadership, promotion of the highest standards, and recognition of technical excellence.
ACM supports the professional growth of its members by providing opportunities for life-long learning, career development, and professional networking.
With over 100,000 members from over 100 countries, ACM works to advance computing as a science and a profession. www.acm.org
The Distinguished Speakers Program is made possible by
For additional information, please visit http://dsp.acm.org/
ees.elsevier.com/cag 2014: 460 submissions
29 days to respond 17 weeks to accept Ranked 4th CGJGoogleScholar
Back to the future?0
Bologna 1088
Wikimedia commons
Modern Universities started with Bologna in 10887
XIX Century
In Europe we still follow the Prussian Model from the XIX Century (Pictured: University of Heidelberg )8
Back to the Future ?
However Education models have not changed much in the last two centuries9
Are MOOCs the Answer?
Even though people tend to think about MOOCS as the latest advance, THEY FOLLOW the conventional classroom paradigm10
One Size Fits All!
All students get the same materials and Educational Content11
Same content x 1000s
Except that classroom size is now 10 of 1000s of students (instead on 100s) 12
Industrial Revolution
This is because for all the adoption of technology the classroom of the XXI century still follows the same approach as those of the XIX century: everybody is learning the same content in lockstep13
Standardized Experience
Indeed University (and then some!) Education follows the Industrial Revolution symbolized by the Ford Assembly Line. You can have the Model T in Any Color as Long as it is Black!14
From here
So MOOCS may very well follow the fate of other e-learning initiatives of the past decades15
To Here
For lack of a clear business model and for their XIX century approach MOOCs will likely join other e-learning experiences the graveyard of technology-enhanced learning16
Not About Tools!Strandberg [CC-BY-SA] http://commons.wikimedia.org/wiki/File:20060513_toolbox.jpg/
THIS TALK is not about SOFTWARE TOOLS! Rather we focus on THE learning experience!
17
Long Journey Ahead
safuanstyx.deviantart.com/
And Indeed there is still a LONG WAY TO GO. Because we have been focusing on the wrong things18
Gamification
So in My Talk I will discuss Gamification as a tool to increase student participation19
towardsAdaptation
And in the second part of the talk, I will discuss our results to enable adaptation of content delivery20
GamificationI
Games Students were born into themby William Hook @ Flickr
and become more familiar with ubiquitous technology and especially with games
22
Educationby Izaias Buson @ Flickr
Education is a great concern of modern society and many efforts have been done to make it more effective and available for everybody. Traditional courses and teaching techniques rely on blackboards, oral lectures, books and written exercises as the main vehicle to transmit knowledge. 23
Wavebreak Media Ltd/Bigstock.comBoredom
We face difficult problems: lack of student motivation and BOREDOM
Hye You | Daily TrojanDemotivating
Which leads to classrooms full of bored or distracted studentsThis is tough for students and professors alike
Can gamification improve learning?by thebarrowboy @ Flickr
Sheldon 2010the use of game design elements in non-game contextsgamification /emfke()n/
The pioneering experiences by Sheldon in 2010 transformed the classroom into a game27
NOT Playing games in classThats just playing a game
HOWEVER, Gamification is NOT (playing games in class) students already do that better 28
NOT teaching with a game
THOSE are Serious Games!29
NOT (just) giving points for activitiesThats Points-ification
The course becomes the game
Use game elements, mechanics, dynamics
Badges
Avatars
Votes
LeaderboardsAchievements
Boss Fights
Virtual Goods
Guilds
Quests
Rewards
Progress BarsSkill Trees
Experience Points
Stat Points
Just enumerate a few: XP, Badges, Rewards, Achievements, Skill Trees32
SDTCompetenceRelatednessAutonomyAdapted from [Deci and Ryan, 2004]Its about motivation!
Gaming is about motivation!One of the most recent models is Self-determination Theory by Deci and Ryan. Or SDT for short.
SDT is a comprehensive Theory on Human Motivation which answers the question on what characteristics should embody an activity to motivate us to perform it It builds on three foundations:
Autonomy
Related to our ability at making choices and taking control of our lives and destiny
34
Relatedness
Comes from the universal need of human beings to socialize and belonging to groups (or tribes) which makes football so popular35
Competence
Competence, or feeling empowered to Perform tasks, Achieving goals while become proficient
36
Multimedia Content ProductionII
MCP is a course of the Master program in Information Systems and Computer Engineering at Instituto Superior Tcnico37
MSc Course
IST/UL
. It is part of the Multimedia Systems specialization area. Instituto Superior Tcnico has two campii, Alameda and Taguspark, where this area is a Minor or a Major, respectively. The MCP Course, as part of either one, runs simultaneously at both campii. Pictured is TagusPark outside Lisbon
38
Two Campi
And this is the other campus, in downtown Lisbon.39
300+ Students Gamified Experience
We have been running the gamified experience since 201040
Blended Learning
We combine classroom (traditional) lectures with online experience41
Gamified since 2010/2011
2012/20132011/20122010/20112009/20102008/20092007/2008
2013/2014
2014/2015
Adding game elements to the mixby gfpeck @ Flickr
Based on several successful social games, like Foursquare or FarmVille, we decided to embody our course into such a game, where players would have the chance to progress in the game experience as they do on the course.
43
The course is a game!
We picked five of the most used game elements to drive student behavior towards reverting boredom and distractedness: points, levels, leaderboards, challenges and badges.
44
IMMERSION
IMMERSION
GOAL: To increase IMMERSION driven by novelty45
ENGAGEMENT
And ENGAGEMENT by participation in course activities46
XP, not grades!
To engage students in course activities, we reward them with experience points (XP) for almost every action they perform, as a form of instant gratification, which has been previously used with success to motivate college students. Examples include posting something on the forums, reading class slides, attending classes on time, reporting bugs in class material, etc.
47
LEVELS (20 of course)
Portuguese Grading system is structured from 0-20, where 10 is a passing grade and 20 is max!48
Grading system
MaxMin2003-2010Gamified02020000 XP500 XP
The final grade was represented by a value between 0 and 20.
In 2010-2011, instead of grade points, the students engaged in a game-like experience and were awarded Experience Points (XP). The more XP users had, the more they progressed in the gaming experience, advancing into new levels, reached at 1000 point intervals. Each level corresponds to a single grade point (1 out of 20), and students needed 20000 points to reach the maximum grade.49
PHAT L00T
ALMOST EVERYTHING EXCEPT SNEEZING (yet) IS GRADED50
Badges
PostmasterParticipate in the forumsClass AnnotatorFind relevant referencesRise to the ChallengeTinkering with class subjects
51
CompetitionProactiveBe the first to respondArchivistArchive challenge resultsQuiz, Lab, Presentation, Exam KingBe the best!
The first motivational factors to be explored by us were peer pressure and competition. 52
Leaderboard
A key component to spur competitiveness is for players to and compare themselves with others, to ascertain who is winning and how many points apart they are from other players.
Our game has a leaderboard that displays enrolled students by row, sorted in descending order by XP. Each row portrays the players rank, photo and name, campus, XP, level and # achievements they were awarded by completing challenges.53
Personal Page
Each student has an individual page where they can see their progress (see charts on the right)And how they compare to the outher students (see the XP and Badge histograms Bottom-Left)
Earn Achievements for more XP!
Most of our challenges are multi-level. For each level there is a badge, players have to perform iterations of the same task to reach the maximum level. Higher levels have increased difficulty. A an example of such challenges we have Bookworm where the user has to read 50% of the lectures slides for level 1, 75% for level 2 and 100% for level 3.
55
Three levels for each achievement
Up to Three Levels
Another good example is the Proficient Tool User challenge, in which students had to compose one creative artifact using multimedia tools (e.g. Gimp, Audacity) for level 1, two for level 2 and 3 for level 3. Multi-level challenges not only make things more interesting to students but might also provide them with a sense of autonomy.
Multi-level challenges not only make things more interesting to students but also provide them with a sense of autonomy.
56
Unlock a level to get XP
Description & Conditions for next lvl
IMMERSION and ENGAGEMENT57
Extra credit available
67 Badges Overall
32 will get you up to Level 20 35 will get you beyond! (EXTRA GRADE!)different ways to reach max bonusChoose your preferred path!
58
CooperationGuild WarriorEveryone in your lab does wellGuild MasterYour lab is the best there is!
We also rewarded cooperation (to increase connectedness)59
Cooperation: Quest!
TREASURE HUNT in teams15 different levelsStudents need to cooperate to achieve the last levelWe used this to combat post-midterm break disconectedness60
Reward Behavior
Right on TimeBe here when the lecture starts, dont be lateAmphitheatre LoverAttend theoretical lecturesLab LoverAttend Labs
61
Dozens of different achievements
62
Highlights Participate!
PostmasterParticipate in the forumsClass AnnotatorFind relevant referencesRise to the ChallengeBi-Weekly tinkering with class subjects
Examples of competition include the above63
Highlights Participate!TalkativeAsk intelligent questions in class
This was an interesting experience. For students who asked intelligent questions or remarks in class.Lurkers hated them though! and complained that 10% of the class got 90% of these badges64
Self-Determination: Skill Tree
Skill-tree is an example of self-determination: students can complete tasks at their pace and pick up XPThere are four levels and multiple paths to complete the tree,Tiles become active when pre-requisites are achievedWhen a task is completed, its tile is highlighted in blue aura65
Quality Over Quantity
by EssG @ Flickr (CC BY-NC-SA 2.0)
Drawing on the design implications from the previous study, posts started to be graded based on a rating from 0 to 4 assigned by faculty, with the aim of promoting quality over quantity. Therefore, a student making poor quality posts would have to make four times more posts, to have the same grade as a student making quality posts. 66
Creativity: AvatarWorld
Students are represented by an avatar that they can use to explore the world. Avatars can be customized with equipment items and handheld objects, which can be unlocked by acquiring certain course badges. Students can also create custom content for the game, such as buildings and equipment items, using tools and techniques introduced in class. Submissions were made via posts and graded by faculty, based on their creativity and technical correctness. The main goal here was to appeal to the students needs for self-expression and social recognition (Richter etal., 2015). We wanted to allow them to develop a sense of identity (Crumlish and Malone, 2009), by enabling them to customize their learning experience and to be creative.67
Getting started made easyby bokeh burger @ Flickr
Usually, getting started is the harder part. We added a few challenges of least difficulty so that, right at the,
students could collect a few achievements and become more. For multi-level challenges, we give greater score to level#1, to encourage 1st steps. For example, Bookworm level 1 is worth 100XP, level 2 is 70XP & level 3 is 40XP.68
by EssjayNZ @ FlickrA transparent process
Achievement awards make it as transparent as possible: students know when them or others received 1 achievement and why. By clicking a student in the leaderboard, we can see the history of all earned achievements, including date, the XP, the challenge and the respective badge. This allows students to get acquainted with the game, understand what others did it and strive to compete and perform better.69
moodle
Our system is based on Moodle, a Virtual Learning Environment
We use moodle, since it is free and provides basic functionalities such as support fora, quizzes ad exams as well as learning materials
Forum Posts
MANY game-related activities evolve around posts. BUT NOT ALL.71
Scripts
Peter van Grieken @ floickrActivitiesScoresGrades
Since Moodle does not support gamified environments (YET)We developed an applicationTo extract content from google database and toCollect MOST activity information (forum posts, quizzes, intermediate grades teacher assigned etc.)72
Manual Grading
Chris Lawrence @ flickr
Grading is done manually by course instructors heavier load than a traditional course.INSTANT GRATIFICATION: Students start to complain if grades are not assigned after a few hours
EVEN THOUGH we told them, at the beginning that grading would be done by humans 73
Spreadsheets to collect
We use googledocs intensively to record activities outside moodle and database-driven applications to collect student activity74
Generate Pages
The scripts generate the leaderboard and individual pages (HTML5) for each studentAs well as badges and computing statistics. We use PHP, Python, HTML5 and Javascript75
ResultsIII
We deployed the gamified version of the MCP course on the second semester of the 2010-2011 academic year. Data were collected regarding many aspects of the students performance and satisfaction, which we compared with those from the year before (2009-2010), in order to assess the impact of gamification.
In 2010-2011, there were still significant gains in the number of downloads of the courses reference material, of the number of Moodle posts and the number of attended lectures. We will address these changes in the next subsections, along with the feedback we got from students, based on their answers to a satisfaction survey at the beginning, middle and at the end of the course.76
Participation & Proactivityby woodleywonderworks @ Flickr
COMPARED TO PREVIOUS YEARS Our data show that students do not necessarily perform better, but they perform more first and reply posts on most components, which shows that they participate more and are more proactive. 77
Student feedbackby albertogp123 @ Flickr
We got valuable information via questionnaires issued three times during the semester.78
Easier to Learn From
Students considered our course to be easier to learn from as compared to other courses.79
Creativityby theodevil @ Flickr
Students consider that our course allows them to be creative.A comparison between the two academic years (gamified vs non-gamified) reveals that there was a larger gap between the amounts of posts in the first two months than in any other period. We believe that one of the reasons behind this were the challenges that were issued in lab classes. The results not only suggest that creative challenges are a powerful weapon to boost student engagement and motivation, but also that they should occur more sparsely over time.
Miho, PLEASE do not read this: These required students to come up with interesting ideas and become authors themselves, by creating funny multimedia contents, such as montages, audio remixes and video parodies. The fact that as few as three challenges, issued during three weeks, managed to get 19% of the total amount of posts, indicates that they were indeed a major contribution to student participation. We also had informal feedback from students complimenting the creativity of the challenges. Another plausible reason seems to be the theoretical challenges, which were responsible for 23% (336 out of 1439) of the total amount of posts. These, in turn, occurred during the first two months.80
More Of What They Like
Students consider that our course allows them to do more of what they like81
Useful for the Future
And that it teaches them useful stuff for the future.82
More Interesting and Motivating
by johnwiechecki @ Flickr (CC BY-NC-SA 2.0)
Besides the key findings described above, we also found that students considered our course more motivating and interesting than other non-gamified courses, and that they considered the gamified experiences to have gone well and that they would like to see it applied to other courses. 83
Towards personalizationIV
Student Model
by velkr0 @ Flickr (CC BY 2.0)
We studied student activity model and patterns of grade acquisition.These support a (stable) cluster structure observed in the 3rd and 4th gamified iterations of the courseWe use the findings and data collected since the first gamified course to develop a model of student performance and participation in a gamified environment. This model offers an overview of what we consider to be the four archetypes of our gamified course, with the main goal of providing a clearer understanding of what distinguishes the students of gamified MCP.85
Cluster Students
We managed to cluster students following grade acquisition patterns and leaderboard rank86
Leaderboard Rank
This tracks evolution of different students throughout the semester (numbers are days from start of experience)Colors tag different student types or clusters87
Students by Cluster
Of the 76 students that took the course last year, we could identify 4 types according to grade acquisition patternsAchievers are go-gettersRegular students tend to budget their efforts towards a desired gradeHalf-Hearted start like achievers but lose steam as the semester progressesUnderachievers show the least level of engagement88
PerformanceDeferredLearningContinuousLearningTheoreticalMasteryDelivery
Data from radar charts was computed by normalizing each measurement to a value between 1 and 5, where 1 represented the lowest performance value and 5 the highest. These values were then averaged on all years in which the cluster was represented. The further away it is from the center of the radar, the better a cluster performed on average in comparison to others, in all years it was observed.For the performance measures, only the XP earned from the Skill Tree and AvatarWorld (a game within a game) were consistently discriminant over time. However, we kept the other features as they are important evaluation components in the context of our course. It provides a subjective yet organized view on our data.
89
Performance
Here we can see that each cluster shows typical (different) performance profiles90
ParticipationCollectingPostingGamingSkill Acquis
Most of the participation features included in the model were consistent between the two years. The exceptions were the number of badges earned from the Artist achievement and the number of posts made on AvatarWorld, which were only significant in one of the years, but still provide valuable insights. 91
Participation
Were we can also see that levels of activity highlight different levels of participation for each cluster92
Engagement
Self-reported engagement values typical of students for each cluster93
Predicting Student Profile
We used classification algorithms to train statistical models with data from one instance and predict the type of a student population data from another year, as early as of midterm. predicting a student type early in the course is key to create smart gamified learning environments that can adapt to different students and cater to different needs, predicting student types from one year with data from another, with high accuracy, would further support the consistency of our cluster structure.
94
Automatic Early-on Classification
THIS IS OUR MAIN RESULT SO FAR: By using data mining and pattern classification techniques we were able to predict current years performance using the previous year data to train classifiers and a gamer-type questionnaire to match students to clusters.Initial predictions are not very good, BUT later ones become very effective: After 1/3 semester we can predict >70% accuracy the cluster a student belongs to.95
Grade accumulation over 2 yrs
ALSO We can see that the grade accumulation patterns are consistent across two different years for each cluster
96
Self-Reported Satisfaction
Another way to differentiate students comes through self-reported student satisfaction. This was done on the non-gamified course, but findings seem applicable here too.97
98Satisfaction w content
Here we used data from three questionnaires collected at beggining, mid-term ad end-of semester and could identify three different clusters according to perceived usefulness, ease of use (moodle interface) and perceived quality of contents98
99
In this graph we highlight different (student) cluster matches according to different delivery types (the 4 quadrants in CSCW)99
Focus per cluster100
Duffirent students focus on different issues:100
Satisfaction
And student profiles were repeatable across different years too101
We found student profiles across different clusters consistent across different years too102
Design issuesV
Meaning
Students must consider gamified tasks and assignments meaningful. If they do not seen any value in them, no game elements can help you.104
Mastery
Provide students with rich feedback via points, badges and progress bars about goals and tasks that are meaningful to the student. This will improve their sense of mastery and competence.105
Autonomy
Provide several ways for students to draw their own learning path. 106
Diversity
We already mentioned the importance of creating different types of assignments to provide diverse paths for students to choose from. However, it is important that these paths cover varied types of subjects, catering to different needs and preferences. This will provide a more flexible learning environment, more likely to create the conditions for new behaviors to emerge, thus reaching out to more students and making the gamified experience easier to learn from. This may allow different types of students to achieve similar levels of performance through different routes. 107
Social Recognition
Badges, titles and levels provide social recognition which can be motivating.108
Manage the Workload / Rythm
Provide ways for students to manage their pace109
Competition vs Cooperation
Need to balance competition vs cooperation. HOWEVER, our findings indicate that students FOCUS on competition while they keep asking for more cooperation!110
(Mass)Customize the Experience
Our findings support this idea. This is where we are going to focus for the next editions of the courseWe believe that it is possible to support better gaming experiences through personalization made possible by data-mining and pattern classification, using even more student activity data.111
Adaptive Gamified Learning
by SMBCollege @ Flickr (CC BY 2.0)
Our research brings us towards adaptive gamified learning environments, which can promptly adapt to students with different preferences and needs.112
The Team
From Top to Bottom, Left to Right:Daniel GonalvesGabirle Barata (who is about to defend his PhD thesis on gamification)Joaquim JorgeSandra Gama (she was TA and co-author of several papers on the gamified environment)Manuel Joo FonsecaJoo Amaral (who worked on the scripts)113
[email protected]@[email protected]
International Conference PapersBarata, G., Gama, S., Jorge, J.A., and Gonalves, D.J. (2014b). Relating gaming habits with student performance in a gamified learning experience. In Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play, CHI PLAY 14, pages 1725, New York, NY, USA. ACM.Barata, G., Gama, S., Fonseca, M.J., and Gonalves, D. (2013a). Improving student creativity with gamification and virtual worlds. In Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification 13, pages 9598, New York, NY, USA. ACM.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2013e). Improving participation and learning with gamification. In Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification 13, pages 1017, New York, NY, USA. ACM.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2013d). So fun it hurts - gamifying an engineering course. In Proceedings of the 15th International Conference on Human-Computer Interaction, HCII 2013, pages 639648, Las Vegas, Nevada, USA.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2013b). Engaging engineering students with gamification. In Proceedings of 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), 2013, pages 18.
International Conference PapersBarata, G., Gama, S., Jorge, J.A., and Gonalves, D.J. (2014b). Relating gaming habits with student performance in a gamified learning experience. In Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play, CHI PLAY 14, pages 1725, New York, NY, USA. ACM.Barata, G., Gama, S., Fonseca, M.J., and Gonalves, D. (2013a). Improving student creativity with gamification and virtual worlds. In Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification 13, pages 9598, New York, NY, USA. ACM.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2013e). Improving participation and learning with gamification. In Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification 13, pages 1017, New York, NY, USA. ACM.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2013d). So fun it hurts - gamifying an engineering course. In Proceedings of the 15th International Conference on Human-Computer Interaction, HCII 2013, pages 639648, Las Vegas, Nevada, USA.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2013b). Engaging engineering students with gamification. In Proceedings of 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), 2013, pages 18.
Best Paper Award
Journal PapersBarata, G., Gama, S., Jorge, J., and Gonalves, D. (2015b). Predicting student profile with performance and gaming preferences. IEEE Transactions on Learning Technologies, (submitted).Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2015c). Studying student differentiation in gamified education: A long-term study. Computers in Human Behavior, (accepted).Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2015a). Gamification for smarter learning: tales from the trenches. Smart Learning Environments, 2(1):10.Barata, G., Gama, S., Jorge, J., and Gonalves, D. (2014a). Identifying student types in a gamified learning experience. International Journal of Game-Based Learning (IJGBL), 4(4):1936.
Daniel Gonalves (DG) - E ainda aquele ebook que estive a escrever h uns tempos (s co-autor)Gabriel Barata (GB) - E qual a ref disso? :DDaniel Gonalves (DG) - Will try to find out...Gabriel Barata (GB) - Assim que tivermos, actualizo.
We will have an Educational Program in Eurographics 2016 TOO!Please submit your work and come to EG 2016118