notice please note that this document may not be the...

41
This may be the author’s version of a work that was submitted/accepted for publication in the following source: Philip, Robyn & Greener, Elizabeth (2016) Robotics MOOCs Evaluation Report 2015. Queensland University of Technology, Australia. This file was downloaded from: https://eprints.qut.edu.au/103552/ c 2016 Queensland University of Technology and authors This work is covered by copyright. Unless the document is being made available under a Creative Commons Licence, you must assume that re-use is limited to personal use and that permission from the copyright owner must be obtained for all other uses. If the docu- ment is available under a Creative Commons License (or other specified license) then refer to the Licence for details of permitted re-use. It is a condition of access that users recog- nise and abide by the legal requirements associated with these rights. If you believe that this work infringes copyright please provide details by email to [email protected] Notice: Please note that this document may not be the Version of Record (i.e. published version) of the work. Author manuscript versions (as Sub- mitted for peer review or as Accepted for publication after peer review) can be identified by an absence of publisher branding and/or typeset appear- ance. If there is any doubt, please refer to the published source.

Upload: others

Post on 11-Jun-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:

Philip, Robyn & Greener, Elizabeth(2016)Robotics MOOCs Evaluation Report 2015.

Queensland University of Technology, Australia.

This file was downloaded from: https://eprints.qut.edu.au/103552/

c© 2016 Queensland University of Technology and authors

This work is covered by copyright. Unless the document is being made available under aCreative Commons Licence, you must assume that re-use is limited to personal use andthat permission from the copyright owner must be obtained for all other uses. If the docu-ment is available under a Creative Commons License (or other specified license) then referto the Licence for details of permitted re-use. It is a condition of access that users recog-nise and abide by the legal requirements associated with these rights. If you believe thatthis work infringes copyright please provide details by email to [email protected]

Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.

Page 2: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs

Evaluation Report 2015

Writtenby:

DrRobynPhilipandMsElizabethGreener

foreLearningServices,TILS,QUT

Page 3: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design
Page 4: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOC Report 2015

1

Contents1.ExecutiveSummary.......................................................................................................................................3

2.KeyFindings...................................................................................................................................................4

3.Introduction...................................................................................................................................................7

3.1Backgroundandcontext........................................................................................................................7

3.2Coursedesign.........................................................................................................................................8

3.3Methodology..........................................................................................................................................93.3.1Aim................................................................................................................................................93.3.2Objectives......................................................................................................................................93.3.3Evaluationframeworkandtheelearninglifecycle........................................................................93.3.4Datacollectionandanalysis........................................................................................................10

4.Demographicsandpatternsofengagement...............................................................................................11

4.1Demographics......................................................................................................................................11

4.2Studentmotivations,completionsandpatternsofengagement........................................................134.2.1Motivationforenrolmentandpathways....................................................................................134.2.2Retentionandcompletionrates..................................................................................................134.2.3Engagementpatterns..................................................................................................................14

5.Responsetothecoursedesignandlearningandteachingstrategies.........................................................15

5.1Studentcoursesatisfaction..................................................................................................................15

5.2Successofthecoursedesign................................................................................................................16

5.3Levelofdifficulty:designingforamixedcohortofstudents...............................................................18

5.4Assessment..........................................................................................................................................19

5.5Engagementstrategiesandcommunity..............................................................................................195.5.1Thecentralityofthediscussionforum........................................................................................195.5.2Liveevents:GoogleHangouts.....................................................................................................205.5.3Otherengagementactivities.......................................................................................................21

5.6Accessandequity.................................................................................................................................21

6.Technicalandpedagogicalinnovations.......................................................................................................22

6.1TheEdCastplatformandcollaborationwithindustrypartners...........................................................226.1.1IntegrationofMATLABauto-gradingtool...................................................................................236.1.2Automatedonlinepeerreview...................................................................................................256.1.3Textbookpublishersupport........................................................................................................25

6.2ReuseofcourseresourcesforENB339................................................................................................26

7.Institutionalimplications.............................................................................................................................27

7.1Productionissues.......................................................................................................................................27

7.2Evaluationandlearninganalytics........................................................................................................28

7.3Staffdevelopmentandinnovation......................................................................................................28

7.4Marketing.............................................................................................................................................29

Page 5: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

2

8.Conclusion...................................................................................................................................................30

8.1Generalconclusions.............................................................................................................................30

8.1Futureresearchandopportunitiesforinnovation..............................................................................31

Acknowledgements....................................................................................................................................31

References.......................................................................................................................................................32

Appendices......................................................................................................................................................33AppendixA:Objectives,sampleindicatorsanddatasources..............................................................33AppendixB:Datasources,samplinganddatatypes............................................................................34AppendixC:Detailsoftheevaluationelearninglifecyclefortheproject............................................35AppendixD:Thebestaspectsofthecourse........................................................................................36AppendixE:Qualityoftheresources...................................................................................................37AppendixF:Assessment.......................................................................................................................38

Related eLearning Services documents • CourseDesignPlan:IntroductiontoRobotics&IntroductiontoRobotVisionMassiveOpenOnlineCourse

(MOOC)(September2013)• EvaluationPlanandResearchDesignforRoboticsMOOCs(March2015)• EvaluationReportforENB339:ReuseofMOOCResourcesandtheFlippedClassroom(August2015)• MATLABReportforRoboticsMOOCs(October2015)• MOOCProgram:LessonsLearnedfromDesignandProduction(August2015)• QUTEthicsApprovalCertificate1500000201(June2015)• RoboticsMOOCsPreliminarySnapshot(July2015)

Page 6: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOC Report 2015

3

1. Executive Summary InFebruary2015,QUTlaunchedthefirstoftwoMassiveOpenOnlineCourses(MOOCs)inscienceandengineering:Introduction toRobotics(IR)andRoboticVision(RV).Aslarge-scale,online,distanceeducationcourses,theseMOOCsallowedthousandsofstudentstosimultaneouslyattendandengageinthesamecourseattheonetime.FundingforthecourseswasinitiallyallocatedfromtheVice-Chancellor’sTransformationFund.

Studentsforbothcoursescamefrom161countriesaroundtheworld.12,894enrolledinIntroductiontoRobotics,whichranfromFebruarytoApril2015;and7,824enrolledinRoboticVisionMOOC(ApriltoJune2015).Enrolmentwasnotrestrictedbyageorprioreducation,andnocoursefeesapplied.InbothMOOCs,themainreasonforenrollinginthecourseswasforgeneralinterestorenjoymentpurposes.Thesecondmostpopularreasonwasprofessionalinterest.

Intotal,1,096certificateswereawardedtostudentswhomettheassessmentrequirements(621forIR;475forRV),andthetopperformerswhogained90%oraboveweresentpersonalemailsofcongratulationsfromtheleadacademic,ProfessorCorke.Inaddition,27studentssuccessfullycompletedtheoptionalrobotbuildingprojectforthefirstMOOC.Toassesseachstudentprojectasystemofpeerreviewwasdesigned.ThesystemwasjointlydevelopedfromanopensourcetoolbyQUTstaffandtheplatformprovider,EdCast.

Overallcoursesatisfactionwasveryhigh.Studentscommentedontheeffectivedesignofthecourse.InparticularstudentscomplementedProfessorCorkeforhisenthusiasmforthesubjectmatterandclearexplanationofconcepts.Studentsalsolikedtheapplicationoftheorytorealworldexamples,andaseventhoughthemathematicsandprogrammingwerechallengingforsomestudents,acoregroupappreciatedthischallenge.Studentsalsocommentedonthesupporttheyreceivedviathediscussionforumfromtheteachingassistants(TAs),andtheinvaluablesupportofpeersonline.Somestudentsalsosaidthatcourseswerethebestonlinecoursesofthistypethattheyhadevertaken.

Thepurposeofthisdocumentistoreportontheeffectivenessoftheelearningenvironmentregardingthedesign,productionandimplementationofthefirsttwoQUTMOOCs.TheevaluationwasframedwithinthePhillips,KennedyandMcNaught(2012)elearninglifecycle.FindingsdiscussedhererelatetodesignandsupportoftheMOOCenvironment,learningandteachingissuesandprocesses,productionandtechnicalmatters,andorganisationalissues.ParticulartechnicalandpedagogicalinnovationsrealisedthroughtheprojectincludethecollaborationbetweenQUTanditsindustrypartnerstodevelopanautomatedpeer-reviewsystem,automatedassessmentforMATLABsoftwareprogramming,integrationofliveeventsintotheEdCastplatform,andthereuseofMOOCresourcesinthethird-yearroboticsunit,ENB339.QUT’svaluedindustrypartnersforthiscollaborationwereEdCast,MathWorks,andSpringer.

ThefindingsfromthisreportwillinformfutureiterationsoftheroboticsMOOCs,subsequentQUTMOOCsfromotherdisciplines,andonlinelearninggenerallyattheUniversity.Inaddition,thelargeamountofdatacapturedfromtheMOOCscanpotentiallyformthebasisformuchresearchintolearninganalytics.Thisisexpectedtobeofinteresttotheuniversitygivenitsstrategicgoalsregardinginnovationandresearch,asoutlinedintheQUTBlueprint4(QUT,2014).

Page 7: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

4

2. Key Findings Thekeyfindingsfromthereportaresummarisedinthissection.Detailsofeachfindingareexplainedanddiscussedwithinthebodyofthereport. 1. Learningandteachingdesignprinciples

1.1. Sufficienttimeneedstobeallocatedfortheteachingteamtodevelop:(a)theirconceptualunderstandingofthespecialrequirementsofonlinelearning,(b)thedemandsoftheMOOCenvironment,and(c)therolesandresponsibilitiesofworkingwithateamoncoursedesign,productionandimplementation.

1.2. ThelengthofthistypeofMOOCisbestkeptshort,ideallyafewweeks,asstudentengagementandparticipationdropssharplyafterweeks1and2.However,wherestudentsneedtoacquireadditionallearningresourcesorinstallsoftware,extratimeshouldbefactoredintothecourseorientationperiodtoallowforthisessentialstudentpreparation.

1.3. Designissues,scopeandtimelinesneedtobecollaborativelynegotiatedandmanagedwithallteammembers.Rolesandresponsibilitiesoftheteachingandsupportstaff,thedesign,productionandtechnicalteamneedtobeclarifiedearlyinthedesignprocess.

1.4. Systemsformanaginganddisseminatingthefindingsderivedfromthelearninganalyticsneedtobeestablishedasearlyaspossible.Just-in-timereportinghelpswithmonitoringandtroubleshootingproblemsasthecourseprogresses,andallowsfortimelyfeedbackoncetheredesignanddeliveryprocesscommences.

2. Supportforstudentsandcommunitybuilding

2.1. Theroleofcommunitymanageriscriticalforsupportingcommunicationsbetweenandamongstthousandsofstudents.Itisanessentialpositionformanagingandcoachingtheteachingassistants,andbuildingandengagingapositivestudentcommunity.Oncetherolesandresponsibilitiesofthispositionhavebeendeterminedanddocumented,theactivitiescanbeallocatedtoeitheroneperson,ordesignatedtoteammemberswhohavebeencoachedintheironlineresponsibilitiesandtheneedforaccountability.

2.2. StudentcohortsmayvaryaccordingtotheMOOCsubjectfocusanddiscipline.ThestudentsintheroboticsMOOCswereoftenparticularlypassionatehobbyists,studentsorteachersdedicatedtothefield.

2.3. Asystemforspecialacknowledgementofinformallearningwaswellreceived.Informalacknowledgementcanbeforcontributionsoverandabovecourseassessmentrequirements.Forexample,inthefirstroboticsMOOCstudentswhomadeasignificantcontributiontothediscussionforumreceivedapersonalemailfromProfessorCorkethankingthemfortheirspecialparticipation.

3. Theplatformandrelationshipswithindustrypartners

3.1. Theplatformiskeytothewholeproject,soselectionoftheplatformandotherdigitaltechnologiesusedneedstocommencewellinadvanceofthecoursedesignprocess,andcontinueinparallelasthedesignevolves.

3.2. Itiscriticaltomaintaineffectiverelationshipswithindustrypartners.TheMOOCprojectwasauniqueopportunityforQUT,asanelearningprovider,toshapethetechnologyincollaborationwithitstechnologypartnersinordertomeettheUniversity’spedagogicalneeds.ThereisaneedtocontinuetoworkwithEdCastonthecommunicationtoolsinparticular,refiningtheuserinterfaceofthediscussionforum,andenablingdevelopmentofgroupandindividualmessagingtools.

Page 8: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

5

3.3. Assystemsdevelop,continuousimprovementprocessesneedtobeapplied.Inparticular,thisappliestocopyrightacquisitionandmanagement,accessibilitycompliance,styleguidesandworkflowmanuals.

4. Production

4.1. Theelearningdesigncycleandproblemsolving:MOOCdevelopmentrequiresadesignapproach.Thisnecessarilyincludesphasesforiterativelygeneratingideas,conceptdevelopment,rapidprototyping,testingandqualityassurancepriortoimplementation.Itisessentialtofactorinpreparationandoperationaltimeforallstaff(design,productionandtechnical)forproblemsolving,researchingthefieldandsourcingalternativetechnicalandmediasolutions.

4.2. Conceptualdevelopment:TheMOOCteam(teaching,designandproductionstaff)needtimetobuildmutualtrustandtheirownconceptualunderstandingofthespecialrequirementsofonlinelearningintheMOOCenvironment.Rolesandresponsibilitiessharedamongstthedesignandproductionteam,andtheteachingteamneedtobenegotiated,andtimeallocatedforthisaspectoftheprocesstomature.

4.3. Qualityassurance:Theprocessofqualityassuranceisextremelyimportantforaprojectwith

suchhighpublicexposureasaMOOC.Thismeansthattimeandresourcesshouldbeallocatedforallstagesoftheelearninglifecycle,includingstudentandtechnicalsupportduringimplementationandevaluation.

4.4. Copyright:Tracingandsecuringcopyrightpermissions,ornon-copyrightalternativesisa

particularlytime-consumingprocess.Developersneedtobeawareofthisconstraintfromtheverybeginningoftheprocess.

4.5. Systems:WithanewelearninginitiativesuchastheMOOCproject,understandablynotallsystemswillbeinplacetocoverallaspectsoftheproject.Thisincludestechnicalprocesses,datastorageandmanagement,workflowsandguidelines,andlearninganalytics.Newsystemsneedtobecreatedandundergocontinuousimprovementtoensuresustainability,andtomaintainrelevanceandcurrency.

5. Analyticsandresearch

5.1. Streamlinedsystemsforcollecting,managingandanalysinglargedatasetsarecriticalfortroubleshootingduringimplementationandforsupportingfuturedevelopments.Thereisaroleforananalystwithexpertiseinlearninganalyticsandaknowledgeofteachingand/orlearningdesigninordertoenabletimelyreportingandefficientdisseminationofresults.

6. Professionaldevelopment,organisationallearningandstaffing

6.1. TheMOOCspaceisaninnovativeenvironmentwherestaffcanbeencouragedtoexperimentandinnovate.

6.2. Inprojectswherecomplexproblemsmustcontinuallybeaddressed,uncertaintyabounds,andinnovativeanswersarerequired.Muchlearningandresearchtakesplaceinformallyonthejob.

6.3. ThefindingsofthisreportconfirmthoseofGrajek(2015),thatoneofthetoptenITissuesof2015is‘hiringandretainingqualifiedstaff,andupdatingtheknowledgeandskillsofexistingtechnologystaff’(p.12).ThiswasthecasefortheroboticsMOOCs,whereflexiblemanagementstrategieswererequiredtolocate,retainandtrainstaffwiththerightmixofskillsfortheproject.

6.4. Inaninnovativeworkenvironment,wheredeadlinesareshortandthequalityofoutputiscritical,staffrewardandrecognitionareessential.Thishelpsmaintaingoodwillandcommitment.

Page 9: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

6

7. Resourcing,marketingandfuturedirections7.1. Thereisalifecycleforelearningprojectssuchasthis(seeFigure2).Itisimportanttorecognise

thataftertheinitialcycle,itisnecessarytoensurethereareresourcesforongoingmaintenanceandredeliveryoftheproject.

7.2. TherelationshipwithMarketingandCommunicationswaskeytothesuccessoftheproject.ThereisnowaneedtocontinuebuildingawarenessoftheMOOCprogramwithintheuniversity,aswellasextendingitsnationalandinternationalreachandprofilebeyondQUT.

7.3. Thereisvalueincontinuingtoexploreopportunitiesandexperiments,capitalisingonthewaveofinterestgeneratedbythesetwoMOOCsonrobotics.ConnectionswithMOOCalumnishouldbemaintainedandstrengthenedinthisregard.

7.4. Theindustrypartnershipswereveryproductivewithregardtothisproject.ItwouldbeadvantageoustocontinuetoworkwiththeseexternalpartnerstoinnovatearoundtheEdCastplatform.ItmaybeworthconsiderationtoapproachLegotosponsoravailabilityoftheMindstormKitsfortherobotmakingproject.

7.5. TheremaybeacasefordesigningroboticsMOOCstargetingparticulardemographics,forexamplehobbyistsorpractitioners.Asagendergapwasnotedintheseroboticscourses,consistentwithmuchscience,technology,engineeringandmathematics(STEM)education,furtherresearchcouldbedirectedtowardsfindingstrategieswhichencouragemorefemalestoenrolinthesecourses.Also,buildingonthehighnumbersofpostgraduatestudentsinterestedintheroboticsMOOCs,theUniversitymaywishtoconsidercreatingonlinemodulesforthecorporateandpostgraduatemarket;theUdacityfee-payingNanodegreemodelmaybeonetoexplore(https://www.udacity.com/nanodegree).

Page 10: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

7

3. Introduction

3.1 Background and context

TheMassiveOpenOnlineCourse(MOOC)projectwaslaunchedin2015.Itwasanewandexperimentalinitiativefortheuniversity.AlthoughQUThasastrongbackgroundinblendedlearningandmultimediadevelopment,theUniversityhaslessexperienceinthedesign,productionanddeliveryoffullyonlinecourses.Itsfocustodatehas,inthemain,centredonthesupportofon-campusstudents.However,asMOOCsbegantoemergeworldwideasanewformofonlinelearning(Gallagher&Garrett,2013),manyuniversitiesincludingQUTsawanopportunitytoinnovateandallocateresourcesforMOOCdevelopment.QUTinitiallyassignedfundingfordevelopmentoftwoMOOCsinrobotics.TheseMOOCsweredesignedanddevelopedthroughout2013and2014,andimplementedinFebruaryandApril2015.QUT’seLearningServicesandtheScienceandEngineeringFacultyworkedcooperativelytolaunchthetwoMOOCs,Introduction toRoboticsandRoboticVision,onthedeliveryplatformsuppliedbyEdCast(http://www.edcast.com).

FundingwasprovidedfromtheVice-Chancellor’sTransformationFund.TheMOOCprojectmetuniversitystrategicgoalsandbuiltonQUT’sinternationalresearchreputation.Theprojectalsocontributedtouniversitypriorities(QUT,2014),intermsofhighqualitylearningandteaching,engagementwiththecommunity,andtheUniversity’svision-onethatvaluesaspiritofexperimentation,innovation,andentrepreneurialism.Inaddition,theprojectbuiltonQUT’sinternationalreputationinthefieldofrobotics(seetheARCCentreofExcellenceinRoboticVision:http://roboticvision.org),andthequalityandrigourofitslearningandteachingattheUniversity.TheroboticsMOOCsprovidedimportantmodelsforthedesignanddevelopmentofMOOCsinothersubjectareas,andonlinelearninggenerallythroughouttheuniversity.

Figure1providesatimelineshowingtheprojectmilestones,fromconceptinitiationin2013,throughtoanalysis,design,development,prototyping,pilotingandtestingin2014,implementationanddeliveryofthetwoMOOCSinthefirsthalfof2015,summativeevaluation,andthenredeliveryofthesameMOOCsinthesecondhalfof2015(seeFigure1).

Figure 1: Project milestones for Robotics MOOCs, 2013-2015.

Page 11: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

8

3.2 Course design

Thetwoseven-weekMOOCswerebasedonasinglethird-year,on-campusunit,ENB339:IntroductiontoRobotics.ENB339isathirteen-weekQUTundergraduateelective(https://wiki.qut.edu.au/display/cyphy/ENB339+Introduction+to+Robotics).Thetwocoursesweredesignedtobesequential.ThismeantthatIntroduction toRoboticsprovided afoundation forRoboticVision.StudentscouldattemptoneorbothMOOCs.Studentswereencouragedtocompletethecourses insequence, butthiswasnotmandated,asenrolmentwasopentoanyone,atanytime,andparticipantscouldcompleteeitherorbothMOOCSinwhateverordertheywished,andre-enrolasoftenastheywanted.

TheMOOCsweredesignedforindividual,self-pacedlearning.Interactionsviathediscussionboard(whichwasopenatalltimes),andtheliveevents(whichwereheldthreetimespercourse)wereoptional,andsupportedbyfourteachingassistants(TAs).Itwasexpectedthatstudentswouldspendabout4-8hoursperweekstudyingeitherofthecourses.Studentsworkedthroughtheactivitiesatatimeandplaceoftheirownchoosing,withintheconstraintsofassignmentduedatesandreleaseofonlineresourcesonaweeklybasis.Collaborationwithotherstudentswasvoluntaryandself-organisinggroupswereencouraged.Thecourseresourcesprincipallyincludedshortthreetotenminutelecture-stylevideos,narratedanimations,recordedvideotutorialsupports,excerptsfromthee-textbook(providedfreeforthedurationofthecoursebythepublisherSpringer),computerprogrammingtutorialsandassessmentsusingMATLABsoftware,andtheonlinediscussionforums.ParticipantscouldformgroupsiftheywishedandsomeopenedrelatedFacebooksites.Forstudentstoreceiveacertificateofcompletiontheywererequiredtosubmitandpassfourquizzesandfourprogrammingassignments.Theseweremachine-markedonsubmission.

ThefirstMOOChadanoptionalpracticalprojectwherestudentswhowereabletoaccesstheLegoMindstormNXTrobotmakingkitorsimilar,couldbuildandprogramaworkingroboticarm.Studentswererequiredtovideotheirassembledandprogrammedrobotarmasitcompletedasettask.TheresultingvideowasthenuploadedtoYouTubeorasimilarmedia-sharingplatform,andtheURLsubmittedasevidenceforthepeerreviewprocess.Forthosewithaccesstothenecessaryresources,successfullybuildaworkingrobotcapableofcompletingarequiredtaskwasexpectedtobeahighlightofthefirstcourse.

Page 12: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

9

3.3 Methodology

3.3.1 Aim

TheaimoftheevaluationwastoinvestigatetheefficacyoftworoboticsMOOCs,IntroductiontoRoboticsandRoboticVision,anddeterminetheimplications,impact,issuesandconcernsforstudents,teachersandtheuniversityasaresultoftheinitiative.Datagatheredandanalysedwasexpectedtobeusedtorefinethetwocoursesforfuturedelivery,andinformthedevelopmentofotherMOOCs,andonlinecoursedevelopmentgenerallyatQUT.

3.3.2 Objectives

Theobjectivesoftheevaluationwereasfollows:

1. TodeterminethereachoftheprogramanddemographicsofMOOCparticipants.2. TodeterminepatternsofstudentengagementineachoftheMOOCS.3. TodeterminetheeffectivenessoftheteachingandlearningstrategiesemployedintheMOOCs

tosupportstudentlearning.4. Todeterminethesuitability(affordancesandfunctionality)oftheEdCastplatformasa

foundationfordeliveryandsupportoftheMOOCs.5. TodeterminetheimpactofissuesrelatingtouseoftheMOOCresources.6. Todetermineissuesofaccessandequitythatimpactthedesign,developmentanddeliveryof

theMOOCs.7. Todeterminefactors(e.g.technical,organisational,social,professional)thatimpactthedesign

andproductionoftheMOOCs.

Thisreportisframedinresponsetotheobjectives.Relevantobjectivesappearbeforeeachsection.AlistofindicatorsmappedtoeachobjectiveisprovidedinAppendixA.

3.3.3 Evaluation framework and the elearning lifecycle

TheframeworkthatunderpinsthisevaluationdrawsonthePhillips,McNaughtandKennedy(2012)designevaluationelearninglifecycle.Phillipsetal.arguethatinvestigationoflearningtechnologiesinvariablyincludesamixtureofevaluationandresearch,andtheyidentifyfivestageswhereevaluationand/orresearchtypicallyoccurinanelearningproject(seeFigure2).FromthePhillipsetal.perspective,researchisdefinedasinquirywhereinformationisgathered‘toinformourunderstandingofhowpeoplelearnusingane-learningartefactorenvironment’(p.1109);whereasevaluationismoreabout‘gatheringinformationtohelpmakejudgmentsaboutthevalueandworthofanelearningartefactorenvironment’(p.1109).Thisreportfallsintothelattercategory,andrepresentssummativeevaluationofanoperationalsystem(stage5).Itisexpectedthatfurthereffectivenessresearchintothelearningandteachingprocessesandoutcomeswillfollowinthenextstageoftheproject(stage6)whenthelargedatasetscollectedfromtheprojectcanbeinterrogatedmoredeeplyforwhattheycanrevealregardinga‘holisticunderstandingofhowlearnersengagewiththeelearningenvironment’(Phillipsetal.,2012,p.1112).

Figure2demonstratesthepositionofthissummativeevaluationintheelearninglifecycle.Thefigureshowsthatevaluationoccursthroughouttheproject,fromtheverybeginningwheretechnologiesandpedagogiesarechosen(stage0),throughthemanystagesofformativeevaluationateachpointinthedesigncycle(stages1-4),untilthelaststageswheresummativeevaluationtakesplace(stages5and6).Thecyclethenbeginsagain.AppendixCprovidesmoredetailonthelifecyclemodel.

Page 13: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

10

Figure 2: The design evaluation elearning lifecycle for the robotics MOOC project, 2013-2015. Based on Phillips, McNaught and Kennedy (2012).

3.3.4 Data collection and analysis

Giventhecomplexityoftheissuesrelatingtothisevaluation,amixedmethodsapproachwasadoptedfordatacollectionandanalysis.Siemens,GasevicandDawson(2015)confirmthisapproachasasoundoneforMOOCresearchandevaluation.MultiplesourcesofdatawerecollectedandtriangulatedanddetailsofthemethodologycanbefoundintheeLSdocument:EvaluationPlanandResearchDesignforRoboticsMOOCs.Inthisprojecttherewerestagesofformativeandsummativeevaluation,reflectingtheevaluationlifecyclestagesidentifiedinthePhillipsetal.(2012)model(seeFigure2).Inaddition,alow-riskethicsapplicationwassubmittedtotheQUTEthicsCommittee(1500000201)toallowforpublicationofthefindings,andasthebasisforfurtherresearch.

Thetypeofdatacollectedfortheevaluation(quantitativeandqualitative),alongwiththenumberssampledisoutlinedindetailinAppendixB.Insummary,however,datasourcesincludedstatisticsfromtheEdCastplatformandGoogleanalytics,theYouTubechannel,ProfessorCorke’sMATLABToolboxwebsite(http://www.petercorke.com/Robotics_Toolbox.html),discussionforumpostsandemailstotheQUTMOOCsupportteam,pre-,mid-andpost-coursesurveys,andstatisticsfromtheQUTMarketingandCommunicationDepartment.

Descriptivestatisticalwerecalculatedforthequantitativedata,andthequalitativedatawasanalysedusingconstantcomparativemethods(Charmaz,2011).Todate,quantitativeanalysishasbeenlimitedtoreportingonstudentsasacohortontheplatform.Individualswerenottrackedthroughthesystemasthiswasbeyondthescopeofthefirstinvestigation.Giventherequisiteresources,however,thisisanareaforfutureresearch(stage6oftheelearninglifecycle).

Page 14: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

11

4. Demographics and patterns of engagement Objective1:TodeterminethereachoftheprogramanddemographicsofMOOCparticipants.

4.1 Demographics

Thetwocoursesattractedhighnumbersofstudents,eventhoughQUTwasnotalliedwithconsortiasuchasEdXorFutureLearnthathaveestablishedglobalmarketingplatformsandstrategies,andMOOCcoursecatalogues.StudentrecruitmentreliedsolelyonapurposefulQUT-centricdigitalmarketingcampaign.Therewere12,894registrantsinIntroductiontoRobotics,and7,824registrantsforRoboticVision.Participantswerefrom161countries,withthetopthreecountriesbeingIndia,AustraliaandtheUnitedStates.ThetoptencountriesarelistedintheTable1.

Table1:Geographicallocationofstudents Introduction to

Robotics Robotic Vision

Enrolments from 161 countries (n=12,894)

161 countries (n=7,824)

Enrolment by country (top 10)

Country Students (%)

Country Students (%)

1 Australia 19 1 Australia 17 2 India 16 2 India 16 3 United States 14 3 United States 15 4 Egypt 3 4 Germany 3 5 Germany 3 5 Brazil 3 6 United Kingdom 2 6 Egypt 3 7 Canada 2 7 United Kingdom 3 8 France 2 8 Mexico 3 9 Mexico 2 9 Canada 2 10 Brazil 2 10 France 2

Note: Source: Google analytics Theglobalnatureofparticipationenabledaccessforstudentswhowouldnotnormallyhaveaccesstohighereducation.1

ThankyouProf.PeterCorkeincooperationwiththeQueenslandUniversityOfTechnologytoprovideopportunityforstudentindevelopingnationsespeciallyPapuaNewGuinea,whereROBOTICSisnottaughtasaCourseinComputerScienceandEngineering.(IR,forumpost)

Anothergreataspectisseeingpeoplefromallaroundtheworldparticipatingandhelpingeachother.AmazingtobeabletoparticipateinaglobalcourseinroboticsfromwayupinthewoodsinNorway.(IR,post-coursesurvey)

Themajorityofparticipantsforbothcoursesweremale(IR85%;RV88%),inthe20-30agerange(IR,52%;RV,53%),whoalreadyhadanundergraduatedegreeordiploma(IR&RV,38%)(seeFigure3,genderandage).ThisfitsthetypicalcourseregistrantprofileaccordingtoHoetal.(2014),butastheyexplainintheirHarvardXandMITxreport,registrantsarenotedfortheirdiversity.AnunusuallyhighnumberofregistrantsIntheQUTMOOCsdidnothaveuniversityqualifications(41%,seeFigure3,

1Note:Thespellingandgrammarofstudents’quotationshavebeenleftunedited.

Page 15: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

12

educationalattainment),and20%wereunder20yearsofageforthefirstroboticscourse.Thecurrentpopularityofroboticsamongstyoungenthusiastsmaybethereasonforthis,andthedropinnumbersofparticipantsinthisagegroupforthesecondcourse(downto13%)mayreflectthechallengeoftheincreasedcomplexityofthesecondcourse(seeFigure3).

Figure 3: Demographics for Introduction to Robotics and Robotic Vision. Graphs show gender, age and highest educational attainment.

Page 16: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

13

4.2 Student motivations, completions and patterns of engagement Objective2:TodeterminepatternsofstudentengagementineachoftheMOOCS.

4.2.1 Motivation for enrolment and pathways

SurveyresponsesregardingwhystudentsenrolledinthetwocoursesprovidedvaluableinformationaboutmotivationsforMOOCenrolmentsandconsiderationsforfuturecoursedesign.Forexample,surveyfiguresshowthatthemajorityenrolledinIntroductiontoRoboticsforreasonsofgeneralinterestorenjoyment(77%;n=5569).Thisfiguredroppedto25%(n=1689)forthemoreadvancedsecondroboticscourse,RoboticVision.Nonetheless,professionalinterestwasstillthesecondmainreasonforenrolmentinbothcourses(48%forIRand19%forRV;seeTable2).

Table 2: Motivation for enrolling (top 5 reasons)

Introduction to

Robotics %

Robotic Vision

% Survey responses (n) 5569 1689 General interest or enjoyment 77 25 Professional interest 48 19 I am studying university level robotics 34 12 Earning a statement of attainment 29 14 To help my employment prospects 24 9

Note: Source: Pre-course surveys. Participants could choose more than one option.

Asoneprogrammerexplained:

IamgladIdidthecourseasIhavebeenunemployedasaprogrammerfor2yearsandthiswillgivemealegup.BeforethisIhadtotryandgetagranttodoashortTAFEcoursebutwiththisbeingnearlyfree(datacostme$110)andself-serveredoingitIfeelIambetteringmyselfandgettingmyconfidencebackagain.StudentshelpeachotherwhereasjobapplicantsandcompetitorsarealwaysfightingsothiswasamuchbetterenvironmenttorebuildmyITandmathsskillsandconfidencetogetanITjobagain.Thanks.Greatideaandwelldone!(IR,post-coursesurvey)

CommentssuchastheoneaboveindicatethatopenonlinecoursessuchastheroboticsMOOCsmaywellhaveaplaceintermsofprofessionaldevelopmentandlearningpathwaysforstudents.ForsometheMOOCsfunctionedasrefreshercourses,oratasterofuniversity-levelstudyforthosethinkingaboutattendingregularundergraduatecourses.Eventhoughthiswasathird-yearcourseitclearlyprovidedopportunitiesandlearningpathwaysforanumberofstudents.ThissuggeststhatthereareopportunitiesforQUTtoconsiderotheropenonlinecoursesasuniversitybridgingcoursestargetingsecondarylevelandtechnicaltrainingmarkets.

4.2.2 Retention and completion rates

SinceMOOCsbeganattractingextraordinarilylargeenrolments,therehasbeenmuchdiscussionaboutretention(Freitas,Morgan,&Gibson,2015).Previousresearchhadindicatedthatcompletion ratescouldbeaslowas3–5%(Gallagher &Garrett,2013),7–9%(Knowledge@Wharton,November, 2012),andbelow13%(Onahetal.,2014).Thisofcoursedependsonhowretentionismeasured.

Page 17: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

14

ConcernsaboutcompletionshaveoftenoverlookedthemultitudeofreasonswhystudentsenrolinMOOCs(refertopfivereasons,Table2).Forexample,thereareparticipantswhoregisterforaMOOCbutneverenterthecourse;othersmayenterthecoursebutsimplybrowseforgeneralinterestortoviewthecoursedesignanddeliverymodels;andsomeregistrantsmayhaveagenuineinterestinviewingorstudyingthecoursetopic,butdonotwishtobeassessed.MITusesthenomenclatureofregistrant,forthosewhoregisterforanaccountonly,andparticipantforsomeonewhoactuallyentersthecourse(Hoetal.,2015).GenerallyinMOOCsstudentscancomeandgofromthesecoursesasoftenastheyplease,attemptingcoursesasmanytimesastheywantand,whileacourseremainsopenandonoffer,thestudenthasinfiniteopportunitiestoregisterandbuildontheirlearningattheirownpace.SotypicallythosewhocompletetheassessmentsandreceiveacertificateofparticipationforaMOOCareintheminority.ThiswasthecasefortheroboticsMOOCs.

Inaddition,giventhelimitationsofthetechnicalsystemsandmeansofgatheringanalytics,wewereunabletomatchindividualmotivationstooutcomes.Wehaveonlybeenabletotrackcohortbehaviournotindividualbehaviouratthisstage.However,wecanindicatebehaviourandtrendswithinacohort.Wecansaythat29%(1615)ofthosewhocompletedtheIntroductiontoRoboticspre-coursesurveyindicatedthattheyaimedtoreceiveacertificateofcompletion.621ofthecohortattainedapassgradeandreceivedacertificateinthatcourse.Theymayormaynothavecompletedthesurvey.InRoboticVision14%(236)indicatedtheirmotivationtoreceiveacertificate.475certificatesofcompletionwereawardedforthesecondMOOC.

4.2.3 Engagement patterns

Theanalysisofpre-andpost-coursesurveyandGoogleanalyticsprovidedsomeindicationofengagementpatternsbutatthesametimehighlightedtheneedtogainmorefocusedanddetaileddatafromtheplatformand/orGoogleanalytics.Surveydataindicatedthetimestudentsspenteachweekonthecourse.Amajorityofsurveyparticipantsclaimedtospendonaveragefourtosixhoursperweekstudyingthecourse(IR43%;n=385;RV46%,n=254;post-coursesurvey),butasignificantnumberspentsevenormorehoursperweekonthecourse(IR36%;RV28%).

Googleanalyticsenabledaviewofhowthecohortallocatedtheirtimeandattentionoverthedurationofthecourse.InIntroductiontoRoboticsthenumberofparticipantsonanydaywashighatthebeginningofthecourseuntiltheendofweek3.Inweeks0–3therewereonaverage1,439,1,338and1,006participantseachday,afterwhichthenumberslevelledoffaround500-800,withanincreaseinthelastweek.InRoboticVisiontheaveragenumbersofdailyparticipantswassteadierforthefirst4weeks(591,536,460and405),beforedroppingoffslightlyinweeks5-7(278,220,207).Itisdifficulttodeterminehowindividualsinteractwiththecourse;howeverthedataindicatesthat53%(IR)and60%(RV)ofthedailyvisitorswereactiveforumparticipants.Thereappearedtobeastrongcoregroupwhohelpedeachother,interactedwiththeTAsandcompletedtheassessmenttasks.Othersmayhavepursuedthecourseinisolationorwithofflinecohorts,butwedonothavedatafromthesecoursestoconfirmthisexpectedbehaviour.

ThesemultiplepatternsofengagementreflectthepurposefulMOOCdesignthataimedtosupportdiverselearningstylesandmotivations.AsourownandEdCastmethodsofgatheringanalyticsmature,itisenvisagedthatwewillbeabletomapindividualstudentmotivation,engagement,achievementandsatisfactionwiththecourse,therebybetterinformingfutureeducationaldesignapproachesandimprovingtheuserexperience.

Page 18: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

15

5. Response to the course design and learning and teaching strategies Objective3.TodeterminetheeffectivenessoftheteachingandlearningstrategiesemployedintheMOOCstosupportstudentlearning.Objective5.TodeterminetheimpactofissuesrelatingtouseofMOOCresources.

5.1 Student course satisfaction

Overallcoursesatisfactionwasveryhigh,indicatingthatthelearningandteachingstrategiesadoptedwereeffective.Inthepost-coursesurveys,studentsratedthecoursesaseitherexcellentorgood(96%IntroductiontoRobotics;92%forRoboticVision;seeTable3).Somestudentsalsosaidthatthecourseswereoneofthebestcoursestheyhadtaken(post-coursesurveycomments).Commentsonthediscussionforumalsoconfirmedthispositiveresponse.

Table 3: Overall course rating Introduction to Robotics (IR)

(n = 364) % Robotic Vision (RV)

(n = 246) % Excellent 62.9 53.7 Good 32.7 38.2 Satisfactory 3.0 6.5 Poor 0.8 1.2 Very poor 0.5 0.4 Total 100 100

Note: Post-course survey (IR: M = 4.57, SD = 4.43; RV: M = .650, SD = .707)

Onehighachieverrespondedasfollows:

Ithankyouprofessorforyourefforts.Thecoursewasattractiveandwellexplained.Mygradewasjustareflexionofthecourse'squalitycombinedwithastrongwilltolearnandachild'sloveforrobots.(email,IR)

Forsomestudentsitwasatransformativelifeexperience,asonestudentsaid:

It’sindeed[beena]lifechangingexperience,openingupnewvisionsandthoughtprocesses.(email,RV)

Theprojectisatreasurechestforenthusiasts.(RV,post-coursesurvey)

Theleadacademic’spassionforthesubjectwasalsooftencommentedupon,andconsideredbymanyasthebestaspectofthecourse:

Firstofall,thepassionofPeterCorkeforrobotics,it'sreallyreallymotivatingtobetaughtbyaprofessor,evenfaraway,inanotherlanguage,thatmakesyoufeelhe'sheretotakeyoutoanotherlevelofknowledge.(IR,post-coursesurvey)

Page 19: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

16

Commentsaboutanindividual’spersonalgrowthand‘theexcitementoflearning’asonestudentputit(IR,post-coursesurvey)indicatedthedepthofengagementforsomestudents.Forsome,thecoursesopenedupopportunitiestheywouldnothaveotherwisehad,andafewstudentssaidtheywouldkeepworkingonmakingtheirrobotsafterthecoursefinishedastheyranoutoftimetocompletetheirprojectintimeforpeerreview.Forexample:

CongratulationProfCorke.Yourcourseisexcelent.Iwillcontinuetoworkonmyrobotuntiliwillcompletethegiventask,evenifitwillbeoutsidethetimelimit.Itismyfirstrobot!!!:-)))(IR,post-coursesurvey)

Itseemedtobeverypractical,comparedtootherMOOC'sI'vetakeninthepast.IfeellikeIlearnedsomethingthatIcanstartputtingtouseimmediately.(IR,post-coursesurvey)

I'mamechanicalengineerandIusedthisknowledgeinmyworklastweek:aclientsentusaCADfilewithadifferentcoordinateframethanthedrawing.ThedifferentposeswereincludedinthedrawingsoIwasabletorecreatethecorrectcoordinateframeintotheCADfile.Mybosswasquiteimpressed.Thepointisthattheknowledgeacquiredinthiscourseisusefulinmanydifferentaspectofengineering.(IR,post-coursesurvey)

5.2 Success of the course design Thecoursedesign,includingthelearning,teachingandsupportstrategies,wasacollaborationbetweenProfessorCorkeandeLSstaff.Itwasconductedoverthree-months,atthesametimeascourseresourceswerebeingcollaborativelydevelopedandtestedbyProfessorCorkeandtheeLSMOOCteam(seesection3.2forcoursedescription).Whiletheproductionofhighqualityresourceswasitselfaconsiderablechallenge,thepedagogicalchallengeswereequallysignificant.Themainissuesforthedesignteamcentredaroundstrategiesformaintainingengagementofadiversemixofstudentsrangingfromhobbyiststoexperiencedprofessionals,agedbetween10and70,andtodesignforinteractivity,automatedassignmentmarking,andmeaningfulfeedbackonlearningtothousandsofstudents.Addedtothesecomplexities,theratioofteacherstostudentswasapproximately5:12,000).Allmembersofthecourseteamwerechallengedintermsofproblemsolving,learningnewskills,timepressures,communicationandworkload.Atthetimethattheleadacademic,ProfessorPeterCorkewasdevelopingMOOCresourcesincollaborationwitheLSstaff(e.g.videoscriptingandrecording,writingquizzes,reviewingcontent),hewasalsopreparingforthenewblendedlearning/flippedclassroommodelintherelatedENB339roboticscourseinsemester2,establishinganewcentreofexcellenceinroboticsatQUTandtendingtoalltheusualadministrativeandcommunityoutreachtasksrequiredofanyacademic.Inaddition,whiletheeLSteamhadconsiderabledepthofexperienceandexpertiseinmediadesignandproductionofblendedlearningresources,itwasnotsetupintermsofinfrastructuretospecificallyleadandmanagefullyonlinelearning.eLSstaffmemberswerealsooftensimultaneouslyworkingontheMOOCprojectaswellasotherQUTprojects. Despitetheseconstraints,asaresultofthestrategicapproachestakenbytheMOOCteam,thecoursedesignwassuccessful,asevidencedfromthepost-coursesurveysconfirmed.ManystudentscommentedpositivelyaboutarangeofcoursedesignelementsasTable4indicates.Studentsratedthelecturevideos,thetextbook,theformativequestions,theforums,emailcommunicationsandtheGoogleHangoutsasexcellentorgood(seealsoappendicesDandEforfulldetailsandsamplecomments).

Page 20: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

17

Table 4: Student rating of key course elements

MOOC course items rated as excellent or good IR (%)

RV (%)

Video 98 97

Textbook 88 88 ‘Check understanding’ questions 88 84

Forums 82 83

Email 82 83 Google Hangouts 80 82

Note: For full details see Appendix E, where percentages, means and standard deviations are provided.

Qualitativeresponsestoaquestioninthepost-coursesurveysaskingstudentswhattheythoughtwerethebestaspectsofthecoursewereanalysedintocategoriesandthenquantified(seeAppendixD).Themajorityofcommentsreferredpositivelytothequalityofthecoursematerials(IR38%;RV39%),inparticularthelecturevideos2,theanimationsandthedemonstrations.Studentswelcomedtherangeandchoiceoftopics(IR27%;RV22%).Theeffortputintothecoursedesignandproductiondidnotgounnoticed(IR36%;RV21%),andstudentscommentedonthetheclearinstructionsandexplanationofconcepts,themodularisedformat,theabilitytocontrolthespeedofthevideos,thetextbookasacomplementtothevideos,theinclusionofrealworldexamplesandthepacingoftheunit.TheyenjoyedinteractiveelementssuchastheliveandrecordedGooglehangoutsandtheopportunitytolearnwithandfromotherstudentsintheforumsandtheexperienceoftheTAs.

Thevalueoflearningtouseandapplymathematicalandprogrammingskillswasoftencommentedon(IR21%;RV19%),aswastheopportunitytoputtheoryintopracticeviatheformativeandassessablequizzes.TheyvaluedhavingaccesstothefreeMATLABlicenceforthedurationofthecourse,andProfessorCorke’sopensourceMATLABToolboxwhichextendedthefunctionalityofthebasicMATLABsoftware.Forthosewhohadtheresourcestodotherobotprojectitwaswellreceived.Studentsalsocommentedontheeffectivemixoftheoreticalandpracticalprogrammingtasks,andthefeedbackontheMATLABexercises.

ProfessorCorke’spassionforrobotics,hisskillexplainingcomplextopicsandabilitytorelatethetheorytoreallifescenarioswasoftencommentedon.Inaddition,thesupportprovidedbythewholeMOOCteamwasrecognised.Manystudentsalsocommentedonthegrowthintheirownlearningasaresultofparticipatinginthecourse.Afewstudentscommentedthattheywouldlikemoreroboticscoursesorthattheywouldre-enrolinthecoursestoimprovetheirlearning. Examplesofstudents’commentsonthecoursedesignfollow(alsoseeAppendixD):

Theeffortthathasgoneintothecoursereallyshows.(IR,postcoursesurvey)

Thevideocoursesbymr.peterweresoeffectiveandsoeasytounderstandbecauseheexplainsthemwithpicturesandtoysratherthanformulaeandequations.(IR,post-coursesurvey)

2 Foranexampleofthevideossee‘OutandAboutwithRobots’:https://www.youtube.com/watch?v=GF1DwjAFIWg

Page 21: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

18

Thelogicalflowandthewaytheknowledgewaspresentedwasexemplary!Assignmentswereverynicelydesignedwhichreallytestedtheconceptslearnt.Itisagreatcourseforsomeonelookingtogetajumpstartinthefieldofroboticvision.(RV,post-coursesurvey)

Itwasjustlikeacompletestoryorcompletemovie.Thediverseinformationprovidedinaninterestingmannermakecourseveryattractive.Difficultconceptswereexplainedineasyandnicemanner.(RV,post-coursesurvey)

Thelectureswereextremelygood.Allcontentwaswellexplained.Alltheproblemsthatcouldbefacedbyuswastakenintoconsiderationbeforehandandsupportmaterialwasgivenaccordingly.(IR,post-coursesurvey)

5.3 Level of difficulty: designing for a mixed cohort of students

Whilesomestudentswerecomfortablewiththelevelofdifficultyandenjoyedthechallengeandthepaceofeitherorbothcourses,othersstruggled;anothergroupwantedevenmoredifficulty.Thisundoubtedlyreflectsthevarieddemographicofthecohort,andtheeducationalandlanguagebackgroundsofparticipants.ThecoursewasprovidedinEnglishonly,andassumedknowledgewasarticulatedontheregistrationpagesoftheMOOCwebsites.LinkswereprovidedtomathematicsandprogrammingtutorialsattheKhanacademy(https://www.khanacademy.org/),andMATLABtutorialswereprovidedbyMathWorks.Sothoughthesesupportswerehelpfulforsome,thelevelofdifficultywassetatthatsuitableforathird-yearundergraduateengineeringunit.Consequently,asolidfoundationinmathematicsandprogrammingwasessentialforthosestudentswantingtodomorethanwatchthevideos.CommentsinTable5showtherangeofabilitiesandthevariedstudentresponsestothestandardexpected(seeTable5).

Table5:Variationinresponsetothelevelofdifficulty

Level of difficulty Comment

Struggled with the difficulty

I'm spending many hours per week on the course but I'm still struggling to keep up. (IR, mid-course survey)

Appreciated the challenge

It is not that easy for someone like me lacking a technical background but it is still a lot of fun, hard but awesome (always loved math) and everything is very well done. (RV, mid-course survey) I'm also left with a lot of pride in the progress I made and all that I learnt. I left high school with the impression that I was no good at maths. In this course I really showed myself that I am capable of learning and applying maths - especially when there's a goal which is so compelling to me. (IR, post-course survey)

Course perceived as well within student’s capabilities

It is so far a great course. It is very basic. I have already completed a Robotics course in University, but this is an excellent refresher course for me. I am looking forward to the vision course after this. (IR, mid-course survey) The problems were great. Very interesting. The questions can be harder :) (RV, post-course survey) Recommended reading and links to background material make it suitable for diverse audience. (RV, post-course survey)

Page 22: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

19

ResponsessuchasthoseaboveandtheexistenceofanenthusiasticroboticshobbyistcommunityindicatethatthereispotentialtoofferadditionalQUTMOOCspitchedatvariouseducationalentrylevels.ThiswouldalsosupporttheideaofpathwaysthroughfreeMOOCcoursesintofee-payingQUTundergraduateorpostgraduatedegrees.Further,theteachinganddevelopmentteamssuggestanotherinnovativeideaworthpursuingisthebuildingofaresourcebaseforrobotics,similartotheKhanAcademy(https://www.khanacademy.org/).

5.4 Assessment

Asexplainedabove(section4.2.2),courseregistrantscouldengagewiththecourseinanumberofways,butthoseparticipantswhosuccessfullycompletedallassessabletasksreceivedacertificateofparticipation.Theoverallassessmentwasworth240points,comprisingassessablequizzes(120points)andMATLABprogrammingtasks(120points).Toqualifyforacertificatestudentshadtoachieveanoverallscoreof50%(120points).Thequizzesandprogrammingtaskswereweightedequally,soitdidnotmatterhowthe120pointswereaccumulated.621studentsreceivedacertificateofparticipationforIR,and475forRV.Twenty-sevenstudentsparticipatedsuccessfullyinpassingtheoptionalpeer-reviewedrobotbuildingproject.(SeeAppendixFforadditionalfiguresregardingassessment.)

Theassessmentwaschallenging,andwassetatthird-yearengineeringstandard;however,theremaybeacaseforreviewingtherequirementsnecessarytoachieve‘completion’standard,versusverifiedaward‘credential’standard.Nonetheless,foronestudentwhometthestandardandthechallenge,therewardwassignificant:

IamdeeplygratefulandalsoveryproudtoearnthecertificateasIfinishedmystudy8yearsago(electricalengineeringinGermany)andfeelalittlebitoutofbusinessduetomyjobasflighttestengineer.MyjobhasalmostnothingtodowithwhatIstudiedandyourgreatcoursegivesmethepossibilitytowarmupagain:).Forme,itwasveryhelpfultohavethevideolecturesandtowatchthevideosseveraltimeshelpedalot.(IR,post-coursesurvey)

5.5 Engagement strategies and community

Considerableeffortwasputintodesigningandimplementinganumberofstudentengagementstrategiesinordertobuildasenseofcommunity,supportstudents’learningneeds,andtopromptlyresolveanytechnicalissuesthatarose.Thisincludedmonitoringoftheasynchronousdiscussionforumsandemailmessaging,threesynchronouseventsconductedusingGoogleHangouts,aroboticsT-shirtdesigncompetition,andface-to-face‘meetups’withProfessorCorkewhilsthewastravellingintheUSA.

5.5.1 The centrality of the discussion forum

Staffandstudentscommentedonthefunctionalityofthediscussionforumtool.Itwaslessthanintuitivetonavigateandonlyallowedforconversationsfromonetomany;noone-to-oneorprivatemessagingwaspossible.Nonetheless,theTAs,eLSsupportstaff,ProfessorCorke,andtheMATLABrepresentative,BrandonArmstrong,madeeveryefforttorespondtostudentqueriesandkeeptheforumsactive,livelyandproductive.

Theteachinganddevelopmentteamrecognisedthatmanagementofthelearningcommunityduringthedeliveryofthecourseswascritical.Thereforeacommunicationsplanwasdevisedforthemanagementoftheforums,andspecificcommunitymanagementroleswereassignedtomembersoftheteam.FourTAsweretrainedinhowtorespondmeaningfullyandstrategicallytothousandsofstudentpostsandmessages.TheTAsmetregularlywiththeMOOCteamtoshareexperiencesanddevelopstrategiestomanagethediversestudentcohort.TheTAsrespondedtostudentqueries,

Page 23: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

20

moderatedthediscussions,andposteditemsofinteresttofocustopicdiscussions.Inadditiontheycollecteddataaspartofdocumentationandevaluationprocesses.

Asindicatedearlier,studentsratedthequalityandvalueoftheforumsasexcellentorgood(82%forIR;83%forRV;seeTable4andAppendixE).Theforumswerethemaincommunicationspaceforstudents,andanextremelyimportantself-helpmechanismforstudents.Thepost-coursesurveyscontainedmanycommentsabouttheimportanceoftheforumsasresourcesforstudentstohelplearnersachievetheirgoals.Someratedtheforumsandpeer-to-peerinteractionasoneofthebestaspectsofthecourse(seeAppendixD).Thefollowingstudentcommentsareillustrativeofthegeneralmoodofstudents.

Generallythecoursewasgood.Butthediscussionforumwasawesome.(IR,post-coursesurvey)

[Thebestthingaboutthecoursewas]beingabletogethelpfrompeers.(IR,post-coursesurvey)

ThankyousomuchProfessorCorkeandthesupportstaffforthiswonderfulcourse.AnequallybigthankyoutoalltheotherparticipantsinthecoursewithoutwhomiwouldnothavebeenabletoachievehalfthescoreIhave.Hopetoseeyouguysinthesequel.(IR,forum)

Theforumswerealife-saverasotherseitherhadsimilarideasandhelpedmeorIhelpedthem.IthinkforumsisvitalforMATLABandQuizzeswherethereislittleorpoordefinitionofquestions.YoucorrectedquestionsthathaderrorsinquizorMATLABafterfeedbackandgavepeopleasecondchance.(IR,post-coursesurvey)

StudentsidentifiedbytheTAsandthecommunitymanagementteamashavingmadeasignificantcontributiontoeithertheIRorRVcoursecommunitieswerealsoacknowledgedviaemail.

5.5.2 Live events: Google Hangouts

ThreesynchronousInterneteventsforeachMOOCwerehostedusingGoogleHangouts3.Liveone-houreventswerehostedeitherbyPeter,theTAsoramemberoftheMOOCteam.Thesewererecordedandwhileonlyasmallnumberofstudentsattendedtheeventslive(IR,143;RV,92),manyviewedtherecordingslateron(IR,2424;RV,944).Nodoubttimedifferencesacrosstheglobeaccountforlowliveattendancefigures.StudentscouldalsosendinquestionsaheadoftimeandProfessorCorkeandtheteamwouldanswertheseduringtheliveHangout.

Thehangoutswereratedhighly(excellentorgood;80%forIR;82%forRV;seeTable4andAppendixE).Studentsgavefeedbackthattheseopportunitieswerewellappreciatedasstudentscould,forexample,hearstoriesaboutProfessorCorke’sinterestingcareerwithrobots,orlearnabouttheTAsresearchprojects.

ParticipatingintheeventsandaskingquestionslivewerealotmorefunthanIanticipated.IespeciallylikedthatIwasabletounderstandandaskabouttheTAs'research.Atthebeginningofthecourse,Idon'tthinkIwouldhaveunderstoodthedetailsnorwouldIhavebeenabletoaskthequestionsthatIdidask.(RV,post-coursesurvey)

ThevideosofPeterCorkewereverygoodathangouts.Ienjoyedhimreminiscingabouthispastroboticdays.(IR,post-coursesurvey)

3GoogleHangoutsallowforlive,streamedvideochatandinstantmessaging.Uptotenparticipantscancontributesimultaneouslytoasinglevideosession;anynumbercanwatchthelivesession.

Page 24: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

21

5.5.3 Other engagement activities

OtheractivitiesthathelpedbuildaspiritofcommunityandcreativeengagementincludedaroboticsT-shirtcompetitionwherestudentswereinvitedtosendintheirdesignsforaT-shirt.CourseparticipantsvotedonthebestdesignandthewinnerwasawardedafreeT-shirtwiththeirowndesignprintedonit.Anotheractivitythatsparkedsomefriendlyandhumorousrivalrywasalivelydiscussionaboutthecricketworldcup.AustraliaandIndiaaretwomajorcricketingnationsand,asAustraliaandIndiaarethecountriesfromwheremoststudentsoriginated,therewasalevelofinterestinthisinternationalsportingevent.

Also,duringthesecondMOOC,ProfessorCorketravelledtotheUSAforbusiness.Duringthattimehesentoutageneralinvitationtostudentsviatheforum,tomeetupwithhimatthevariouscitieshevisitedatdifferenttimes.Someenthusiasticroboticistswereabletotakehimuponthisoption.Thispreparednesstomeetstudentsshowsthededicationoftheleadacademic,theabilityoftheMOOCteamtocontinuetofacilitatethecourseintheabsenceoftheleadacademic,andthepowerofmobiledigitaltechnologiesthatallowforlearningandteachingtooccursuccessfullyfromalmostanywhereontheglobe.Thisclearlypaidoffandasonestudentsaid:

Iamveryimpressedwithallinvolvedintheteachingthiscourse.Iamateacherinmyfield,butthepassionanddedicationshownbyProfCorkeandhisteamisamazing.(IR,forum)

5.6 Access and equity

Objective6.Todetermineissuesofaccessandequitythatimpactthedesign,developmentanddeliveryoftheMOOCs.

Issuesofaccessandequityneedattentioninanyhighereducationcourse,butespeciallyintheonlineenvironmentwherethepurposeistoofferfreeandopenaccesstostudentswhoaregeographicallydispersed.Issuesthatwerenotedinthiscourse,forexample,relatedtoaccesstocoursematerialsfortheoptionalrobotmakingproject.StudentsexplainedthatforsomethecostoftheLegoMindstormNXTrobotmakingkit(c.$AU500)wasprohibitive,evenifpurchasedsecondhandfromeBay.Also,oncepurchased,timelydeliveryofthekittomeetcoursetimeconstraintsstillhadtobefactoredin.Asonestudentcommented:

ItistoohardtopurchaseaMindstormKit,insteadofthat,IorderedaArduinoMega...onebay,butitwilltaketimetoarrivetoSriLanka.(RV,mid-coursesurvey)

Andanother:

EventhoughIhavelotsofArduinoparts,Ihadtoskiptheoptionalproject.Thatbummedmeoutalittle.ButnothingsaysIcan'tgobackanddoitlater.:-)(IR,forum)

AnotherfactortobeconsideredinthedesignprocesswasthatcertainregionsororganisationsdidnotpermittheuseofYouTube,theonlinevideochannelthathousedmostofthelecturecontent.Whilethecoursewaspromotedas‘free’,andtherewerenoregistrationfees,therewerestillpersonalcoststhatastudentmustbeabletopayaroundInternetconnectivity,datacharges,andmaterialcosts.ThankstoMathWorks,theMATLABlicencewasprovidedfreetostudentsforthedurationofeachcourse,andexcerptsfromthetextbookwerealsoprovidedasPDFsatnochargebySpringeronthecoursewebsite.Toengageintheoptionalrobotbuildingprojectstudentsalsoneededacameraorcameraenabledmobilephonetorecordtheirrobotinaction.

Nonetheless,onestudentsaidthatthebestthingaboutthecoursewasthatitwas‘openforeveryone’(IR,post-coursesurvey):anyonecouldenrolinthecourse.OneofthemosthearteningstoriescamefromaSyrianstudentattendingthecourse.Havingescapedfromhisworntorncountryhewasverygratefulforthefreecourse,andevensentvideofootageofhisbombedhomebackinSyria.Thathe

Page 25: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

22

couldcontinuelearninginsuchdramaticcircumstanceswasimmenselyinspiring,andshowsthepowerofeducationtobringaboutchangeandprovidehope.

TheprovisionofcoursessuchastheroboticsMOOCstostudentswithnoaccesstosimilarcoursesintheirowncountryisatremendousexampleofcommunityoutreachonthepartofQUT.Asonestudentcommented:

Trulyspeakingyoudidfantasticwork.Inmycountryroboticsisnotmuchdeveloped.Irarelyfindanycoursehere.Yourlectureislikeboontostudentlikeus.Videosofthislecturesareverysimpleandtothedepth.Itwillbegreatifyouprovidemorecoursesbecausewereallyneedit.THANKYOUagainsir.(RV,email)

6. Technical and pedagogical innovations Theplatformprovider,EdCast,andtheprogrammingsoftwareproviderMathWorksworkedhardtocollaborativewithQUTtodevelopandintegratetoolsintothenewlyengineeredplatform,usingemergenttechnologiesandavailablesocialmediaapplications.ThisensuredthatthelearningtechnologieswerecustomisedandsupportedtomeetQUT’spedagogicalneeds.TherewerethreeinnovationsintegratedintotheEdCastplatformthatdeserveparticularacknowledgment.Theywere:(1)theintegrationofGoogleHangoutsasliveeventsheldwithinthecourse;(2)thepeerreviewsystemfortherobotproject,whichtrainedstudentsineffectivepeerreviewandmanagedtheassessmentoftheoptionalstudentprojects;and(3)theMATLABauto-gradingassessmenttool.ThesethreeitemshadnotpreviouslybeenintegratedintotheEdCastplatform,andQUTdrovethisinnovation.Becauseofthiseffort,moststudentsfoundthesiteeasytouse,andwhilethereweresometeethingproblems,thesewereremediedasthecourseprogressed.Theexceptionwasthediscussionforumtoolthatwaslessthansatisfactory,butnonethelessstaffandstudentsmanagedtoworksuccessfullywithit.AnadditionalinnovationwasthereuseofMOOCresourcesintheoriginalundergraduateroboticscourseENB339.

6.1 The EdCast platform and collaboration with industry partners

Objective4.Todeterminethesuitability(affordancesandfunctionality)oftheEdCastplatformasafoundationfordeliveryandsupportoftheMOOCs.

Thechoiceofplatformwaskeytothewholeproject.AfterexplorationofvariousMOOCconsortiaoptions,QUTchosetheopensourceplatformofferedbyEdCast(www.edcast.com).EdCastisaStanfordStartXcompanybasedinCalifornia.EdCastworkedincollaborationwithQUTfrommid2014tobuildacustomisedplatform,whichincludednewtechnicalinnovationstailoredtoQUT’sneeds.TheserversusedtostorecoursedataweremaintainedonshoreinAustralia,inSydney.EdCastmaintainedthecourseplatformprogrammedwithOpenEdXsourcecode,andprovidedQUTwiththenecessaryMOOCservices,including24/7technicalsupport.

Whiletheplatformchoicemeanttherewereopportunitiesforinnovation,asEdCastwasnotpartofaMOOCconsortium(e.g.likeEdXorCoursera),theadvantagesofgroupmarketingwerenotachievable.(Thisisdiscussedfurtherinsection7.4.)However,bypartneringwithEdCast,QUThadclearownershipofthecoursecontent,andfullrightstorepurposeandre-useanyresourcescreatedontheplatform.

Becauseofthisproductivecollaborationwithindustrypartners,moststudentsfoundthesiteeasytouse,andwhilethereweresometeethingproblems,thesewereremediedasthecourseprogressed,orworkaroundsputinplace.

Aswithanytechnologytherearealwayslimitationsandaffordances.Twoparticularlimitationsoftheplatformwerethelookandfeelofthediscussionboard,andtheneedtodevelopalltheaccompanyingprocessesandprotocolsforanopensourceproduct.Whilegeneralprinciplesregardingthefunctionality

Page 26: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

23

ofalearningmanagementsystemcanbetransferredfromoneplatformtoanother,eachplatformhasitsownspecificfeatures.Thereforetimewasrequiredforteaching,supportandtechnicalstafftotestoutandlearnthecapabilitiesofthenewsystem.Forfutureprojects,sufficienttimeforthisprocessmustbefactoredintothedevelopmentschedule.

TherelationshipwithEdCastwasakeypartnership,andwhilebothEdCastandQUTwerenewtoMOOCdesignanddevelopment,overallthepartnersworkedwellontheinitiative.EdCastprovided24/7technicalsupportforstaffandstudentswhichwasessentialforacourseofferedgloballyacrossmultipletimezones.QUTwouldhavepreferredmorenoticeanddocumentationaboutchangestotheplatformwhilethecourseswereinprogress,butthiscommunicationissueisnowbeingaddressed.Althoughdesignissuesemergedwiththelackofeaseofuseofthediscussionforumtool,functionalityofthequiztoolandreliabilityofthegradebooktool,errorswerecorrectedduringthecourse,andworkaroundsfoundforproblemsthatcouldnotbesolvedatthetime.

Overall,theEdCastplatformwassuitableforofferingaworld-classonlineroboticscourse,androbustenoughtodelivertwocoursestoover20,000students.Thiswasaconsiderableachievement.Thesameplatformisbeingusedtodeliverthesecoursesinthesecondhalfof2015,whichindicatesthattheplatformwasfitforpurpose.TheissuesaroundcommunicationtoolsremainthemainconcernandQUTisworkingwithEdCasttoimprovethefunctionalityanduserexperienceofthesetools.

6.1.1 Integration of MATLAB auto-grading tool

IncollaborationwiththeMATLABdevelopers(whomaintainedtheprogrammingsoftwareproducedbyMathWorks),QUTandEdCastpioneeredtheintegrationofanassessmenttooltoautomaticallygradeandprovidefeedbackontheweeklyprogrammingtasks.Thetoolhadtobereliableenoughtoreturnaccuratefeedbackonstudents’programming,androbustenoughtobescaleduptomanagetheassessmentofthousandsofstudents.ThecustomisedMATLABtoolalsohadtopassstudentassessmentresultsbacktotheEdCastplatformgradecentre.Whilesomeissueswereencounteredintheearlystagesofthefirstcourse,EdCast,MATLABandtheQUTMOOCteamworkedquicklytorectifyproblemssothatstudentswerenotdisadvantagedduetotechnicalglitches.Figure4showsthatgenerallythestudentsweresatisfiedwiththeirexperienceusingMATLABandthesupportingMATLABtutorials(seeFigure4).

Page 27: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

24

Figure 4: Experience using MATLAB. Note: 1 = Good, easy to use, useful, no issues; 2 = mixed or qualified answer; 3 = poor, had problems or difficulties; 4 = other. The results are based on a post-course survey question: ‘Please describe your experience using the MATLAB component integrated in with the assignment’. Responses were open-ended. Comments were analysed and divided into 4 categories, and then assigned a percentage, relative to all the qualitative responses for that question.

Asdiscussedabove,MATLABgrantedthesoftwarelicencefreetostudentsforthedurationofeachcourse.MATLABalsopreparedcustomisedvideotutorialstohelpstudentsgainthemostoutofthesoftware,andthiscombinedwithProfessorCorke’sRoboticsToolboxforMATLAB(http://petercorke.com/Robotics_Toolbox.html)optimisedapproachestotheprogrammingtasks.AmoreindepthreportonstudentresponsetoMATLABcanbefoundintheeLSMATLABReport,butinsummarystudentsworkedwellwiththeMATLABtoolandthosewhousedtheMATLABvideosfoundthemhelpful.Whenaskedtodescribetheirexperienceusingthetoolstudentscommentedthatitwasan‘excellenttool’alongwiththeToolboxwhichwas‘amazing’(IR,post-coursesurvey).SomestudentshadfrustrationsandwantedmoredocumentationfortheToolbox,ormoreexamplesintheproprietaryvideotutorials.

Doingtheprogrammingassignmentsilearnedthemost.(IR,post-coursesurvey)

MATLABexercisesweregreatbutverytough.(IR,post-coursesurvey)

IthinkyourstyleofteachingwithMatlabandnotgoingintoverydeepisperfectforengineerslikeme.Ihavearegularjobintelecomengineeringandparticularlyin5Gdevelopment,butformeRoboticsisserioushobby.(IR,highachiever,email)

ProfessorCorkeisreallygood.I’mprettybadinmathbutImostlyunderstoodalloftheformulas.Ithinkisreallyimportanttoexplainwherethevariouspartofanequationarecomingfromsothestudentcanhaveagoodandvisualmappingbetweenmathandreality.Comparedtootherfreecoursesonlinethiswasbyfarthebest.InsteadoffeelingfrustratedbecauseIhaven’tunderstoodthecontentsofformulasthrownatme,afterthiscourseIactuallyfeelIlearnedalotandIhaveagoodfoundationtogodeeperonthissubject.(RV,post-coursesurvey)

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4

IRPost-coursesurvey(n=223)

RVPost-coursesurvey(n=145)

Page 28: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

25

6.1.2 Automated online peer review

OneofthemajorinnovationsoftheMOOCprogramwastheonlinepeerreviewsystem.IntroductiontoRoboticsincludedanoptionalpracticalprojectwherestudentswereofferedtheopportunitytobuildandprogramafunctioningrobotarmusingaLegoMindstormKit.Althoughthisdidnotcounttowardscertification,successfulstudentsreceivedacertificateofachievementformakingtheirrobotarm,alongwithapersonalemailofcongratulationsfromProfessorCorke.Twenty-sevenstudentscompletedthepeerreviewproject,andexamplesofcompletedprojectscanbeseenonYouTube:https://youtu.be/CpMm0j1zB8o.

Becauseofthesizeofthecourse,therobotprojectsneededtobeautomaticallymarked.EdCast,incollaborationwithQUTdesignedanditerativelydevelopedasuitablepeerreviewtoolandprocessforthis.

Thosestudentswhooptedtobuildtherobotarmandthecontrolsoftwarefortherobotwerementoredthroughtheprocessviaweekly,pre-recordedvideodemonstrations.Thedemonstrationswerecreatedusingvideoandscreencapturetechnologies,andpresentedonscreenbyaQUTPhDengineeringstudent(LiamO’Sullivan).Thetutorwalkedstudentsthroughtherobotbuildingandprogrammingtasks,replicatingthesupportstudentswouldreceiveinalaboratorysessionintheundergraduateunit,ENB339.

Itshouldbenotedthatifthecourseweretobeshortened,thetimeforstudentstosetupandprocurethenecessaryresourcesforthecoursewouldbeadesignconcern.MOOCstendtorangefrombetween2-12weeks,with6weeksbeingatypicallength(Bayne&Ross,2014),andashortMOOCispreferableformaximisingstudentretention.Withcoursessuchasroboticswherestudentsneedtoprocureequipment,lead-timeforthismustbefactoredintothecoursedesign.

OncestudentshadcompletedtheirrobotmakingforIntroductiontoRobotics,theywereaskedtovideotherobotperformingthesettask.ThevideoevidencewasuploadedtoYouTubeorasimilarvideohostingwebsite,andtheURLsubmittedsothatpeerreviewcouldtakeplace.Onlythosestudentswhohadalsocreatedarobotandsubmittedavideowerepermittedtoengageinthepeerreviewprocess.

Topreparestudentsfortheprocessofpeerreviewingothers’work,anautomatedpeerreviewtrainingprocesswasdevelopedbyEdCasttechnicalstaff,theQUTlearningdesignersandQUTTAs.Projectstobemarkedwererandomlyallocatedtoeachstudentbythesystem,andeachstudentgradedthreeprojectsbeforethegradecentrereleasedtheirownmark.

6.1.3 Textbook publisher support

Asindicatedearlier,anagreementwasrealisedwiththepublisherSpringer,allowingfreeaccesstoexcerptsfromProfessorCorke’stextbook(Corke,2011).Theseexcerptswereuploadedase-bookchaptersviathecoursewebsites.Purchaseofthetextbookwasoptional,howeverthepublishersofferedthetextbookforpurchaseasanebookorinsoftcovertextatadiscounttoregisteredstudents.

Page 29: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

26

6.2 Reuse of course resources for ENB339

In2014,aselectionofMOOCresourceswaspilotedinanauthentic,livesetting.TheMOOCresourceswerereusedintheQUTroboticsengineeringunitENB339:IntroductiontoRobotics,theunitfromwhichtheyhadoriginallybeenderived.Thispilotingofdesignedresourcescorrespondstostage3oftheelearninglifecycleasoutlinedinFigure2andAppendixC.Principallythelecturevideosweresuccessfullyrepurposedinthethird-yearunitinmodularisedformat,aspartofablendedlearningapproach,incorporating‘flippedclassroom’4pedagogicalstrategies.

Becauseofthedecisiontoembedthemodularisedvideoresourcesinthesecondsemesterunit,therewasconsiderablepressuretopreparetheresourcesintimeforimplementationwiththethird-yearclasses.TheexperienceisdescribedindetailintheEvaluationReportforENB339:ReuseofMOOCResourcesandtheFlippedClassroom.Itissufficienttosayherethatthisformofcontentreuse,combinedwithawell-designedandsupportedflippedclassroommodelwassuccessful.Studentcoursesatisfactionratingoftheunitimproved(datafromQUTInsightsurvey),andlecturersandtutorssawadvantagesinthemodel.ThewillingnessoftheteachingteamandeLSstafftocollaborativelyengageindevelopingthisnewlearningandteachingmodelwaskey.Itwasclearthatstudentsneededongoingcoachinginthenewlearningandteachingapproach,andthisnewapproachsuitedsomebutnotallstudents.Somestudentsneededmoresupporttobecomeself-regulatedlearnersandbettermanagetheirtime.

OneofthemaintechnicalissuesrealisedthroughtheENB339trialofMOOCresourceswastheissueofstreamedvideocontentovermobileWi-Finetworks.Manystudentswouldhavepreferredtheoptiontodownloadthevideofilesastheycouldeasilyexceedtheirdataquotasonmobiledeviceswhenaccessingcoursecontentvia,forexample,theirsmartphone.TheissueofdatacostsandquotaswassimilarlycommentedonintheMOOCs,althoughthisdidnotemergeasamajorissue,basedonpost-coursesurveysandforumcomments.SoaswiththeMOOC,whilestudentsinENB339gainedgreaterflexibilityintermsofwhenandwheretheyaccessedtheresources,atthesametimeotheraccess,equityanddatacostissuesemergedasnewconsiderations.

Nonetheless,despitesometechnicaldifficultiesthemajorityofENB339studentsaffirmedthatthenewpedagogicalmodelwasagoodone,thatis,theflippedclassroomincorporatingrepurposedMOOCresources,andtheyratedtheunitas4.7(outof5)intheendofsemesterInsightSurvey,2014.

TheexperienceoftriallingtheuseoftheMOOCresourcesinamoremodestsettingwithundergraduatestudentswasanimportantphaseofthedesignanddevelopmentelearningcycle.TheENB339trialalsocomplementedthesubsequentpre-coursetrialofRoboticVisioninDecember2014.Importantly,hadinsurmountableproblemsforstudentsorteachingstaffemergedduringthetrialstudentswouldnothavebeendisadvantagedasthelecturerscouldalwayshaverevertedtothetraditionalmodelofunitdelivery(thelecture/tutorialmodel).Further,astheresourceswerereusedinthenextofferingofENB339,insemester2,2015,thisindicatestheefficacyoftheintegrationofMOOCresourcesandthenewblendedlearning/flippedclassroompedagogicalapproach.

4Theflippedclassroomisamodelofteachingwherelowerlevelcognitivetasks,suchasgaining‘firstexposuretoknowledge’(Brame,2013;Walvoord&Anderson,1998)occursoutsideofclass.Moretimeisdevotedinclasstohigherorderthinkingandapplyingandprocessingknowledgewithguidancefromtheteacher,andoftenincollaborationwithpeers(Brame,2013).

Page 30: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

27

7. Institutional implications Objective7.Todeterminefactors(e.g.technical,organisational,socialandprofessional)thatimpactthedesignandproductionoftheMOOCs.

7.1 Production issues TheMOOCprojectsrequiredconsiderableresourcesintermsofstafftimeandeffort.Aninnovativeelearningenvironmentwascreatedandthecoursewassuccessfulfrommanyperspectives.Typicalofanydesign-basedinitiative,therewereaplethoraofunknownsanduncertaintiestobeaccommodatedandresolved.Managementhadtobeflexibleandresponsive,andsecuringandretainingtherightmixofstaffwiththenecessaryskillsfortheprojectwaschallenging.

AsFigures1and2show,theMOOCteamfollowedadesigncycleincludingphasesofanalysisoftheproblem,designoftheMOOC(educationalandinterfacedesign),prototyping,testinganddevelopment,piloting,andimplementation.Allstageswereunderpinnedbyevaluation,asindicatedintheelearninglifecycleinFigure2andAppendixC.Evaluationwaseitherformativeorsummative.Sooveratwenty-fourmonthperiod,theMOOCwentfromconceptionthroughtothefirstimplementation(approximately19months),anddeliveryofthefirsttwoMOOCs(5months).

Foranumberofreasonstherewasadelayinselectionoftheplatformandthisimpacteddesignandproductionschedules.NonethelesstheMOOCwasdeliveredontime.AttheconclusionofeachoftheMOOCs,adebriefingsessionwasheldwitheLSstafftocapturelessonslearnedandinformthenextiterationofeachcourse.

Thefindingsfromtheformativedebriefingsessionshavebeenwrittenupasaseparatereport(MOOCProgram:LessonsLearnedfromDesignandProduction),butsomeofthekeyissuesaresummarisedbelow.

1. TheelearningdesignlifecycleandproblemsolvingAdesignapproachnecessarilyincludesphasesforiterativelygeneratingideas,rapidprototypingandtesting,qualityassurance,implementationandsupportandevaluation.Sufficienttimemustbefactoredinforallstaff(design,productionandtechnical)tosolveproblemsandresearchthefieldasnecessarythroughouttheproject.Asdigitaltechnologieschangerapidly,soonthejoblearningisextremelyimportant.

2. ConceptualdevelopmentTheMOOCteam(teachinganddesignandproductionstaff)needtimetobuildtrustandconceptualunderstandingofthespecialrequirementsofonlinelearningintheMOOCenvironment.Rolesandresponsibilitiessharedamongstthedesignandproductionteam,andtheteachingteamneedtobenegotiated,andtimeallocatedforthisaspectoftheprocesstomature.

3. QualityassuranceTheprocessofqualityassuranceisextremelyimportantforaprojectwithsuchhighpublicexposureasaMOOC.Thismeansthatsufficienttimeandresourcesmustbeallowedforconceptdevelopment,coursedesignandproduction,securingofcopyrightpermissions,iterativetestingofallaspectsofthecourse,transcription,editingandproofing,andstudentandtechnicalsupportduringimplementation.

Page 31: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

28

4. Copyright

Tracingandsecuringcopyrightpermissions,ornon-copyrightalternativesisaparticularlytime-consumingprocess.MediausedwithinaMOOC,inparticularartisticworks(e.g.photographs,maps,graphs)thatQUTdoesnotownarenotcoveredbythesamecopyrightandfairuseregulationsthatgovernstandardAustralianuniversitycourses(e.g.itemscoveredbyPartVBoftheCopyrightAct1968).Itcanbethecasethatevenaftermuchsearchingorwaitingforaresponsefromthecopyrightownerpermissionmaynotbegranted.

5. SystemsWithanewelearninginitiativesuchastheMOOCproject,therearefewsystemsinplaceforalloperationalandsupportmatters.Thisincludestechnicalprocesses,datastorageandmanagement,workflowsandguidelines,andlearninganalytics.Theseallassistwithstreamlinedandsustainablepractices.Wherethesesystemsdidnotexist,newsystemshadtobeestablishedandpiloted.

Thesetwoinnovativecoursesweredesigned,developed,implementedandsupportedwithanintensityofstaffeffortthatisnotsustainableoverthelongterm.Theriskisstaffburnout,andalossofexperiencedstaffmeansalossofvaluableprojectandcorporateknowledge.However,assystemsareputinplacetodocumentandregulateworkflowpatternsandprocesses,somedesignandproductionissueshavebecomeeasiertomanageandtrack.

7.2 Evaluation and learning analytics

Johnsonetal.(2011)definelearninganalyticsas‘theinterpretationofawiderangeofdataproducedbyandgatheredonbehalfofstudentsinordertoassessacademicprogress,predictfutureperformance,andspotpotentialissues’(p.29).TheoverwhelmingamountofdatarealisedthroughtheMOOCshasalreadybeenusedtoinformthedesignofthenextphaseofMOOCdesignandproductionand,astheMOOCsunfolded,statisticsweregatheredfromtheEdCastplatform,GoogleAnalytics,YouTube,socialmediasourcesandthecoursesurveystoinformlearningandteachingdecisionsasthecoursesprogressed(seeAppendixB).

AslearninganalyticsisanewareaofbusinessforQUTanduniversitiesingeneralthereisstillaneedtodeviseandrefinethenecessarysystemssoanalysisandreportingcanbetimelyandinformtheimplementationof,designandredesignofQUTMOOCs.eLSisestablishingprocessesandworkingwiththeQUTStrategicIntelligenceUnitinordertofurtheranalysecurrentandfuturedata.ThedatacanbeusedtodeterminenotonlycohortpatternsofengagementinQUTMOOCs,butalsoindividualbehaviours,preferencesandoutcomes.Thereisroomhereformuchresearch.

Oneofthemainissuesinthisareaoftheprojectwastheissueofrecruitmentandretentionofstaffwithexpertiseinlearninganalytics.ThisreflectswhatGrajek(2015)hasidentifiedasoneofthetoptenITissuesof2015,namely‘hiringandretainingqualifiedstaff,andupdatingtheknowledgeandskillsofexistingtechnologystaff’(p.12).

7.3 Staff development and innovation

Staffneedtimetofamiliarisethemselveswithanewlearningmanagementplatform(LMS),todiscoverandmasteritsaffordancesandlimitations.Thisincludes,academics,teachingassistants,learningdesigners,webdevelopers,videoproductionstaff,communityandtechnicalsupportstaff.Duringthedepartmentaldebriefingandfeedbacksessions,theeLSMOOCteamremarkedthattheyvaluedtheopportunitytocreativelyaddressandresolvechallenging,non-routineproblemsaspartoftheMOOCdesignandproductionprocess.Thisspacetoinnovateandtakerisks,toworkcollaborativelywith

Page 32: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

29

industrypartnerstodeveloptechnologiesandprocessestosupportonlinelearningwashighlyvalued.ThiskindofprofessionallearningonthejobisinlinewiththevisionandvaluesoftheUniversity,asoutlinedinQUT’sBluePrint4(2014)whereitstatesthattheUniversityvalues‘aspiritofexperimentation,innovation[and]entrepreneurialism’(p.4).Toencourageongoinginnovation,itisimportantthatexperimentationissupportedintheMOOCenvironmentatQUT.Thismeansprovisionforstafftimetopursueandresearchdevelopmentsinthefield,andtoreflectonlessonslearnedtofurtherinformdevelopmentsinonlinelearning.

7.4 Marketing

QUTmarketingspecialistsdevelopedahighlyeffectivecustomiseddigitalmarketingcampaign.Recognisingthehighlevelofinterestinrobotics,thecampaigntargeteddomesticandinternationalmarkets.ThecampaignresultedinfrontpagecoverageinTheAustralian,andarticlesinmanyglobalmagazinesincludinginComputerWorld,TechworldandPDadvisor.ThesinglemosteffectivestrategywastheuseofaScienceAlertonFacebook(https://www.facebook.com/ScienceAlert),whichaccountedfor124,214uniquepageviews(1/12/2014–3/9/2015).Figure5showsthespikesincourseregistrationsfollowingpostingtoScienceAlert.Thebenefitsoffocussedmarketing,beyondthehighnumbersofregistrations,includeddrawinginterestedpartiestotheQUTwebsite,withtheQUTlandingpage(https://www.qut.edu.au/study/open-online-learning)receiving162,973uniqueviewsbetween1December2014and3September2015.EdCastdescribedtheQUTcampaignasexceptionallysuccessfulincomparisonwiththeirotherclients(seeFigure5).

Figure 5: Marketing campaign related to MOOC enrolments. Note: The Science Alerts were posted on Facebook on 7 Feb and 8 April 2015.

ThereisnowaneedtoincreasegeneralawarenesswithinQUTabouttheMOOCprogramandtheroleitplaysmorebroadlyinthedevelopmentofonlinelearningopportunities.AdditionallyQUTrequiresanongoingmarketingstrategytoraiseawarenessofitssuiteofMOOCs,andpathwaystootherQUTcoursesandresearchopportunities.ThereisalsoagrowingcommunityofMOOCparticipantalumniwithaspecificinterestinrobotics.ThissignificantcommunityoflearnersmaybeinterestedinfurthercontinuingeducationopportunitiesofferedbyQUT.

Page 33: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

30

8. Conclusion

8.1 General conclusions

Thepurposeofthisdocumentwastoreportontheeffectivenessoftheelearningenvironmentregardingthedesign,productionandimplementationoftwoQUTMOOCs.Asetofsevenobjectivesfocusedtheevaluation.

Thefindingsindicatedthatthecourseswereofaveryhighstandard.Thiswasthecaseintermsofthequalityandappropriatenessoftheresourcesprovidedandtheeducationaldesignandsupport.Studentswereprovidedwithavariedsetofcourseresourcesthatincludeddemonstrationsandlectures‘inthefield’.Theoryandpracticewerecombinedinauniqueway,oftenusingexamplesfromtherealworldofrobotics.Studentswerechallengedthroughassessablemathematicalandprogrammingtasks,whileatthesametimebeingsupportedinafunandengagingway.Thediscussionforumwasakeycommunicationtoolforpeer-to-peerlearningandwaswellsupportedbytheMOOCteam.

TheroboticsMOOCsreachedadiverseinternationalaudiencein161countries,with20,718registrants.1,096certificatesofparticipationwereissuedforsuccessfulcompletionoftheassessmenttasks.Thecourseswerechallengingsoftwareengineeringcoursesbasedononeelectivethird-yearunit,andyettheyattractedhobbyistswithoutformalqualificationandpostgraduateswithvaryinglevelsofcompetencyinmathematicsandcomputerprogramming.Coursesatisfactionwasextremelyhighforbothcourses.

Asexpected,registrationsintheearlyweeksofeachcoursewerehigh,withactiveparticipationdroppingoffmarkedlyafterthefirstfewweeks.Themainreasonforenrollingineithercoursewasforpersonalinterest,whichsuggeststhatattaininganawardorpassingassessmenttaskswerenotthegoalsofmanyregistrants.Alargeproportionofstudentsmayhaveenrolledineithercoursefortheopportunitiesitofferedforinformalratherthanformallearning.

Someissuesofaccessandequitywerenoted,mainlyinrelationtoacquiringadditionalresourcesfortheoptionalrobotmakingproject.Howeveropenandfreeenrolmentintothecourses,andtheprovisionoffreelicencesfortheMATLABprogrammingsoftwareplusexcerptsfromthetextbookforthedurationofthecourse,maximisedpossibilitiesforwidestudentparticipation.

TherewereteethingproblemswiththeEdCastplatform,butEdCastprovidedasufficientlyrobustplatformforsuccessfuldeliveryofcoursestothousandsofsimultaneousparticipants.BypartneringwithEdCastandMathWorks,QUTpursuedopportunitiesforinnovationincourseprovisionandassessment.SolutionsdevelopedforthiscontextcannowbeusedtoinformonlineandblendedlearningatQUTgenerally.

TheefficacyofreusingresourcescreatedfortheMOOCsinanothercontextwasproven.ModularisedvideoswereintegratedbackintoENB339,theQUTunitfromwhichtheMOOCsoriginated.Thisreusewasaccompaniedbypedagogicalinnovationusingaflippedclassroomapproach.Itwasanexampleofinnovationsinpedagogyandtechnologytogetherdrivingchange.

TheMOOCprojecthighlightedinstitutionalissuesaroundproduction,learninganalytics,professionaldevelopmentandmarketing.ItwasclearthatproductionoftheMOOCswasachievedthroughtheeffortsofacommittedandinnovativeteam,whichincludedstafffromeLearningServicesandtheFacultyofScienceandEngineering.Theprojectdemonstratedthattomaintainquality,oncetheinitialprojecthadbeenestablished,therewasanurgentneedtocreatesustainableproductionandtechnicalsupportsystems.Manycomplextechnical,designandmanagementproblemswereencounteredinthisproject,andthecapacityofstafftoproactivelyfindandimplementworkablesolutionswasnotable.

Page 34: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

31

Twofinalissuesemergedasimportantfindingsinthisevaluation.Firstly,theprojectdemonstratedthevalueoflearninganalytics,andtheirplaceininformingcoursedesign,andunderstandinglearnerbehaviourandmotivationsincontext.Secondly,theimportanceoffindingthemosteffectivecommunicationschannelformarketingthecourseswascrucial:locatingacommunitythatwaslikelytobemostinterestedintheseparticularcoursescontributedtocoursesuccess.ScienceAlertwasthesolutionforrobotics.

8.1 Future research and opportunities for innovation

ThesuccessfulimplementationofQUT’sfirstMOOCsinroboticssuggeststhattheUniversitycouldwelldevelopacatalogueofmassiveopenonlinecoursesthatalignwithdevelopmentsinQUT’sGraduateProfessionalEducation(GPE)program.Thiswouldbeanopportunitytobuildnewcoursesaroundcurrentundergraduateandpostgraduateunits.ThesuccessoftheENB339experimentregardingthereuseandrepurposingofMOOCresourcessuggeststhatsimilarinitiativescouldbeundertakenasMOOCcoursesaredevelopedforotherdisciplinesatQUT.Theseareallareasforfutureresearchandinnovation.

ThenextphaseofinnovationisexpectedtofocusontheintroductionofMOOCsfromotherdisciplines,capacitybuildinginthosealreadyinprogress,improvementofthelearnerexperienceandrefinementofassessmentandcommunicationtools.Itisanopportunityforpedagogicalandtechnologicalinnovation.StrategicthinkingisrequiredtodesignpathwaystoconnectcurrentQUTawardcoursesandthenewopenonlinecourses.Thisdiversificationofofferingwillrequireexaminationofcertificationoptionsanddevelopmentofnewsustainablebusinessmodels.Thecreationofaroboticsresourcebase,similartotheKhanAcademy,isanexampleofapossibleareaforfutureinnovation.Alongsidetheseopportunitiesforinnovation,QUTwillneedtosupportprocessesforcontinuousimprovementtoensurethatthecurrentMOOCcoursesmaintaintheirpositioninthesector,andthattheUniversitycontinuestoinnovateanddeveloprelationshipswithindustrypartners.

Thenextstagetobepursuedintheelearninglifecycleisstage6:Evaluationresearchonthematuresystem(seeFigure2andAppendixC).Thisiswhereamoreholisticunderstandingofhowlearnersengagewiththeelearningenvironmentcanbeinvestigated.Thelearninganalyticsalreadygatheredfromtheroboticscoursesshowthatthereisawealthofdatathatcouldformthebasisforfurtherinquiry.Inparticular,theUniversitymaywishtodirectefforttowardsuncoveringmorearoundthepictureofstudentdemographicsastheyrelatetoachievementandoutcomes.Thereisalsomuchtobelearntaboutstudentlearninginthiscontext.Asyetweknowlittleabouthowstudentsactuallylearninthesemassiveopenonlinecourses,andwhatengagementwithsuchonlinecoursesmeansforstudentsintermsofcareerandprofessionaldevelopment,andthereforeQUT’sroleasanelearningprovider.

Acknowledgements

Projectssuchastheserequirethetimeandcommitmentofalargeteam.Wegratefullyacknowledgetheexceptionaleffortsoftheacademicteam:ProfessorPeterCorke,teachingassistantsObadiahLam,JamesSergeant,JackSlaterandEdPepperell,andAssociateProfessorMichaelMilfordforhisENB339contributions.WethankQUTMarketingandCommunication,inparticularSimoneLong,andourindustrypartnersEdCast(especiallyDanielMcKelveyandTonyJiang),MathWorks(inparticularBrandonArmstrong),andSpringerpublishing.FinallywethanktheeLearningServicesteamfortheircommitmenttothedesign,production,technicalservicesandlearninganalyticscomponents:TabethaBozin,LuisaRossitto,LuzetteDonohue,PaulFenn,ChrisNewman,AndrewThomson,JasonShanks,JasonMarkland,LiamMoushall,RogerCook,DavidDennis,KushanYapa,DrTristanPhilip,DanielCrane,StephenBooth,KimManningandDesleyGorle.Wethankeveryonefortheirexceptionalenthusiasmandsupportforthisproject.

Page 35: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

32

References Bayne,S.,&Ross,J.(2014).ThepedagogyoftheMassiveOpenOnlineCourse:TheUKview.Retrieved

fromhttp://www.heacademy.ac.uk/assets/documents/elt/HEA_Edinburgh_MOOC_WEB_240314.pdf

Brame,C.(2013).Flippingtheclassroom.CenterforTeaching,VanderbiltUniversity.RetrievedOctober

30,2015fromhttp://cft.vanderbilt.edu/guides-sub-pages/flipping-the-classroom/.

Charmaz,K.(2011).Constructinggroundedtheory.LosAngeles,CA:Sage.Corke,P.(2011),Robotics,visionandcontrol:FundamentalalgorithmsinMATLAB.Berlin,Germany:

Springer.

Freitas,S.I.,Morgan,J.,&Gibson,D.(2015).WillMOOCstransformlearningandteachinginhighereducation?Engagementandcourseretentioninonlinelearningprovision.BritishJournalofEducationalTechnology,46(3),455–471.doi:10.1111/bjet.12268v

Gallagher,S.,&Garrett,G.(2013,31July).Disruptiveeducation:Technology-enableduniversities.Retrievedfromhttp://ussc.edu.au/ussc/assets/media/docs/publications/130801_DisruptiveEducation_GallagherGarrett.pdf

Grajek,S.(2015).Top10ITissuesof2015:Inflectionpoint.EDUCAUSEReview,50(1),10-48.Retrievedfromhttp://er.educause.edu/articles/2015/1/top-10-it-issues-2015-inflection-point.

Ho,A.D.,Ho,A.D.,Reich,J.,Nesterko,S.,Seaton,D.T.,Mullaney,T.,...Chuang,I.(2014).HarvardXandMITx:Thefirstyearofopenonlinecourses,Fall2012-Summer2013,HarvardXandMITxWorkingPaperNo.1.Retrievedfromhttp://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263

Johnson,L.,Smith,R.,Willis,H.,Levine,A.,&Haywood,K.(2011).The2011horizonreport.Retrievedfromhttp://www.nmc.org/pdf/2011-Horizon-Report.pdf

Onah,D.F.O.,Sinclair,J.,&Boyatt,R.(2014,7-9Jul).Dropoutratesofmassiveopenonlinecourses:Behaviouralpatterns.InEDULEARN14Proceedings.6thInternationalConferenceonEducationandNewLearningTechnologies(pp.5825-5834).Retrievedfromhttp://wrap.warwick.ac.uk/65543/1/WRAP_9770711-cs-070115-edulearn2014.pdf

Phillips,R.,Kennedy,G.&McNaught,C.(2012).Theroleoftheoryinlearningtechnologyevaluationresearch.AustralasianJournalofEducationalTechnology,28(7),1103-1118.http://www.ascilite.org.au/ajet/ajet28/phillips.html

QUT(2014).BluePrint4.Brisbane,QLD:QueenslandUniversityofTechnology.Retrievedfromhttps://cms.qut.edu.au/__data/assets/pdf_file/0013/71113/qut-blueprint.pdf

QUT(2015).QUTDigitalRoadMap:Our2020Vision,v3.1,July.

Siemens,G.,Gasevic,D.,&Dawson,S.(2015).Preparingforthedigitaluniversity:Areviewofthehistoryandcurrentstateofdistance,blended,andonlinelearning.Retrievedfromhttp://linkresearchlab.org/PreparingDigitalUniversity.pdf

Walvoord,B.E.,&AndersonV.J.(1998).Effectivegrading:Atoolforlearningandassessment.San

Francisco:Jossey-Bass.

Page 36: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

33

Appendices

Appendix A: Objectives, sample indicators and data sources Objective Sample Indicators and Data Sources

1. To determine the reach of the program and demographics of MOOC participants.

Number of unique ID logins by country; age, educational level, access patterns via device; EdCast/Google analytics, pre-, mid- & post-course survey results; discussion forum posts & emails

2. To determine patterns of student engagement in each of the MOOCS.

Comparison of access by level of activity (registrants, active participants, participants who received certificate); formative & summative assessments attempted; discussion forum posts & emails; feedback from teaching & MOOC team; surveys.

3. To determine the effectiveness of the teaching and learning strategies employed in the MOOCs to support student learning.

Certificates of completion issued, robot project completions, student responses on discussion forum & in surveys, feedback from TAs and Professor Corke (interviews, focus groups, discussion forums, emails, logs, groups)

4. To determine the suitability (affordances and functionality) of the EdCast platform as a foundation for delivery and support of the MOOCs.

Student affirmations on discussion forum, survey feedback, responses from TAs, Professor Corke, eLS team, and MATLAB rep (interviews, focus groups, debriefing sessions, emails, technical issues logged & resolved, data from MOOC pilot, 2014).

5. To determine the impact of issues relating to use of MOOC resources.

Student affirmations on discussion forum, survey feedback, responses from TAs, Prof. Corke, eLS team, & MATLAB rep (interviews, focus groups, debriefing sessions, emails, technical issues logged & resolved, MOOC pilot).

6. To determine issues of access and equity that impact the design, development and delivery of the MOOCs.

Student feedback from discussion forums & emails, surveys, staff feedback via interview/focus group, & the discussion forum, EdCast & Google analytics, data from MOOC pilot, 2014

7. To determine factors (e.g. technical, organisational, social, professional) that impact the design and production of the MOOCs.

Feedback and reports from eLS web, media and learning design staff, status reports, analytics and feedback from EdCast re issues, solutions & workarounds; data from the MOOC pilot, report from Marketing & Communications.

Page 37: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

34

Appendix B: Data sources, sampling and data types Source Sampling Quantitative (Quan)

or Qualitative (Qual)

EdCast – student activity tracking; formative & assessable quizzes; MATLAB programming

Statistics collected weekly from student records (IR, N = 12894; RV, N = 7824)

Quantitative

Google analytics Statistics collected weekly from student records

(IR, N = 12894; RV, N=7824)

Quantitative

YouTube lecture streaming Sampled at intervals over the semester Quantitative

Robotics Toolbox http://www.petercorke.com/Robotics_Toolbox.html Quantitative

Pre-course survey IR, Feb-June 2015, N= 5569; RV, Apr-June 2015,

N= 1689

Quan & Qual

Mid-course survey IR, March 2015, N = 234; RV, May 2015, N = 102 Quan & Qual

Post-course survey IR, April 2015, N = 383; RV, June 2015, N = 256 Quan & Qual

Discussion forums Sampled on TAs feedback report sheet Quan & Qual

Emails from participants Issues collected on JIRA tracking system Qualitative

eLS debriefing workshop 13 participants; April 2015 Qualitative

eLS feedback session 8 participants; August 2015 Qualitative

Page 38: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

35

Appendix C: Details of the evaluation elearning lifecycle for the project

Cycle Lifecycle stage Development activity

Evaluation Research Role of theory and design principles

0 Analysis of problem

Document the problem: online learning at a massive scale, for robotics

Baseline analysis: environmental scan and assessment of user needs.

Define teaching and learning problem based on scholarship

1 Design elearning artefact

Course design planned collaboratively – eLS and SEF.

Ongoing formative evaluation of design

Design based on principles of elearning best practice

2 Prototype elearning artefact

Prototype MOOC assets independently & on EdCast platform

Formative evaluation of the first MOOC and project management processes

Reflect on MOOC characteristics and potential for re-use of resources (conference poster)

Refine principles of elearning best practice for context

3 Piloting: Design elearning environment and conduct pilot study

Trial of MOOC on EdCast development site (Nov- Dec 2014) & in ENB339 (semester 2, 2014)

Formative evaluation of the elearning environment

Refine principles of elearning best practice for context

4 Implementation: Refine elearning environment and conduct full trial

IR MOOC released: Feb 2015; RV April 2015

Formative evaluation: eLS debriefing/feedback session; student surveys and data captured (analytics)

Effectiveness research into learning processes

Initial learning design principles

5 Evaluation research on operational system

Confirm effectiveness of elearning environment

Summative evaluation: Robotics MOOC Report 2015

Effectiveness research into learning processes and outcomes

Refined learning design principles

6 Evaluation research on mature system

Holistic understanding of how learners engage with the e-learning environment

Summative evaluation: Deeper investigation of MOOC learning analytics

Effectiveness research into learning processes and outcomes

Refined learning design principles

Note: This evaluation elearning lifecycle is for the robotics MOOC project for the period 2013- 2015. It is based on the Phillips, McNaught and Kennedy (2012) elearning lifecycle model. The table shows the development activities at each stage and the related evaluation and/or research activities that occur at any particular stage. This evaluation report represents Stage 5 activities, where the system is live and operational (see also Figure 2).

Page 39: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

36

Appendix D: The best aspects of the course In an open-ended question participants were asked about the best aspects of the course. Each response was coded to one or more categories. Categories were derived from the data, not imposed from a pre-determined list. Comments could be positive, negative or relate in other ways to the category. Frequencies were then expressed as a percentage of the total responses to the question (IR = 247; RV = 154). Category Example comment IR

(n=247) (%)

RV (n=154)

(%) Course materials (lectures videos, animations, textbook, demonstrations)

‘The videos were excellent. Most lectures in and outside of the classroom consist of instructor reading bullet points, but professor Corke really knows how to visually complement his lectures.’

38 39

Design of the MOOC (e.g. pacing of the unit, imagery interactivity, modules, learning and teaching strategies, structure, explanations

‘All content was well explained. All the problems that could be faced by us was taken into consideration before hand and support material was given accordingly.’

36 21

Topics covered ‘Started at a base level of understanding, and covered a large range of topics in brief, to really support understanding and interest in the wider vision field.’

27 22

Mathematical and MATLAB learning tasks and programming

‘MATLAB exercises were great but very tough.’ 21 19

Prof. Corke’s passion and teaching approach, and/or support from the MOOC team

‘The passion of Peter Corke for robotics, it's really really motivating to be taught by a professor, even far away, in another language, that makes you feel he's here to take you to another level of knowledge.’

11 14

Students comments about their own learning

‘I think the best aspect of the course is that I learned a lot, I hope to use all this all this learning in a future. I think also that the material of the course is really good and liked the way the quizzes and Peter Corke teaches’

11 8

Forums, community, discussions & interactions with others

‘The socialization of knowledge regarding vision and robotics great importance in mechatronics.’

8 6

Everything – the whole course ‘The entire course was great. The level is perfect. The pace is great.’

7 5

Quizzes (formative and summative)

‘Good reinforcement of teaching material through quizzes and programs.’

6 4

Prof. Corke's MATLAB Toolbox ‘Another useful thing is the explanation of the toolbox it took years of coding and it is quite useful to understand using it.’

4 6

Robot project ‘Hands-on with Matlab and with real hardware in the optional project.’

4 1

Level of difficulty ‘I think the level of math was about right for an introductory course. Clearly there is a lot more math behind that covered in the lectures, but encapsulating much of that deeper math in MatLab routines allow us to focus on the concepts rather than getting lost in the math.’

2 3

Google hangouts ‘I really enjoyed the google hangouts. They made the course feel much more social than previous MOOC's I've taken.’

2 1

Flexibility the course offered ‘Ease of learning from anywhere.’ 0 2

Page 40: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

37

Appendix E: Quality of the resources In the post-course surveys (IR and RV), participants were asked to rate the video lectures, the textbook, the ‘check for understanding questions, the discussion forums, email communications, and the Google Hangouts. The results are expressed as percentages of the total number of responses to each question.

Q4. Please rate the quality and value of the video lectures. IR (n= 382)

% RV (n=251)

% Excellent 77.7 72.5 Good 20.2 24.7 Satisfactory 1.6 2.0 Poor .5 .8 Very poor 0.0 0.0 Total 100.0 100.0 (IR: M = 4.75; SD = .5000; RV: M = 4.69, SD = .550)

Q4. Please rate the quality and value of the textbook.

IR (n=301) %

RV (n=185) %

Excellent 48.5 45.4 Good 39.2 42.7 Satisfactory 10.3 9.7 Poor 2.0 2.2 Very poor 0 0 Total 100.0 100.0 (IR M = 4.34, SD = .743; RV M = 4.31, SD = .737) Q4. Please rate the quality and value of the ‘check understanding’ questions.

IR (n=377) %

RV (n=249) %

Excellent 40.6 30.1 Good 47.7 54.2 Satisfactory 10.9 14.5 Poor .8 1.2 Very poor 0.0 0.0 Total 100.0 100.0 IR M = 4.28, SD = .685; RV M = 4.13, SD = .692

Q4. Please rate the quality and value of the discussion forums.

IR (n = 344) %

RV (n=2321) %

Excellent 37.2 39.4 Good 44.8 43.7 Satisfactory 14.2 13.0 Poor 3.8 3.9 Very poor 0 0

Total 100.0 100.0

IR M= 4.15, SD = .802; RV M = 4.19, SD = .805

Q4. Please rate the quality and value of the email communications.

IR (n=295) %

RV (n=188) %

Excellent 40.3 37.8 Good 42.0 44.7 Satisfactory 14.2 13.8 Poor 3.4 3.2 Very poor 0.0 0.5 Total 100.0 100

(IR M = 4.19, SD = .804; RV M = 4.16, SD = .818)

Page 41: Notice Please note that this document may not be the ...eprints.qut.edu.au/103552/1/Robotics_MOOC_Report_Final.pdf · (MOOC) (September 2013) • Evaluation Plan and Research Design

Robotics MOOCs Evaluation Report 2015

38

Q4. Please rate the quality and value of the Google Hangouts.

IR (n=206) %

RV (n=108) %

Excellent 36.4 38.9 Good 43.7 43.5 Satisfactory 16.5 13.9 Poor 3.4 2.8 Very Poor 0.0 0.9 Total 100.0 100.0 (IR M =4.13, SD = .807; RV M= 4.17, SD = .837)

Appendix F: Assessment Assessment: Introduction to Robotics

Quizzes (EdCast) - one per week Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6

Attempted 2280 1156 851 785 670 574

Passed 1713 766 735 622 538 381

Percentage passed 75% 66% 86% 79% 80% 66%

Tasks (MATLAB) - one per week

Attempted 1060 723 634 434 284 194

Passed 1060 723 597 319 242 157

Percentage passed 100% 100% 94% 74% 85% 81% Note: 621 certified participants received a certificate of participation having passed the assessable quizzes and MATLAB tasks. 27 passed the peer reviewed optional robot building project. Assessment: Robotic Vision

Quizzes (EdCast) - one per week Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6

Attempted 1246 858 682 598 536 481

Passed 1208 709 567 475 504 230

Percentage passed 97% 83% 83% 79% 94% 48%

Tasks (MATLAB) - one per week

Attempted 869 567 387 271 177 158

Passed 869 529 323 271 177 158

Percentage passed 100% 93% 83% 100% 100% 100% Note: 475 certified participants received a certificate of participation having passed the assessable quizzes and MATLAB tasks.