MedbiqxAPIWorkshopReport

Authors:DavidTopps1,CoreyAlbersworth2,EllenMeiselman3,MaureenTopps1,SarahTopps4,SandraMorrison1

1UniversityofCalgary2AthabascaUniversity3UniversityofMichigan4SimonFraserUniversity

Background:Howdoweknowourlearnersarecompetent?Uponwhatdowebasethis?Observationsandjudgmentsfromteachers,usingITERsandMCQsformthebasisofthevastmajorityofourassessments.Whiletheseworkwellforthemajorityoflearnersandteachers,wearelesseffectiveatearlydetectionofmaladaptivelearnersorthosewhoarestruggling.Wetendtogivethebenefitofthedoubt,thinkingthisakindness,butdelayeddetectionanddiagnosisoflearnersindifficultyisdoingnobodyafavor.Inourhighschoolmathtests,weareconcernedthatlittleJohnnynotonlygottherightanswerbutusedtheappropriatestepsandtechniquesinderivingthatanswer.Similarly,inmedicaleducation,weshouldbeconcernedwithassessmentofmorethanmemorization,andlookatproblemsolvingandefficient,effectivetaskcompletioninavarietyofcontexts.Weshouldlookatwhatourlearnersdo,notwhattheyortheirteacherssaytheydo.Hence,theriseininterestinactivitymetrics.ThisconceptisatthepeakofMiller’sPyramidofAssessment(Miller,1990)demonstratingprofessionalauthenticity,movingfrom“knows”to“knowshow”to“showshow”to“does”(althoughdoubthasbeenraisedabouttheevidencetosupportMiller’spyramid,butwedigress(Lalley&Miller,2007)).Thisisnotanewconcept,butourabilitiestomeasuresuchactivitieswiththenecessaryrichnessofdetailhaslargelybeencompromisedbycostandpracticalityandhence,werelyoncompoundobservationsandjudgmentsofhumansensors–ourteachers.Butnow,withtheplethoraofcheapdevicesandsensors,thepossibilitiesforgatheringmeaningfuldataaboutthelearningprocesshavegreatlychanged.Inourpreviousstudies,usingmixedsimulationmodalitiesinourHealthServicesVirtualOrganization(HSVO)project,wewereabletocombinemultiplelearningtoolsandsystems,usingasophisticatednetwork-enabledplatform.Wecombinedvirtualpatients,high-fidelitysimulators,low-latencyvideocollaboration,virtualanatomymodels,andlight-fieldcameraarraystogeneratevirtualpointsofview.Wewereabletocreatepowerfullearningdesignsforcollaboratinggroupsoflearnersacrossmultipledisciplines,institutions,andcontinents,inreal-timewithmanyconcurrentdatastreamstrackingtheireveryaction.

Bothlearnersandteachersenthusiasticallyadoptedthesemulti-modalitysimulationdesigns.Weobservedaprogressionofskillsdevelopmentandcollaborativeproblemsolvingapproachesinallthegroups.Analysisoftheactivitystreams,whichwaslargelybasedonmanualcodingofvideotapedactivities,provedtobeextremelylaborious,withahighcostineffort.Wedidobserveintenseclustersofactivity,centeredonboundarychangesincontext,somewhatanalogoustophasechangesinsimplerbiologicalandphysicalprocesses.Thevigorousup-spikeinsuchactivitieswasoftenhardtokeepupwith,andsometimesmissedbyboredassessorsskippingthroughlongperiodsofminimalstatechange.Humansdonotdowellwiththisapproach–machinesensorsaremuchmorereliableandefficientinsuchmonitoring.TheHSVOprojectdemonstratedsomeextraordinarylearningactivitiesbutwasaverycostlyendeavor,withhighinfrastructureneeds(user-directedlightpipes,highperformanceclustercomputing),highlyskilledpersonnel,anddependenceonhighlevelsoftechnologywithlimitedportability.Whilethecoremiddlewarelayer,Savoir,wasdesignedtoactasanintermediarybetweenmanydifferentdevicetypesandsystems,itwasnotveryflexibleandrequiredanoverlytightdegreeofintegrationandbindingbetweenApplicationProgrammingInterfaces(APIs).Inmanyways,thiswassimilartotheproblemsencounteredwithattemptingtoblendinformationstreamsacrossothereducationalplatformsusingSCORM.ItwasaboldattemptatthetimetointegratedatafrommanysourceswiththecoreLearningManagementSystems(LMSs).ItswasdependentonthecentralstructureoftheLMS-anarchitecturalmodelwhichisnotparticularlysuitabletocurricularstructureofmostmedicalschools.ItsrestrictivedatastructuresmadeSCORMvulnerabletoalltheongoingchangesateachsideofeverysysteminterface.Thisverychallenge,facingtheadoptersofSCORM,drovethedevelopmentoftheExperienceAPI(xAPI,alsoknownastheTinCanAPI).CommissionedbyAdvancedDistributedLearning(ADL),itwasoriginallydevelopedbyAndrewDownesandteamatRusticiSoftware.xAPIhasamuchsimpler,moreflexibleandopenstructurebasedonSubject-Verb-ObjecttripletsthataresimilartotheResourceDescriptionFramework(RDF).Topromotetheadvanceofdatainteroperabilitystandardsinthisareaofactivitymetrics,theMedbiquitousLearningExperienceWorkingGroupwasformed.ThisgroupchosexAPIastheircentralmechanismtocoordinateandconnectsuchactivitymetrics,butisalsomonitoringothersimilarprotocols.ThegrouphasbeenworkingwithkeyplayersandorganizationsrelevanttoxAPI,withacademic,governmentandindustryrepresentation.ThisreportdescribestheactivitieswithinandleadinguptoanintenseworkshopaboutxAPIandblendedlearningattheMedbiqannualconferenceinBaltimore,inMay2016.ThisreportassumessomefamiliaritywiththeprinciplesandprocessessurroundingtheuseoftheExperienceAPI.Therearemanyotherexcellentarticlesoutthereexplainingthesebasicprinciplesbetterthanwecaninthespaceavailablehere.

ThisreportalsodoesnotfollowthetraditionalIMRADlayoutofasingleresearchintervention.Itisintendedtobedescriptiveofthedesign-basedresearchapproachthatwetookinourdevelopmentofthetoolsandresourcesthatculminatedinourworkshop.SeeDiscussionformoreonthis.DevelopingxAPIProfilesTheExperienceAPIissimpleinconcept:theSubject-Verb-Objectconstructhasbeendescribedas:

Bob Did ThisBut,inorderforthexAPIstatementstobemeaningfullycomparedacrosssources,whentheyarestoredintheLearningRecordsStore(LRS),thereneedstobesomestandardizationaroundthevocabulariesused.Withinthissimplestructure,standardizationoftheVerbsused,alongwiththeirmeaningandcontextisthefirstthingtotackle.Groupsofverbs,andotherusagerequirements,thatareassociatedwithacommonsetofactivitiesarecombinedintosetsknownasProfiles.TheMedbiqLearningExperienceWorkingGrouphastakenonthetaskofdefiningaseriesofProfilestosupportmedicaleducationactivitiessuchassimulationsandscenarios.ThegroupchosetostartwiththeVirtualPatientxAPIProfile(Topps,Meiselman,&Strothers,2016),astherealreadyexistssomeexcellentbaseworkaroundtheMedbiqVirtualPatientdatastandards,andtherelatedactivitiesarerelativelywellconstrainedwithgoodcommonalityacrossplatforms.ThegroupplanstocontinuetodefineaseriesofProfiles,relatingtomannequinsandtasktrainers,standardizedpatients,scenariosandblendedsimulations,andvirtualworlds,incollaborationwithotherMedbiqWorkingGroupssuchasCompetencies,alongwithotherpartnersandinterestedorganizations.Developingastandardanddesigningtheprofilesthatdescribeitrequiresacorestructureandtheoreticaldesign,butifthisisnotgroundedinthepracticalitiesofimplementation,itiseasyforthestandardstobecomeisolatedandesoteric.Accordingly,MedbiqhasbeenveryactiveatpromotinglearningsessionsandworkshopsaroundxAPIoverthepast3years.Forthisyear’sworkshop(Meiselman,Topps,&Albersworth,2016),weassembledalearningdesignthatwouldintegrateabroadvarietyoflearningactivities,withactivitymetricsderivedfrommultipleconcurrentsources.Thisendpointandtimelinegreatlyfacilitatedamoreefficientandcollaborativeapproach.ItalsogeneratedmuchinterestinthexAPIcommunityandwewerefortunatebeneficiariesofmanycollaborativeactivitiesandproblemsolving,farexceedingourexpectations.Frankly,wewereastoundedbyhowhelpfuleverybodywasinmovingthingsforwards.ThisisdefinitelyagreatstrengthofworkingwithxAPIatpresent.

DevelopingaworkshopBasedonourexperienceswiththeHSVOProject,anditspowerincombiningmultiplesimulationmodalitiesintoeffectivelearningscenarios,wecreatedasimplescenariothatwoulddemonstrateawidevarietyofactivitymetricsfrommultipleconcurrentsources.TocontrastwiththecomplexityandcostoftheHSVOProject,wealsochosetobaseourlearningdesignonmuchsimplerandcheaperdevices.HSVOcostmorethan$2million;theaverage“disposableincome”foraneducationalresearchprojectismorelike$5-10k,sowewantedtodemonstratewhatispossibleonamuchmorerestrictedbudget.Inparticular,wewereinterestedinexploringwhatcouldbedonewiththecheapsensorsandsimplecomputingcoresavailableontheArduinoplatform,whichisdesignedforhobbyistuseandverylowbudgets.ThereareothersimilardevicessuchasRaspberryPi,whichalsohavegreatpromise,butaquickenvironmentalscanshowedArduinotobethemostsuitabletoourneeds.InourHSVOProject,wefoundtheopen-sourcevirtualpatientplatform,OpenLabyrinth,tobeaveryeffectiveintegratorofsimulationrelatedactivities.Itwasexcellentatprovidingflexible,easilymodifiable“contextualglue”toholdthevariouscomponentsofalearningscenariotogether.Tightcoupling,usingtheSavoirmiddleware,provedtooinflexibleintheHSVOProject.ThemoreopenapproachaffordedbyxAPIwasmorepromising.DevelopingaQuiverofxAPIsourcesThisentailedtheincorporationofxAPIintoOpenLabyrinth.Wewerefortunateintwoways:wehadsecuredfundingviaaCatalystGrantfromtheO’BrienInstituteofPublicHealthattheUniversityofCalgarytoextendOpenLabyrinthwithimprovedactivitymetricsandenhancedintegrationwithotherresearchplatforms;andtheinherentarchitectureofOpenLabyrinth,whichalreadycontainedausefulsetofinternalactivitymetrics,andwashighlyconducivetointegrationofxAPIstatementoutput.Wewerepleasedtoseethatthiswaspossiblewithlessdevelopmenttimethanwehadbudgetedfor,andinashortertimelinethananticipatedbecausetheexistingdatastructuresweresogood.WebasedthexAPIstatementformattingontheMedbiqxAPIVirtualPatientProfile,andalsotooktheopportunitytoreflexivelyimprovetheMedbiqprofilebyconsideringsomeofthepracticalimplicationsastheypertainedtoourcasesandscenarios.Thistwo-wayreciprocaldevelopmentwascarriedoutwithfullcollaborationwithotherworkinggroupmembersandthexAPIcommunity.AllsourcecodeisfullydocumentedonGithubathttps://github.com/olab/Open-Labyrinthandwearehappytocommunicatewithotherdevelopmentgroupsaboutsomeofthesmallchallengesthatwefacedinourimplementation.Whilewewereexploringotherwaysinwhichwecouldcaptureactivitymetricsinourscenario,usinganumberofdifferentcollectionmechanisms,wecameacrossthe

GrassBladexAPICompanion(www.nextsoftwaresolutions.com/grassblade-xapi-companion),asimpleusefulutilitythatprovidesxAPIstatementsfromWordPressactivities.SincewealreadyuseWordPressforourOpenLabyrinthsupportsiteandblog,thispresentedanidealopportunitytoextendouruseofwebplatformsforadditionaldatagatheringandactivitymonitoring.WethenimplementedtheGrassBladeLRSonourservers.WewerepleasedtonotethattheGrassBladeLRSalsousesMySQLforitsinternaldatabase,asdobothOpenLabyrinthandWordPress.Thisgreatlysimplifiedsomeoftheintegrationandtestingofdatatransferbetweenthesesoftwareapplications.Wehadbeenquiteconcernedaboutwhattypeofdatabaseinfrastructuretouse,asourinitialreadingsonthetopicofLRSsseemedtosuggestthatanoSQLdatabase,suchasMongoDB,wasdesirable(Abbey,2016)(Kaplan,2014)(Korneliusz,2014;Mei,2013).SinceourdevelopmentteamsweremuchmorefamiliarwithSQL,thispresentedasignificantpotentialbarrier.TherewasathirdfactorwhichwefoundattractiveforchoosingGrassBladeasourprimaryLRSfortesting.ThereisaflatpricingstructureperLRSinstance,withoutadditionalcostsbasedonthenumberofstatementsstored.Inourearlyphases,wehadverylittleideaofthenumberofstatementsthatwewouldbestoringorofthesizeofthedatabaseswewouldgenerate.WewerealarmedtohearofanothersimilarprojectatapreviousMedbiqconferencethatmanagedtogenerate300,000xAPIstatementswithin36hours,whichcreatedasizeablebill–somethingourfunderswerekeentoavoid.Forourinitialtesting,anotheradvantageofthisapproachbecameapparent:itwassimpletogeneratexAPIstatementsfromourWordPresssite(http://openlabyrinth.ca)andexaminetheminourGrassBladeLRS.ThisgaveusgreaterpracticalfamiliaritywiththeformattingofxAPIstatementsandsomeofthesyntacticalrulesimposedbytheLRS.WewerefortunateinthattheGrassBladeLRSisrelativelyforgivinginitsxAPIstatementinterpretation,whichaffordedgreaterexperimentationwithdatasourcesintheearlyphases.DuringourexplorationsofWordPressandxAPIstatementgeneration,wealsocameacrossH5Pwidgets(https://h5p.org).TheH5Pprojecthasbeendevelopinginteractivewidgetsthatcanbeincorporatedintoanumberofplatforms,suchasWordPress,Moodle,andDrupal.TheinitialattractionisthatmanyoftheseH5PwidgetsgeneratexAPIstatementsoftheirown.Forthis,theyrequireasimpleintermediaryframework–thereisoneavailableasaWordPressplugin.ThismeantthatwewereabletoimplementanothersourceofxAPIstatementswithlessthananhour’swork.Wethennotedthat,inadditiontobeingopen-source,H5Pmakesiteasytoincorporatetheirintermediarysupportingframeworkintoone’sownapplication.ThiswasidealforincorporatingH5PwidgetsintoOpenLabyrinthandthiswasaccomplishedwithonlyamodicumofdevelopertime,givingusyetanothersourceofxAPIstatements.TheH5Pwidgetdesigninterfaceisverysimpletoworkwith,allowingustorapidlygenerateanumberofinteractiveuserinterfaceextensions,extendingtheutilityofboththe

WordPressandOpenLabyrinthplatforms.Wewerealsodelightedtodiscoverthatwidgetscreatedforoneplatformareeasilyportabletoanother,furthermultiplyingourdevelopmentefforts.DevelopingSensorHardwareFromthesemultipleexperimentswithgeneratingxAPIstatementsfromallthesesources,wewerethenmoreeasilyabletovisualizehowonemightgeneraterawxAPIstatements,usingsomebiometricsensorsandtheverysimpleArduinohardwareplatform.Initially,thiswasonlyintendedtobeaproofofconceptdemonstrationofthefeasibilityofgeneratingsuchxAPIstatementsandthesimplicityoftheirformatting.TheArduinoplatformisrenownedforbeingcheapandeasytoworkwith,andalreadyhasawidevarietyofcheapsensorsthatarelargelyplugandplay.Inaccordancewiththeoveralllearningdesignfortheworkshop,wherewehopedtodemonstratetheabilitytomeasureabroadvarietyofconcurrentactivities,weselectedsensorsthatmightgiveussomecrudeindicationofstresslevelsintheparticipants.Weanticipatedthatthrowingsuchaplethoraofactivitiesintothemixmightbecomequitechaotic(makesforafunworkshop!)andhardtofollow.

Figure1:showingtheArduinosensorsinuse(lefthand)whilethesubjectplayedanOpenLabyrinthvirtualpatient.Rightscreenshowssensoroutputinreal-time

Forthebiometricsensors,wechoseacombinationofheartrateandgalvanicskinresponse(GSR).Thesearewellknowntobeassociatedwithstresslevelsandare,indeed,twooftheparametersusedinaliedetector.Whatsurprisedusinourinitialtestingwasthesensitivityoftheseindicators.

Withaconveniencegroupofco-investigatorsintheproject,weexaminedhowtheheartrateandGSRvariedastheynavigatedachallengingOpenLabyrinthvirtualpatientcase.Forthecases,wemodifiedtwoexistingVPcases,Gail’sDilemma(http://demo.openlabyrinth.ca/renderLabyrinth/index/727) andRushingRoulette(http://demo.openlabyrinth.ca/renderLabyrinth/index/723).TherewerealreadysometimepressuresandintensityinherentinthesecasesbutwedecidedtouptheantebyusinganewfunctioninOpenLabyrinth,withtimer-enforcedpopupwarningmessagesandnavigationaljumpspropellingthemonwardsatanever-increasingpace.Thefirstcasegenerallylastsabout15minutes,whereasthesecondcasesetof20mini-casescanusuallybecompletedwithin5-8minutes.Thisgaveussufficienttimetoobservechangesbutwasshortenoughtobeabletorunmultipleiterationswithoutparticipantburnout.Ourgroupconsistedofthreeexperiencedphysicianteachers,apublichealthprofessionalwithextensivevirtualpatientexperience,andacomputersciencestudentfamiliarwiththeplatformperformance.Wevideotapedthesessionforlateranalysis,whilewerecordedtheirperformanceontheVPcases.Wealsokeptindependentfieldnotesduringthesessionforlatertriangulationofobservations.WeconnectedeachparticipanttotheheartrateandGSRsensorsthenwatchedthechangeswhiletheyeachnavigatedtheVPcasesinturn.Itwasfascinatingtonotehowthedifferentparticipantsrespondedtothestressofthecases.Asexpected,withtheirvaryingbackgrounds,eachfounddifferentelementsoftheVPcasestobechallengingindifferentways.Allparticipantsfoundthattheincreasinglytighttimingwasstressfulorannoying.Normally,thetimeintervalallowedfordecisionsonthemini-casesis30seconds.Forthisseries,westartedat25seconds,subtractedasecondforeachsubsequentcase,leadingtoonly6secondsforthefinalcase,whichisbarelyenoughtimetoreadit,letalonemakeaninformeddecision.Otherfactorsinthecaseplaywereapparentlystressfultoeachparticipantindifferentways.Somedislikedhavingtomakedecisionsbasedonincompletedata;somefoundthatinterfacequirksthatarosewerequiteannoying.Butwewereeasilyabletodetectthesechangesinstresslevel,evenwhenquitesubtle.Oneparticipant,wellknownforcoolperformanceunderpressure,maintainedaremarkablyconsistentsetofparameters,untilthetime-outcrossedthepointofreasonablereadabilityforthecase.Assoonasaforcedjumpkickedin,therewaschangeinbothHRandGSR.Twooftheparticipantsinthisgroupwereonsignificantdosesofbetablockersforanunrelatedmedicalcondition.Weanticipatedthatthismightbluntthestressresponseandlimittheusefulnessofthesensors.Whilebothofthemhadverylowrestingheartrates(58and62beatsperminute(BPM)respectively),thesensorswerestillabletoeasilydetectachangeintheirstresslevelsastheychallengedthecases.Weweredelightedbytheseearlyfindings,whichweremuchmoresensitivethanwesuspected.Despitethemildbutintentionalstressorsinducedinthisscenario,allparticipantsreportedthatthecaseswereengagingandtheexperiencewasenjoyable.Theywereintriguedtoseecleardemonstrationthattherewasanapparentlytightassociationbetweentheirmildstresslevelsandsensorfindings.

DevelopingthexAPISensorInterfaceAllthestepsuptothispointhadbeenfairlyeasytoimplementandwerestartingtoshowaninterestingconfluenceoffactorsanddata.WeanticipatedthatcompletingtheinterfacebetweensensortrackingandtheLRSwouldalsobequitesimple.Weenlistedtheassistanceofcomputersciencestudent(CA)incodingtheArduinosensorinterfaceandxAPIintegration.Thiswastobepartofaself-learningproject,withrecognitionforcoursecredits.UsingtheProcessingIntegratedDevelopmentEnvironment(PIDE),theinitialstepswiththesensorandtheArduinoenginewerequitestraightforward.UsingtheArduinotosendoutvaluesviatheserialporttotheprocessingPIDEwithacharacterinfrontoftheValuesothatPIDEknewwhatsensorthevaluewascomingfrom.PIDEwouldthendisplaythiswithavisualizationthatwouldshowtheheartwaveformandBPMvalues.DuetothelimitationsofPIDE,itwasunabletosendaproperlyformattedHTTPpostrequesttotheLRSwithaJSONstatement.WehadtoconvertthePIDEcodetothemorepowerfulEclipseIDE.VariousadditionaltoolswereneededtogetthePIDEcodeupandrunningontheEclipseIDE,includingimportingalloftheproperlibrariesfromPIDE.

Figure2:PIDEoutputwindowshowingheartrateandGSRoutputdatafromtheArduinosensors

NowthattheprogramwasfunctioninginEclipse,wewereabletousetheexistingxAPIlibrariestosetupaclientthatwouldsendxAPIstatementstotheLRSatanyintervalthatwechose(inthiscaseeveryfiveseconds).Thesevalueswouldthendisplaysidebysidewithwhatthelearnerwasdoinginthetestcase.{ verb : {

id : http://adlnet.gov/expapi/verbs/imported, display : {

en-US : imported }

}, actor : {

name : medbiq,

mbox : mailto:info@openlabyrinth.ca, objectType : Agent

}, object : {

id : http://demo.openlabyrinth.ca, definition : {

interactionType : choice, choices : [

{ id : http://demo.openlabyrinth.ca, description : {

GSR:3219 : BPM:153 }

} ]

} }, id : a7eb9501-d0ae-4cf9-adbb-a912439f7727, stored : 2016-05-17T18:29:36.140Z, timestamp : 2016-05-17T18:29:36.140Z, authority : {

account : { homePage : http://openlabyrinth.ca/grassblade-lrs/xAPI/, name : 14-1e2c639ccc936c7

}, objectType : Agent

}}Figure3:examplexAPIstatementgeneratedbyourArduinodevice

PleasenotethatFigure3isforillustrativepurposes.Itislikelytherewillbefurtherchangestotheverbsusedbythedevice,andthattheinteractionTypewillbechangedfrom‘choice’to‘device’.Atpresent,theProfilesinthisareaarestillevolving.WehadafewsmallproblemswithxAPIstatementformattingbutwithhelpfrommembersofthexAPIcommunity,wewereabletosuccessfullylinkourGrassBladeLRS.Inparticular,weowemanythanksandwishtoacknowledgetheextensivehelpwereceivedfromPankajAgrawalatGrassBlade,AndrewDownesatRustici,andCoreyWirunatCardinalCreek.

TheformattingofourxAPIstatementsfromadevicestillneedstoberefined.Buttheimportantaspectherewasthattheoutputwasstillusefulandusableinitscurrenttemporaryformat.OtherpotentialxAPIsourcesAnumberofothersourcesofactivitytrackingwereconsidered,exploredandpartiallyincorporatedintotheworkshopscenariodesign.Forexample,weinitiallyexploredthepotentialuseoftheTobiirangeofeye-trackingsensors.Tobiimakesaverysophisticatedrangeofsensorsforresearchpurposesbutmostofthesearequiteexpensive.Oneofthelearningdesignparametersthatwechoseforthisworkshopwasaccessibilityandaffordability.WehadhopedtousethesimpleroutputsfromtheTobiiEyeXController(www.tobii.com/xperience),whichismuchcheaper.However,thecompanywasoverwhelmedwithdemandforthislevelofsensorandwasnotabletoassistuswiththisintegration.Thismaybeworthexploringinfuture.Wealsonotedthatitisnowquitepossibletoblendtheactionsandeffectsofanumberofweb-basedapplications,usingsoftwarelikeZapier(https://zapier.com)andIfThisThenThat(IFTTT--https://ifttt.com).WewereabletocreatesomeinterestingintegrationsbetweenWordPress,Instagram,Evernote,Twitterandthesmartphonecameraitself,inwaysthatcanfurthergeneratexAPIstatementstobestoredinourGrassBladeLRS.Whileintegrationsareverysimpletoimplement,inthefinalsetupoftheworkshop,wedidnotrelymuchontheseintegrations.Theydidprovidesomeoftheworkshopparticipantswithadditionalmeansofcontributingandactivelytracking,withtheuseofxAPIstatements,thecomplexactivityflowsofthisverydynamicsession.WewerecaughtoutbyadiscrepancyinthebusinessmodelofZapier.Fordemonstrationpurposes,weintendedtosetupthefivefreeZapierinteractionsthatareadvertised.However,wefoundthatZapier’sbillingstructuresrequiredpaymentassoonaswesetupthesecondinteractionchannel.Wedidnothavetimetoexplorethisbillinginconsistencyanddidnotconsiderthatthevalueaddedwasworthitatthispoint,sowesimplycontinuedwiththefreeIFTTTservice.

Figure4:simpledataflowdiagramshowinghowxAPIstatementswereusedbetweendifferentapplications

DeployingtheWorkshopAllofthesemultipleactivitiesanddatasourcescametogetherJustInTime.WeareparticularlyindebtedtoCAforanextraordinaryeffortinbringingtogetherthepiecesneededformeaningfulxAPIstatementsfromourArduino-basedsensors.HewasabletocoordinateandcollatemultiplestreamsofadvicefrommanyinthexAPIcommunity.Sometimesthereisnotalotofsleeponthebleedingedge.FortheMedbiqworkshop,wewereabletodemonstratealiveworkingexampleoftrackingactivitydatafromthefollowingresources:

1. Heartrate,viaArduinosensor2. GSR,fromArduinosensor3. MouseclicktimingsfromOpenLabyrinth4. DecisiontreeresponsesfromOpenLabyrinth5. QuestionresponsesfromOpenLabyrinth6. ForcednavigationjumpsandtimeoutsfromOpenLabyrinth7. H5PwidgetinputfromOpenLabyrinth8. H5PwidgetinputfromWordPress9. EvernotenotesviaWordPress10. IFTTTDOCameraimagesviaWordPress

Inthespiritofengagingasmanyworkshopparticipantsaspossible,aswellastrackingactivityandstresssensorsonourvolunteers,wewerealsoabletoengageactive

contributionsandlearningactivitiesfromparallelparticipantsastheyusedtheadditionaldatasourcestocontributetotheoveralldatastream.Theworkshopwassuccessfulinaddressingawidevarietyofparticipantneeds.WehadamixofthosewhowereprimarilyinterestedinthetechnicalaspectsofxAPI;thosewhowereinterestedinexploringwhatcouldbedonewithsuchmetrics;andthosewhowereinterestedinexploringtheenhancedvarietyoflearningdesignsthatcouldbesupportedbysuchamulti-modalityapproach.Thereappearedtobeaveryhighdegreeofengagementinalllevelsofthevariousactivities,whichsometimesmadeitdifficulttokeepeveryoneontrack.Whentimewasup,theroomwasslowtoclearbecauseofthecontinuinglevelofveryactivediscussion.Notethatthiswasoneofthefinalsessionsoftheconference,whenmostpeoplearestartingtoflag.AnalyzingtheLRSDataAssoonaswestartedcollectingxAPIstatementsinourGrassBladeLRS,wewereabletoperformsomebasicanalysisusingitsbuilt-intools.

Figure5:exampledashboardfromGrassBladeLRS

Aswenotedearlier,GrassBladeisveryforgivinginitsparsingofxAPIstatements.Thismakesthingsmucheasiertosetup,ratherthanjustsloggingthroughalongseriesoferrorstatements,whichweverymuchappreciated.WealsogreatlyappreciatedtheaccessibilityofGrassBlade’screator,PankajAgrawal,whowasveryhelpfulintroubleshootingthroughoutallphasesoftheproject.WewerealsoveryfortunatetoreceivemuchhelpandadvicefromRusticiSoftware.TheyprovideduswithfreeaccesstosandboxaccountsonboththeirSCORMCloud(http://scorm.com/scorm-solved/scorm-cloud-features)andWatershedLRSs

(www.watershedlrs.com).SCORMCloudhasanumberoftestingfunctionswhichmakeiteasierfordeveloperstofinetunethesyntaxoftheirxAPIstatements,includingamanualxAPIGenerator(https://tincanapi.com/statement-generator).

Figure6:exampledashboardfromWatershedLRS

TheWatershedLRSisamuchmorepowerfulanimal,withserioushorsepowerandpowerfulvisualizations.Thiswaslargelyoverkillforthesimpleaspectsofthisprojectandwellbeyondthebudgetavailable.However,AndrewDownesfromRustici,andPankaj,wereextremelyhelpfulinfinetuningtheinterfacesintheirrespectiveLRSssothatwecouldexploredirectdataintegrationbetweenLearningRecordStores.ThisgaveusaccesstosomedatavisualizationsavailableinWatershed,bydirectlypullingxAPIstatementsfromtheGrassBladeLRS,andallowedustoexplorethepowerofbigleagueLRS,whilestillusingthesimpleflexibilityofGrassBladeforourinitialdatacapture.WecannotapplaudenoughthecollaborationevidentinthexAPIcommunityinsurmountingthechallengesthatwefoundintakingthismulti-layeredapproach.Webrieflyexploredthehigherlevelsofassessmentbeyondactivitymetrics,basedonxAPIstatements,thatmightbeaffordedbyMozillaOpenBadges(http://openbadges.org).TheWatershedLRSdoesprovidesomebasicfunctionalitythatsupportsintegrationofBadges.However,onfurtherdiscussionwiththeRusticiteam,itwasclearthatfewinmedicaleducationareinterestedinusingMozillaBadgesatthispointandthereforeweredirectedourefforts.

WewerealsofortunatetohaveaccesstoafreetrialaccountontheSaltboxWaxLRS(www.saltbox.com).WeinitiallythoughtthatitwouldjustbeinterestingtosimultaneouslysendxAPIstatementsto3differentLRSsfromOpenLabyrinth.ThiswassurprisinglyeasytosetupanddidnotexacttoomuchofaperformancehitonourOpenLabyrinthserver.Ithassincebeenpointedoutthatthisisasomewhatredundantapproach.ItisinthenatureofLRSstobedirectlyfederated.SoitwouldbemoreeffectivetouseLRStriggersandfilterstosendselectedstatementsfromoneLRSontoothers,ratherthantoexpectthesourcesoftwaretohandlestatementsendingtoall3LRSs.ThetestingwiththeWaxLRShadanadditionalbenefit:itismuchmorestrictinitsparsingofxAPIstatements.Aftersomeinitialgrumblingonourpart,wequicklyrealizedthebenefitsofthis:itenabledourdeveloperteamstobemuchmoreexactintheirxAPIstatementparsing,whichresultedinamuchcleanersetofxAPIstatementsfromOpenLabyrinthandourH5Pwidgets.WethanktheSaltboxteamfortheirassistanceinthis.OpenLabyrinthitselfhasquitesophisticatedinternaltrackingandreportingofactivitymetrics,alongwithsomesimplevisualizations.

Figure7:pathwayanalysisreportgeneratedinternallybyOpenLabyrinth

Indeed,theOpenLabyrinthdeveloperteamwasinitiallyperplexedastowhywewereintroducingtheadditionalcomplexityofxAPIstatementreportingsince,atthatstage,wewereabletoachievethesamelevelofactivityreportingusingitsownfunctions.However,sinceweimplementedxAPIreportingdirectlyintoOpenLabyrinth,severaladvantageshavebecomeclear.

Firstly,ithasaffordedsimultaneousactivitytrackingacrossamuchwiderrangeoflearningactivities,notjustOpenLabyrinth,asdescribedintheparagraphsabove.Secondly,ithasrelievedourdeveloperteamofthecontinuingburdenofcreatingcustomreportsforvariousresearchgroups.Thisisthebaneofmanydatabasedresearchtoolsandcanbecomeahugedrainonresources.Butthirdly,andmostimportantly,ithasgivenusaccesstoamuchwiderrangeofdatavisualizationtools.WiththeintegrationofxAPI,wehavenowmovedintotheBigDataworldinlearninganalytics(Topps,Meiselman,Ellaway,&Downes,2016).Westatethis,basednotupontheVolumesofdatawearegenerating(althoughthesearerapidlyincreasing),butmoreupontheotherVsthatarecommonintheprinciplesofBigDataanalytics:Velocity,Variety,VeracityandVisualization(Kobielus,2013).Byplacingouractivitymetricsinacommonformat,withinanLRSthatisdesignedtoacceptlargevolumesofdatafrommultiplesourcesandfederatethemwithotherservices,wecannowtakeadvantageofsomeofthemorepowerfulvisualizationandanalytictoolsbeyondtheworldofmedicaleducation.ItalsoallowsustoexplorebothpowerfulcommercialvisualizationtoolslikeTableau(www.tableau.com),whichisdesignedforusewithverylargedatasets,andalsocheapbutversatilevisualizationlibrarieslikeD3.js(https://d3js.org),anopen-source,componentizedvisualizationlibrary.

DiscussionForthisTechnicalReport,wehavenotadheredtotherigidIMRADstructurecommontomostacademicarticles,inthebeliefthathighlightingsomeofthedesignissuesthataroseduringthedesign-basedresearchelementsoftheprojectwasmoreconsistentwithourdiscoveryorientedapproach.Hence,severalofthediscussionpointshavealreadybeencoveredearlier.Wewouldhoweverliketofurtherhighlightsomeimportantpointsthatarosefromtheprojectasawhole.ItwasveryclearthattheloweredcostofintegratingdifferentsystemsusingasingleAPIthattrackseventsandmetadataisabigadvantageofusingxAPI.WefoundthatxAPImakesitpossibletoadaptlowcost,opensourcetoolsforsensorsandlearninginteractions,andwecannowtrackdatathatmayilluminatethelearningprocessfarbeyondwhatwaspossiblewithSCORM.Whatwedidinmeasuringstressparametersinlearners,whileengagedinaneducationalexercisewasnotparticularlynew(Hardy,Mullen,&Jones,1996)(Aroraetal.,2010).Butpreviouseffortshaverequiredasophisticatedsimulationlaboratorywithexpensivesuitesofsensorequipment,whicharelargelybeyondthebudgetofmosteducationresearchers.Otherattemptshaveusedsubjectivemeasuresofperceivedstress(Cohen,Kamarck,&Mermelstein,1983)which,whilestillrelevant,havetheirownbiasesandlackofsensitivity.Ourapproach,usingsimplesensorsandcheapcomputingdevices,alongwithaflexibleapproachforgatheringactivitymetricsfrommultiplesourcesrepresentsaninterestingadvanceinthisarea.

ConclusionsDatafrommultiplesourcesgreatlyextendsourabilitytotrackwhatlearnersactuallydo,notwhattheysaytheydoorwhattheirteacherssaytheydo,thuspotentiallyreducingmanysourcesofbias.LooseaggregationofactivitiesviaxAPIismorepracticalthanthetightcouplingpreviouslyenvisionedbySCORMasatranslationmechanism.Bylooselycouplingthedataflows,weexpectthattherewillbegreaterflexibilityinlearningdesignsandlessdependencyonkeepingstructuralchangeswithincommunicatingsystemssuchasLMSs,LRSs,virtualpatients,allalignedwitheachother.CheapcomputingdeviceslikeArduinoimproveresearcheraccesstosensordatathroughgreateraccesstoabroaddevelopercommunitywithanopen-sourceattitudetodesignrecipes;reducedcostofhardwareandsoftwaredevelopment;andaneverexpandingrangeofsensormodalities.

BibliographyAbbey,D.(2016).LearningLockeronGithub.RetrievedJune18,2016,from

https://github.com/LearningLocker/learninglocker/releasesArora,S.,Sevdalis,N.,Nestel,D.,Woloshynowych,M.,Darzi,A.,&Kneebone,R.(2010).The

impactofstressonsurgicalperformance:asystematicreviewoftheliterature.Surgery,147(3),318–30,330.e1–6.http://doi.org/10.1016/j.surg.2009.10.007

Cohen,S.,Kamarck,T.,&Mermelstein,R.(1983).AGlobalMeasureofPerceivedStress.JournalofHealthandSocialBehavior,24(4),385.http://doi.org/10.2307/2136404

Hardy,L.,Mullen,R.,&Jones,G.(1996).Knowledgeandconsciouscontrolofmotoractionsunderstress.BritishJournalofPsychology,87(4),621–636.http://doi.org/10.1111/j.2044-8295.1996.tb02612.x

Kaplan,M.(2014).WhentoUseMongoDBRatherthanMySQL(orOtherRDBMS).RetrievedJune18,2016,fromhttps://dzone.com/articles/when-use-mongodb-rather-mysql

Kobielus,J.(2013).TheFourV’sofBigData.RetrievedJune18,2016,fromhttp://www.ibmbigdatahub.com/infographic/four-vs-big-data

Korneliusz.(2014).MongoDBvsMySQL:ComparisonBetweenRDBMSandDocumentOrientedDatabase.RetrievedJune18,2016,fromhttps://blog.udemy.com/mongodb-vs-mysql/

Lalley,J.,&Miller,R.(2007).THELEARNINGPYRAMID:DOESITPOINTTEACHERSINTHERIGHTDIRECTION?Education,128(1),64–79.

Mei,S.(2013).WhyYouShouldNeverUseMongoDB.RetrievedJune18,2016,fromhttp://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb

Meiselman,E.,Topps,D.,&Albersworth,C.(2016).MedbiqxAPIworkshop.InV.Strothers&P.Greene(Eds.),MedbiqAnnualConference.Baltimore,MD.Retrievedfrom

http://www.slideshare.net/topps/medbiq-xapi-workshop2bMiller,G.E.(1990).Theassessmentofclinicalskills/competence/performance.Academic

Medicine :JournaloftheAssociationofAmericanMedicalColleges,65(9Suppl),S63–7.Retrievedfromhttp://www.ncbi.nlm.nih.gov/pubmed/2400509

Topps,D.,Meiselman,E.,Ellaway,R.,&Downes,A.(2016).AggregatingAmbient_StudentTrackingDataforAssessment.InOttawaConference.Perth,WA:AMEE.Retrievedfromhttp://www.slideshare.net/topps/aggregating-ambient-student-tracking-data-for-assessment

Topps,D.,Meiselman,E.,&Strothers,V.(2016).MedbiqxAPIVirtualPatientProfile.Baltimore.Retrievedfromhttp://groups.medbiq.org/medbiq/display/XIG/Profile%3A+1.+Virtual+Patients