towards the discovery of learner metacognition from reflective … · reflective writing, an...

(2016).Towardsthediscoveryoflearnermetacognitionfromreflectivewriting.JournalofLearningAnalytics,3(2),22–36.http://dx.doi.org/10.18608/jla.2016.32.3

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 22

Towards the Discovery of Learner Metacognition from Reflective Writing

AndrewGibsonQueenslandUniversityofTechnology,AustraliaUniversityofTechnologySydney,Australia

[email protected]

KirstyKittoQueenslandUniversityofTechnology,Australia

PeterBruzaQueenslandUniversityofTechnology,Australia

ABSTRACT:Modernsocietydemandsrenewedattentiononthecompetencies requiredtobestequipstudentsforadynamicanduncertainfuture.Wepresentexploratoryworkbasedonthepremisethatmetacognitiveandreflectivecompetenciesareessentialforthistask.Bringingtheconcepts of metacognition and reflection together into a conceptual model within which weconceived of them as both a set of similar features, and as a spectrum ranging from theunconsciousinner-selfthroughtotheconscious,external,socialself.Thismodelwasusedtoguideexploratory computational analysis of 6,090 instances of reflective writing authored byundergraduatestudents.Wefoundtheconceptualmodelusefulininformingthecomputationalanalysis,whichinturnshowedpotentialforautomatingthediscoveryofmetacognitiveactivityinreflectivewriting,anapproachthatholdspromiseforthegenerationofformativefeedbackforstudentsastheyworktowardsdevelopingcore21stcenturycompetencies.

Keywords:Metacognition,reflection,reflectivewritinganalytics,computationalanalysis,naturallanguageprocessing,learninganalytics,21stcenturycompetencies

1 INTRODUCTION

Continuousrapidchangeisadominantfeatureofmodernsociety.Technologyisrapidlyevolving,andthisisleadingtothedemiseofentireemploymentsectors,andthecreationofcompletelynewones.Withinthiscontext,wecanhardlyanticipatewhatskills,knowledge,andexperiencethestudentsofourformaleducation systems are going to need to know upon graduation, let alone ten years into the future.Learninghowtoacquirenewideas,skillsandapproachesforthemselvesisanimportantstrategythatwillhelpourstudentstosucceedinthisageofuncertainty.Learningtolearn,andtherelatedconceptsofself-directedlearning,self-regulatedlearning,andindependentlearning,arecertainlynotrecentpedagogicalideas(Herber&NelsonHerber,1987;Paul,1990),butthegeneralapproachhas,overtime,gainedgreaterprominence(Cornford,2002;Takanishi,2015;Wilson,2015)aswehavewitnessedsignificantchangesin



variousdomainsofknowledge(CEDA,2015).Whilechangesinthebodyofknowledgehavealwaysbeenwithus, itseemsthattheyhaveneverbeenasrapidastheyarenow,andthiscurrentaccelerationofchangeappearstobe increasingratherthanabating.Wecannotescapethiscontext,andwithin itwemust aim to educate our students in away that prepares them for the future, equipping themwithadaptivecapabilitiesthatcanflourishinthisfluidenvironment.

Historically,formaleducationhasfocusedonknowledgeoffacts,andwhilefoundationalcontentisstillseen as critical, entire domains are starting to recognize the need to shift educational focus frompresentinglargeamountsofcontent,towardsinstillingakindofmeta-knowledge;thatis,thecapabilityforunderstandingwhatknowledgemaybeneededforasituationandwhentoapply it,aswellastheabilitytofindtheidentifiedrequisiteknowledge.Developmentofautonomyinthisrespectisnotpossiblefor the learnerwithout themdrawinguponmetacognitiveabilities toevaluate their current cognitiveposition, and formulate strategies for advancing it. Despite the changes in formal education toaccommodatethisparadigmshift,furtherworkisrequired,andintelligenttoolsareneededtosupportit.

Thismoderneducationalcontexthasprovidedanunderlyingmotivationfortheworkpresentedhere.Inthispaper,wedemonstratepotentialforcomputationalanalysisofreflectivewriting(ReflectiveWritingAnalytics) as a means of discovering evidence of metacognitive activity in the reflective writing of alearner.Asweshall see inSection2, thedistinctionbetweenmetacognitionand reflection isblurred.However,theliteraturesuggestssignificantrelationshipsbetweenthem,andthattheanalysisofwrittenreflectionmayallowforthediscoveryofmetacognitiveactivityonthepartofthelearner.Anempiricalstudywasdesignedtoexplorethisintuition.

Theresearchwasconductedintwoparts,thefirstbeingthedevelopmentofaconceptualmodeldrawnfromthemetacognitionandreflectionliterature(Section2),andthesecondbeinganexplorationofthemodel’sapplicationtothecomputationalanalysisofReflectiveWritingAnalytics(Section3).Weconcludewithanexaminationoftheimplicationsofthisstudytothefieldoflearninganalytics(Section4).

2 METACOGNITION AND REFLECTION

InthecenturysinceJohnDewey introducedtheconceptofreflection ineducation(Dewey,1916),theeducational community has increasingly considered reflection an important part of learning. This hasbecomemorefocusedsinceFlavell(1976)definedthetermmetacognition,whichencouragededucatorsto embrace the significance of thinking about thinking. In a pedagogical context, both reflection andmetacognition regularly co-occur, suggesting a strong link between them (Schön, 1983;Moon, 1999).IndeedBoud,Keogh,andWalker(1985,p.141)statethat“Reflectionisthus‘meta-thinking’”andSanders(2009,p.688)assertsthat“reflectionisametacognitiveprocess.”Thislinkholdssignificanceforourwork,sincewhilemetacognitioncanbehiddenfromview,reflectioncanbemadeexplicitinstudents’reflectivewriting.Ourintuitionisthatexploitingthislinkbetweenreflectionandmetacognitionwillprovideameansfordiscoveringevidenceofmetacognitioninthereflectivewritingofthelearner.



Inthissection,weexaminesomeofthekeyliteratureonmetacognitionandreflection,consideringeachin turn as individual concepts. We then identify key similarities and differences between them, andproposeaconceptualmodelthatencapsulatesthesefeatures.

2.1 Metacognition

John Flavell originally defined the term metacognition as “the active monitoring and consequentregulationandorchestration”of informationprocessingactivities,“usually inserviceofsomeconcretegoalorobjective”(1976,p.232).Flavell’s(1979)definitionmaturedintoamodelofcognitivemonitoringthatcomprisedmetacognitiveknowledge,metacognitiveexperiences,goals,andactions.Henotedthatthesefourcomponentscontinuallyinteractthroughoutthecognitiveprocess.Thesecomponentsprovideanindicationofhowmetacognitionmaypresentifitweremadevisibleinstudentwriting.

StudiesconductedsubsequenttoFlavell’sworkhaveidentifiedtwotypesofmetacognition:regulatorystrategies,andstrategicknowledge (Quirk,2006).Regulatory strategiesalignwith themonitoringandregulationaspectsofFlavell’sdefinition,whereasstrategicknowledgeincorporatestheorchestrationandgoalaspects.Thisapparentlydualnatureofmetacognition isalso identifiedbyothers,butnotalwaysalong thesame lines.Forexample,Amseletal. (2008)examined the relationshipofmetacognition toscientificreasoning,explicitlytakingadualprocessperspective:“Indual-processtheory,metacognitiveskillsfunctiontoregulateconflictsbetweenanalyticallyandexperientiallybasedresponses”(Amseletal.,2008, p. 454). Amsel et al. defines the experiential as heuristic and automatic, comparing it to theanalytical,whichisformalandsystematic.ThesedimensionsappeartoaccordsomewhatwithFlavell’s1979modelwithmetacognitionexperienceandknowledge;however,itseemslikelythatFlavellwouldnot takeAmseletal.’sviewofmetacognitionasamoderatingskill thatoverrides theexperiential, forAmseletal.’sdefinitionisspecifictoastyleofcognitionthatpresentsasscientificreasoning,andmaynottranslatetootherformsofcognition.Lehmann,Hähnlein,andIfenthaler(2014)alsopromotedadualviewof metacognition within a widermodel of self-regulated learning. They note thatmetacognition caninclude a structural component that is relatively stable, including task knowledge and knowledge ofcognition,andaprocesscomponentthat ismore immediatetoaparticularsituation.Thiscomponentincludesplanning,monitoring,andevaluation.WenotethesimilaritiesbetweenthesecomponentsandFlavell’sgoals,actions,andregulatorystrategies.

The view that metacognition is a trainable skill, and that people should be encouraged to be moremetacognitive,hasexisted for as longas the term itself. IndeedFlavell (1979) asserted that cognitivemonitoring is trainable,and that it shouldbepromotedmorewidely: “Iamabsolutelyconvinced thatthereis,overall,fartoolittleratherthanenoughortoomuchcognitivemonitoringinthisworld”(p.910).However,weshouldbecarefultoavoidtheconclusionthatmoreisbetter,asexcessivemetacognitionisevident inpeoplewhoarepathologicallyobsessive,andwhennon-selective, cognitivemonitoringcanbecome rumination on negative outcomes, leading to anxiety related problems (Quirk, 2006). Thissuggeststhatdiscoveryofmetacognitionshouldbemoreconcernedaboutitsqualitiesratherthanthequantityinwhichitoccurs.



Metacognitionhasoftenbeenassociatedwithintentionalandconsciouslyaccessiblethought;however,thisdoesnotrepresentacompletepicture,andignorescognitiveactivitythatoccurswithoutawareness.“Indeed,theconsciouscharacterofmetacognitionwaschallengedbyearlyresearchonmeta-memoryshowingthatpeoplemonitorandcontroltheircognitionwithoutbeingconsciouslyawarethattheydoso”(Efklides,2008,p.281).Koriat(2000)suggestedthatmetacognitionactuallypossessestwofaces:animplicit automatic mode and an explicit controlled mode. In fact, he proposes that metacognition,particularlymetacognitiveexperience,mediatesbetweentwolayersofconsciousnessstatingthat“theyserve to interface between implicit-unconscious-automatic processes on the one hand, and explicit-conscious-controlledprocessesontheother”(p.152).Whileeducationistypicallyfocusedontheexplicitcontrolledmodeofmetacognition, it istheexistenceoftheimplicitautomaticmodeofmetacognitionthatwesuggestdifferentiatesitfromtheactivityofreflection.

2.2 Reflection

Somewhat likemetacognition, reflection tends to be a concept recalcitrant to crisp definition.Moon(1999)notedthatasingledefinitionofreflectioniselusive,despiteattentiontothetopicovermanyyears.She observed that “reflection seems to be a form of mental processing with a purpose and/or ananticipatedoutcomethatisappliedtorelativelycomplicatedorunstructuredideasforwhichthereisnotanobvioussolution”(Moon,1999,p.98).Thismentalprocessingaspectappearstobesuchacommonhumanactivitythatweareindangerofmissingtheimportanceoftheconceptaltogether.Indeed,Boudetal.(1985)identifiedfamiliarityasacontributortoourlackofunderstandingaboutreflection:

Theactivityofreflectionissofamiliarthat,asteachersortrainers,weoftenoverlookitinformallearning settings, andmake assumptions about the fact that not only is it occurring, but it isoccurringeffectivelyforeveryoneinthegroup.Itiseasytoneglectasitissomethingwhichwecannotdirectlyobserveandwhichisuniquetoeachlearner.(p.8)

Reflection can also be understood differently in different contexts. In this paper, we are particularlyconcernedwithreflectionforlearningwithinaneducationalcontext.Howeveritisworthkeepinginmindthatthetermisalsocommonlyusedwithrespecttolearningwithinaprofessionalcontext(i.e.,reflectivepractice),andinapersonalsense(i.e.,personalreflection).Thesedifferentapplicationsofreflectionarenotnecessarilyindependentfromeachother,andsharemanyofthesamefeatures.Asaresult,researchdirectedatoneapplication isoftenreferredto inthecontextofanother.Forexample,Schön’s(1983)work,whichfocusedonprofessionalreflectivepractice,hasoftenbeenusedasabasisforunderstandingreflectionwithinaformaleducationcontext.

Furthercomplicatingtheconcept,reflectionasamentalactivitycanbeeasilyconflatedwithreflectivewriting—a task designed to elicitmental reflective activity, or in the case of assessment, documentmentalreflectiveactivity.Forthepurposesofthisstudy,weconsideredreflectiontobeatypeofmentalactivitythatcanbeevidencedbyreflectivewriting.



Despitetheambiguityaroundtheterm,Boudetal.(1985)identifiedthreefeaturesofreflectionthattheyclearlyobserved.First,asreflectionisapersonalactivity,thelearneriscentralandincontrol,revealingonlywhat theywish aboutwhat they have reflected upon. Second, reflection is goal orientated andpurposive.Third, reflection isacomplexprocesswithanaffectivedimensionthat interactswithothercognitiveprocesses.Thesethreeaspectsofreflectioncorrelatewellwiththeregulation,planning,andexperientialqualitiesidentifiedinmetacognition.

However,unliketheliteratureonmetacognition,thereisagrowingbodyofliteraturethathighlightstheimportanceofasocialdimensioninreflection.Indeedsomewouldarguethatreflectioncannotbefullyunderstoodwithoutexamining itwithina social context.Ryan (2011)proposesamulti-levelmodel—stronglylinkedtothecontextoftheauthor—forteachingandassessingacademicreflection.Thisworkviewsreflectionasasociallysituatedactivityinvolvingadeepexaminationofselfinrelationtothesocialcontext,andtheintentiontotakeactiononthebasisofthatexamination.

ActionisalsoakeyaspectofMezirow’s(1990)workontransformativelearning.Mezirow(1990)proposeda high level of reflection called critical reflection,which involves learners critiquing their pre-existingknowledgebasedonnewinformation,holding“biasesinabeyance,and,throughacriticalreviewoftheevidenceandarguments,make[ing]adeterminationaboutthejustifiabilityoftheexpressedideawhosemeaningiscontested”(p.10).Thisworkcertainlyhighlightedtheimportanceofreflectioninlearning,apointthatRyan(2013)extendedbeyondimprovingone’sownunderstandingtotheverysustenanceofthelearningprocess.Sheargued“thatstudentscanandshouldbetaughthowtoreflectindeep,criticalandtransformativewaystoengendersustainablelearningpractices”(p.145).Hereweseesomesimilaritybetweentheapplicationofreflectionto learningandFlavell’sassertionthatmetacognitivemonitoringandcontrolistrainable.

AlsorelatingreflectionandlearningistheworkofZimmerman(2002)whoappliedreflectionspecificallyto the process of creating the self-regulated learner. He stated that “these learners monitor theirbehaviourintermsoftheirgoalsandself-reflectontheirincreasingeffectiveness”(p.66).Hereweseein“monitoring” and “goals” that again there is alignment between conceptions of reflection andmetacognitionfurthersupportingourintuitionthatthetwoarestronglyrelated.

Zimmerman(2002)conceptualizedself-reflectionintermsofself-judgmentandself-reaction,withself-judgment involvingthecomparisonofthe learners’ownunderstandingoftheirperformanceagainstastandard;andself-reactioninvolvinganaffectiveelement(e.g.,self-satisfaction)andanadaptiveelement(e.g.,adjustmentofstrategy).Thistoodisplayssimilaritieswithmetacognition,withself-judgmentandself-reactionrelatingnicelytothejudgmentandaffectivedimensionsofmetacognitiveexperience.

In relating the various understandings of reflection evident in the above literature, we can see thatreflectionisusuallyconceivedofasadeliberateprocesswithatriggerofsomesortandanunderlyinggoalorobjective.Ittendstobepersonalinnature,engagingfeelingsandemotions;however,itisalsosociallysituatedandthereforenotsolelyaninternalactivity.Italsotendstoincludealevelofmonitoringofthe



particularsituation inquestion,andregulationofbehaviourtowardsachievementof thedesiredgoal.Thereisasensethattruereflectionhasdepthandisadifficultprocess,butonethatcanbetransformative.

2.3 A Model for Metacognition and Reflection

Wehavealreadyidentifiedsomesignificantsimilaritiesbetweenfeaturesofmetacognitionandreflection.Thesesimilarfeaturesprovideafoundationforamodel,butwemustremainconsciousofthedifferencesbetweenthesetwoconcepts.Themostnotabledifferenceisthelackofanimplicitautomatedversionofreflection,andtheabsenceofasociallysituatedtypeofmetacognition.Wefoundsomeresolutiontothese differences in Efklides’ (2008) multilevel model of metacognition. She proposed a model thatcomprisesthreelevels:anon-consciousobjectlevel,themetalevelassociatedwithpersonalawareness,andthemeta-metalevelassociatedwithsocialawareness.Significantly,thisapproachnotonlyrelatesthe non-conscious and conscious aspects ofmetacognition, but also introduces the idea of reflectiontogetherwithmetacognitionbetweenthepersonalawarenessandsociallevels.

Figure1:Aspectrumviewofmetacognitionandreflection.

We conceptualized this difference in the relationship between metacognition and reflection as aspectrum,withtheinternalinner-selfononeendandtheexternalsocial-selfontheother(Figure1).Formetacognition, the leftsideof thespectrumrepresents the implicit,automated,non-consciousmode,whilethecentreincludestheexplicit,conscious,controlledmode(Koriat,2000).Forreflection,therightsideof thespectrumrepresents theexternal, socially situateddimension,and thecentre includes thepersonal,internal,butconsciousaspect.Wesuggestthatthisunderstandingoftherelationshipbetweenmetacognition and reflection not only caters to both implicit–internal and explicit–externalunderstandingsoftheseactivities,butalsoprovidesawayofrelatingtheexplicitmetacognitiontointernalreflectioninthecentreofthespectrum.Inthisarea,thetermscouldbeusedinterchangeably,whileontheleftandrightextremesthetermshavequitedifferentmeanings.Weproposethismodelasawayofembracingthevaryingdefinitionsofmetacognitionandreflectionfoundintheliteraturewhileprovidingadditionalclarityontheuseofeachterm.

However,thespectrumviewprovidedinFigure1doesnotclearlyspecifythewaysinwhichmetacognitionand reflection are alike. To address this, we modelled similar features as interrelated components



(Figure2).Reducingawholeideaintocomponentpartscanbeproblematic,particularlywhentryingtounderstandhowthecomponentsrelatetothemoregeneralspectrummodel.Thecommonfeaturesofmetacognition and reflection do not explain the changing nature of the two activities as viewed ondifferentpartsoftheproposedspectruminFigure1,andwhilethespectrumhighlightsthegeneralnatureofbothactivities,itdoesnotidentifythecommonfeatures.Soourtwoviewsoftherelationshipbetweenmetacognitionandreflectionneedtobetakentogetherandunderstoodastwoperspectivesoftheonephenomena.Selectingasuitablelevelofgranularityforthecomponentmodelwasimportant.Toofewcomponents resulted in themodel lacking explanatory power, and toomany components resulted insomefeaturesappearingfalselysignificant.

Figure2:Acommonfeatureviewofmetacognitionandreflection.

Ourfinalcomponentmodel(Figure2) istheresultofanumberof iterations,withthemostsignificantchangebeingtheintegrationofthetriggerandgoalcomponentsintothemaincomponentofregulationtogetherwithmonitor and control.Weattempted to address the granularity issueby retaining theseelementsassubcomponentstobemappedintheanalysis,butconsideredthemtogetherwithmonitoringandcontrolasonelevelofimportancewithrespecttotheoverallmodel.

Themodelisdesignedwiththreeprimarycomponents,eachlabelledaccordingtothelanguageofthemetacognitionliterature.Themodel’scorecomponentisregulation,whichatitscentreinvolvesamonitorand control loop. Because of the interrelated nature of monitoring and controlling, we tended toconceptualizetheseasoneloop,ratherthantwoindependentcomponents.Wenotethattheregulationcomponentalsocontains initiationandobjective sub-components, respectively labelledas triggerandgoal.Regulationiscontinuallyinteractingwiththeothertwocomponents:Knowledge,whichrepresentsmetacognitive knowledge (the storage of strategies anddecision information), and Experience,whichrepresentsmetacognitiveexperience(feelingsandaffectivecontribution).



Table1:Modelsub-componentsdescribedintermsofmetacognitionandreflection.

Wenotethatthemodelasvisualizedcouldinfercausalrelationshipsbetweenvariouscomponentsandthatthereisthereforeasenseinwhichatemporaldimensiontothemodelcanbeinferred.Thiswasnotourintentionasthecomponentsarecontinuallyinteractingwitheachotherthroughoutmetacognitiveandreflectiveactivity.Also,asweadoptedmetacognition inspiredlabelsforthecomponentview,therelationshipbetween themodeland reflectioncanbe lessobvious.Toassistwith these links,Table1drawsattentiontotheapplicabilityofthemodelforbothreflectionandmetacognition.

2.4 The Use of Reflective Writing

Anessential point is that the proposedmodel centres on the cognition of the learner, and thereforecannot be analyzed directly. An interfacemust be provided between this conceptualmodel and anylearninganalyticsthatwemighthopetoperform.Inthisstudy,wehaveusedreflectivewritingasthisinterface. Reflectivewriting is an existing, well-accepted learning activity, enabling this type of studywithout imposing a non-related task on the learning process.We acknowledge, of course, that othermeansofanalyzingmetacognitionexist(Koriat,2000).

Within the modern educational context introduced in Section 1, the requirement to move towardsteaching skills required for lifelong learning has resulted in increasing attention on reflectivewriting.Reflectivewritinghasthepotentialtodevelopmetacognitiveandreflectivecapabilitiesintheirownright(Reidsema,Goldsmith,&Mort,2010),andsohereweattemptedtoextractanalyticsspecificallydirected

Metacognition Reflection

Trigger(Regulation)

Aconsciousorunconsciouscognitiveevent,particularlyaproblemorincongruence.

Aconsciouscognitiveeventoranexternaleventthatneedsimprovementormodification.

Monitor(Regulation)

Monitoringofcognitiveprocesses,bothconsciouslyandunconsciously.

Consciousthinkingaboutmentalprocessesorexternalbehaviours.

Control(Regulation)

Utilizingpre-learnedstrategiestocontrolcognitiveprocesses.

Takingactiontomodifymentalprocesses,orexternalbehavioursbasedoninputinformation.

Goal(Regulation)

Resolutionofthetriggerproblemordissonance.

Successfulmodificationorimprovementoftriggerevent.

Knowledge Memorydedicatedtostoringmetacognitiveknowledgeinparticularstrategiesandtheirefficacy.Usedinthemonitor–controlloop.

Eitherinternalmemoryorexternalrecordingofthoughtsthatcanbeusedasnecessary.

Experience Affectiveimpactonmonitor–controlloop.Assistswithstrategyformationandevaluation.

Emotionalcontributiontomonitoringandcontrol.Assistswithestablishingpersonalvalueandsignificance.



towardstheanalysisofmetacognitiveactivityasitarisesinreflectivewriting.Although,thismightappeara reasonable approach, wemust be careful when adopting it within a formal education setting. Therequirement to measure what has been learned in order to justify the importance of the learningexperiencecanhaveasignificantnegativeimpactonreflectivewriting.“Assessmentinvolvespresentingone’sbestwork,whereas reflection involvesuncertainquestioning, self-criticism,exploring, tryingoutideasandacknowledgingthemessynatureofreality”(Wharton,2012,p.490).Furthercomplicatingtheassessmentofreflectivewriting,Reidsemaetal.(2010)notethatitallowsstudentsto“engagewiththeirbeliefs,values,uncertainties,desiresandquestionsandtoclarifywhattheyknowandmoreimportantly,do not know about a situation” (p. 10). This can place reflective writing tasks in direct conflict withsummativeassessment,resultinginaconflictofparadigmsbetweenaninstitutionalneedforapositivist,product-driven perspective, as opposed to the learner’s needs for a constructivist process-drivenapproach(Ross,2011).Thisisoftenseenwhenthereisatendencytofocus“onwhathasbeenlearnedrather thanonhow ithasbeen learned,and theemphasis ison improving the reflectivewritingstyleratherthanonlearningaboutlearning(metacognition)”(Mair,2012,p.150).

Our approach avoids these issues by focusing on the discovery of metacognitive activity through alearner’swritingwithout requiring theuseof summative assessment. This approach couldpotentiallyprovideautomatedfeedbackderivedfromtheanalyticstothelearner.Usingthisapproach,metacognitiveactivityandreflectioncanbeencourageddirectlywithinthelearningprocess,ratherthanviaassessment.

3 REFLECTIVE WRITING ANALYTICS

WeconsiderReflectiveWritingAnalytics(RWA)tobetheanalysisofreflectivewritingandreportingofresultantinformationaboutthewritersandtheircontexts.ThecomputationalanalysisthatfollowsisRWAforthediscoveryofmetacognitiveactivityonthepartoflearners.

3.1 Data

The researchpresentedhere involved theanalysisof6,090student reflectionscapturedprogressivelyoverasemesterusingawebapplicationcalledGoingOK.1Sixhundredandfifty-sevenstudentsfromthreedifferentBachelor’sDegreeprogramsparticipatedinthestudy:athird-yearEnvironmentalEngineeringunit(DS-E);afirst-yearScienceunit(DS-S);andafirst-yearInformationTechnologyunit(DS-I).Foreachunit,theuseofthesoftwareandparticipationinthestudywasoptional,andthenumberofstudentswhowrotereflectionsusingthewebapplicationissummarizedinTable2.Ineachcase,reflectivewritingwasa requirement of the unit; however, the nature of this requirement and the extent to which it wasassessed, differed markedly. All data in this study is drawn from reflections written using the webapplication.

1GoingOKwaswrittenbyAndrewGibsonaspartofaQUTEducationresearchprojectthatcollectedthereflectionsofearlycareerteachers as they transitioned from student and developed their professional identities during their first year of teaching(http://GoingOK.org/)



Table2:Reflectivewritingdatasetsizes*

FullDataSet DS-E DS-S DS-I

Reflections N=6,090 188 740 5,162

Authors 657 35 145 477

Meanrefs/author 9.3 5.4 5.1 10.8

*“Reflection”isasingleentryinthewebapplication,“authors”arestudents,andthe“meanreflectionsperauthor(refs/author)”istheaveragenumberofreflectionswrittenbythestudentsoverthecourseofthesemester.3.2 Computational Analysis

Thecomputationalanalysiswasgovernedbytheconceptualmodel(Section2),withparticularemphasisonthekeyfeaturesidentifiedinthecomponentview.Ouraimwastoexploretheextenttowhichtheseconceptualfeaturesmaycorrespondtolexicalandstructuralfeaturesinthereflectivewriting.Ourchoiceoffeatureswasbasedonawiderangeofpreviousresearchidentifyingrelationshipsbetweentextfeaturesandhumanfactors. Inparticular,wedrewonPennebakerandChung’s (2011)workonpronouns,andRyan’s(2011)workonlinguisticfeaturesinacademicreflectivewriting,alongwitharangeofotherwork(Wharton,2012;Tang&John,1999;Reidsemaetal.,2010;Ullmann,Wild,&Scott,2012).

We linked the conceptual model to the reflective text indicators via an algorithm that progressivelymapped low-level grammatical features in the writing (posTags) through to higher level annotations(metaTags),whichwederivedfromthemodel.Thealgorithmthatwedevelopedcomprisedfourlevels:

• Partofspeech(POS)taggingofasentence(posTags)• MatchingofPOStagpatternstoidentifykeyphrases(phraseTags)• Matchingofphrasepatternstoidentifypotentialforannotation(subTags)• Filteringofmatchedphrasepatternstoselectfinalrelatedmodelcomponent(metaTags



ThealgorithmwasimplementedinScala.2POSTaggingwasimplementedusingtheFactorieframework(McCallum, Rohanimanesh, Wick, & Schultz, 2008) with a pre-trained parser model based on theOntoNotesEnglishcorpus.Whilethisisnotreflectivewritingspecific,itprovidesstateoftheartaccuracyinPOStaggingandhasprovedeffectiveforthepurposesofexploringthepotentialoftheapproach.

Informed by both the literaturementioned above and exploratory trials on the data,we selected 17phrasetagpatternsthatcapturedarangeofpotentiallymeaningfulphrases.Thesewerebasedonthe5generalPOSpatterns listed inTable3.ThefinalphraseTagswereobtainedbyfilteringthePOSpatternmatching results by a lexicon for each phraseTag.3 The filtering lexicons were derived by manuallyidentifying phrases from amongst the results from the POS pattern matching that showed potentialalignmentwith themodel.Thepronoun–verbandprepositionmatchingalsohavegeneral tags,whichincludedallphrasesnotspecifically filtered.Totunethealgorithm,the listsofgeneralphrasescanbesearchedforanyimportantphrasesmissedbythefilter.

AsecondmatchingpatternselectssubTagsbasedonthephraseTagpattern,andthenafinalmatchingpatterncombinesthesesubTagstoprovidethefinalmetaTagthatlinkswiththemodel.Thesepatternswereselectedusingtwocriteria:1)theindicativemeaningofthephrase,and2)theextenttowhichthe 2Examplecodecanbefoundathttp://nlytx.io/2016/metacognition3Thefullfilteringlistcanbefoundinthecodeonlineathttp://nlytx.io/2016/metacognition

Table3:POSpatternsfortheselectionofphraseTags

Pattern CorrespondingphraseTags

possessivepronounfollowedbyanyadjectivesandnouns

selfPossessive(e.g.,myteam)groupPossessive(e.g.,ourgroup)othersPossessive(e.g.,herproject)

pronounfollowedbyanyverbs,adverbs,conjunctions,andprepositions

consider(e.g.,wedecidedtogo)anticipate(e.g.,weneeded)emotive(e.g.,iamcomingalongreallywell)generalPronounVerb(e.g.,wehad)

prepositionfollowedbyrepetitionofanyPOS compare(e.g.,likeweare)temporal(e.g.,aftergetting)pertains(e.g.,withblooddonations)manner(e.g.,withoutdistractions)outcome(e.g.,fromourpracticepitchlastweek)generalPreposition(e.g.,athand)

modalfollowedbyanyverboradverb definite(e.g.,cantrust)possible(e.g.,wouldbefocusing)

personalpronounfollowedbyanyPOSandendingwithapersonalpronoun

selfReflexive(e.g.,ifeellikei)andgroupReflexive(e.g.,wetimedourselves)



phraseneededtobecombinedwithotherphrasesinordertosupportthecorrespondingmetaTag.ThefinalchoiceofpatternsisoutlinedinTable4.

Table4:RelationshipbetweenmetaTags,subTags,andphraseTags

metaTag subTag PhraseTagPattern

RegulationmonitorControlAND(triggerORgoal)

Trigger outcome

MonitorandControl temporalOR(pertainsANDconsider)

Goal anticipateORdefiniteORpossible

Knowledge selfPossessiveORcompareORmanner

Experience emotiveORselfReflexive

The progressive layering approach thatwe used in the algorithmwas beneficial in twoways. First, itenabledustomaintainconnectionsbetweenthephrasesinthetextandthecomponentsinthemodelbyprovidingclearlinksbetweeneachlayer.Second,thelinksenabledustotraceinaccuraciesintheanalyticsbylookingattheunderlyingpatterns.However,thesebenefitscomeatacost,andevaluatingeachlayercanbeverytimeconsuming,particularlywithlargeamountsofdata.Weseethisapproachasafirststep,allowingustogainanindicationofhowwellthemodelandthealgorithmworktowardsthediscoveryofmetacognitiveactivityfromthelearner’swriting.AnexampleoftheapplicationofourmappingalgorithmtoasentenceisillustratedinTable5,andfurtherexamplesareprovidedintheAppendix.

Table5:ExamplederivationofmetaTagtagsfromasentence.

Text Onseeing

thesheeramount

ofit , Ithink we lhave

tosplitupthework.

posTag INVBG DTJJNN INPRP , PRPVBP

PRP MDVB

TOVBRPDTNN.

match start(IN)&repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)

start(PRP)&repeat(RB,VB,CC,IN)

start(MD)&repeat(RB,VB)

filter pertains(startsWithAny(across,to,of,that,on,among,about,under,over,with,within,around,whether,for,in,e.g.,i.e.))

consider(containsAny(feel,felt,seem,think,realise,being,thought,decide,know,appeared,learn,experience,focus,found,guess,believe,wonder,find,personal,reflect,brain,understand,understood,notice,myself,recent,imply,record,probably,emotion,physical,trust,deal,barely,pretty,extremely,incredibly,miraculously,supposed))

definite(startsWithAny(will,’ll,ca))

phraseTag

pertains consider definite

match temporalOR(pertainsANDconsider) anticipateORdefiniteORpossible

subTag monitorControl goal

metaTag

regulation



3.3 Early Decisions

Anumberofkeydecisionsweremadeduringthecomputationalanalysis.Theseinvolvedquestionsonhowweapproachedtherawdata,andalsomoreimportantlyhowwewouldapproachthegranularityoftheresultsthatarosefromthealgorithm.

Thewebapplicationusedforcollectingthereflectivewriting(GoingOK)includestemporalinformationintheformoftimestamps.Anearlydecisionregardingthedatawastoignorethistemporal information.Althoughthetimestampdatacouldpotentiallybeusefulinshowingthedevelopmentofmetacognitiveactivityovertime,wefeltthatutilizingitforthisprojectwouldaddunnecessarycomplexitytotheanalysiswithoutprovidingsignificantvaluetoaddressingtheresearchquestions.Thus,ourattentionwasfocusedonevidenceofmetacognitiveactivityratherthanondevelopmentofthatactivity.

Initially,wehadalso consideredcombiningall reflections fora singleauthor intoonedocument.Thiswouldhavesignificantlyreducedthenumberofreflections(1perauthor)andforauthorswithverybriefreflections,itwouldhavepotentiallyprovidedahigherprobabilityforthedetectionoffeatures.Howeveritbecameapparentfromearlyobservationsofthedata,thatthemodelwewereseekingwasevidentinsingle individual reflections (even short reflections), and thatwe gainedmore information about theauthor’smetacognitiveactivitybykeepingthereflectionsseparateforanalysis,andthenconsideringtheresultant analysis as a whole with respect to each author, such as, for example, the frequency ofmetacognitiveactivity.

3.4 Categorization of Metacognitive Activity Asthisworkwasexploratoryinnature,wetookacautiousapproachwithrespecttothegranularityoftheanalysis. We settled on three categories of reflective writing: 1) reflections that showed significantevidenceofmetacognitiveactivity,termed“strong”reflections;2)reflectionsthatshowedverylittleornoevidenceofmetacognitiveactivity, termed“weak”reflections,and3) reflectionsthatwereneitherclearly strongnorweak, termed “undetermined.”Originally,weanticipated thatdistinctionsbetweenthesegroupswouldbereasonablyeasytomake,however,thevariabilityinthedatameantthatdecisionsregardingtheircategorizationweremorecomplexthanwefirstthought.

Weundertookaseriesoftrialstodetermineappropriaterulesforclassification.Toreducecomplexity,thesewereallundertakenonthesmallerDS-Edataset.Inwhatfollows,wewillexplainhowwearrivedattheheuristiccategorizationrules(listedinTable6)appliedtotheotherdatasets(seeTable7).Thiswillthenleadtofurtheranalysisandexplorationoftheresults.

Initialcategorizationwasbasedonthenumberofuniquemetacognitivetagsassignedtothereflection,assumingthatitcontainedaregulationtag.Thisrequirementwasappliedbecauseregulationisacorecomponentofthemodel,andsoalackofregulationwasexpectedtoidentifyaweakreflection.Ifthereflectionwastaggedwithallthreetags(regulation,knowledge,andexperience),thenitwasclassifiedasstrong.Weakreflectionswereidentifiedbyzerotagsorasingletagthatwasnotregulation.



Weusedaprocesscalledanomalyrecontextualization(Gibson&Kitto,2015)torefinetheheuristicsofthecategorizationrules.Thisinvolvedrunningtheanalysiswithagivenrule,identifyinganomaliesintheresults,thenusingadditionalfeaturestoresolveorrecontextualizetheanomalies.Thefirstcategorizationresultedinanumberofanomaliesinboththestrongandweakcategories.Furtherexaminationofthesereflections revealed a correlation between the reflections’ sentence counts and the anomalies in theresults. Although the number of sentences did not directly correlate with metacognitive tagging, allanomalousreflectionsthatshouldhavebeenidentifiedasstronghadaminimumofthreesentences.Wewereabletousetheminimumthreesentencefeaturetoresolvetheanomalyanditwasaddedtotherules.

Itwas found inearlyexperimentation that thecomputationalanalysiswasbiased todeemreflectionscomprisingasmallnumberofsentencesas“strong.”ThiswasduetoametriccalledsubTagDensity,whichnormalizedtagcountstothenumberofsentencesbytakingthetotalnumberofsub-componenttagsanddividing themby thenumberof sentences.Weutilized thisbecauseof thesignificantvariation in thelengthofreflections.However,thishadtheadverseeffectofcausingsomereflectionswithaverysmallnumberofsentences(e.g.,2)andanormalnumberoftags(e.g.,4)toberatedhigherthanreflectionswithslightlyhighersentencecount,makingthemmorelikelytobecategorizedasstrong.Weaddressedthis issue bymodifying subTagDensity to divide by the log of the sentence count (instead of just thesentencecount),andbyascribingavalueof0 tosubTagDensity forany reflectionswith fewer than2sentences.

Aspartoftheinitialcategorizationtrialswehadalsoidentifiedreflectionsthatweredifficulttocategorizeandtaggedthemaccordingtotheirlikelihoodtobeascribedacategory.IfthereflectionswereveryclosetobeingcategorizedasstrongthealgorithmascribedatagofpossibleStrong,andiftheywereclosetobeing categorized as weak, then they were tagged as possibleWeak. We examined the resultantreflections and noted that the potential changes identified above would resolve these boundaryreflectionsinfavouroftheexpectedcategory.Consequently,wemadetheadjustmentsdescribedandthecombinedchangesresultedinafinalcategorizationalgorithmthatutilizedtheheuristicrulesshowninTable6.

Table6:Heuristicrules

Class Rules

Strong sentenceCount>=3ANDmetaTagCount=3 OR sentenceCount>=3ANDmetaTagCount=2ANDmetaTags(regulation)ANDsubTagDensity>=4

Weak NOTmetaTags(regulation) AND metaTagCount<=1ORsubTagDensity<=3



Table7showstheresultingreflectionquantitiesforeachdatasetafterclassification.ItispossiblethatthesmallnumberofstrongandweakcategorizationsfortheDS-Edatacomparedwiththeotherdatasetsmayproveproblematic,asthecategorizationalgorithmwasinitiallytunedontheDS-Edataset.Weexaminethispotentialproblemfurtherbelow.

Table7:Reflectionnumbersaftercategorization.

3.5 Sentence Level Check

Atthispoint,wewantedtocheckwhethertheheuristicruleswereworkingappropriatelywithdatamoregenerally.Wewereparticularlyinterestedinstrongclassificationsasthismostdirectlyaddressedtheaimoftheresearch.ThischeckwasperformedbycomparingthemetaTagsofindividualsentencesannotatedbythealgorithmagainstamanualcodingofthesentencesundertakenbyoneoftheauthorsofthispaper.AstherewereonlyfourreflectionsintheDS-Edataset,wedecidedtocode“strong”reflectionsfromtheDS-Sdataset.Howevertokeepthetaskmanageable,wecodedthesmallernumberof“weak”reflectionsfromtheDS-Edataset.Wealsolimitedthetotalnumberofreflectionsbychoosingreflectionsbyauthorwhereatleastthreeoftheirreflectionswerecategorizedinthecategorybeingchecked.Thatis,forthestrongclassification,wechoseanauthor’sreflectionswhereatleastthreewerecategorizedasstrong.

Theintuitionbehindthisapproachwasthatalthoughwewerecheckingatthesentencelevel,theaimwastodiscovermetacognitiononthepartoftheauthor,andthereforethesentencesshouldultimatelybeunderstoodinthiscontext.Byincludingallofaselectedauthor’sreflectionsweretainedagoodselectionofreflectionswithinanauthor-centredcontext.Wealsoexpectedthattakingthisapproachwouldassistwithmitigatingagainstwritingstylevariance.ExamplesentencescanbeseeninTable8.ThecheckisofthemetaTagascribedtothesentencethatoccurspriortothestrong/weakclassificationperformedatthereflectionlevel.

All % DS-E % DS-S % DS-I %

Strong 837 13.8% 4 2.1% 59 8.0% 774 15.0%

Weak 4,180 68.6% 160 85.1% 564 76.2% 3,456 67.0%

Undetermined

1,073 17.6% 24 12.8% 117 15.8% 932 18.0%

Total 6,090 100.0% 188 100.0% 740 100.0% 5,162 100.0%



Table8:Examplesentences.

Check Sentence MetaTag SubTag PhraseTag

DS-SStrongAuthors

truePos ComparedtoothergroupsIfeltasthoughweareunder-preparedandthisismostlikelyduetothefactthatwestartedwellaftermostothergroups.

regulationexperienceknowledge

monitorControlgoalexperienceknowledge

anticipateemotivegeneralPronounVerbcomparetemporalpertains

falsePos Sowestartedthisweek’smeetingfirstlybydelegatingjobs,Ihadprintedoffthetemplatesuppliedonlearningresourcesinwhichwecouldallocateajobtitletoeachmemberpermeeting,asIwasthescribelastweekwenominatedKeithtodoitthisweek.

knowledge triggergoalknowledge

anticipategeneralPronounVerbpertainsmanneroutcomepossible

trueNeg Duringthischatthewholegroupco-operatedandeachhadavaluedinputintothetask.

falseNeg NexttimeImightjustwaitforsomeoneelsetochangethediscussionbacktothetopicsoI’mnottheonlyone.

goal generalPronounVerbpertainspossible

DS-EWeakAuthors

truePos Alltheassignmentsaredrivingmecrazy,andIfeellost. experience experience

considergeneralPronounVerbselfReflexive

falsePos YesterdayIsubmittedmyGroundwaterreport! knowledge knowledge

selfPossessivegeneralPronounVerb

trueNeg Justreportedtheprojectprogresstothelecturer.

falseNeg Itisalwaysdifficulttoinitiateaproject,whereeverymoveseemsheavyanduncertain. generalPronounVerb



3.5.1CheckofmetaTagforStrongAuthors(DS-SDataset)SelectingfromtheDS-Sdataset,basedonauthorswiththreeormorestrongreflections,yieldedatotalof272sentencesforfourauthors.Wemanuallycodedthedataforfalsepositives,falsenegatives,truepositives,andtruenegatives.Table9showsthat,overall,78%oftheclassificationswerefoundtomatch,with22%notmatching,themajorityofthesebeingfalsenegatives.

Table9:DS-SstrongauthormetaTagevaluation

Positive Negative Total %

true 55 156 211 78%

false 15 46 61 22%

Further analysis of thenon-matching classificationswas conducted todeterminehow thesemightbeavoided.Forthefalsepositives,wefoundthatnineofthe15relatedtoametaTagofknowledge,andcould have been classified as true negatives if the trigger subTag was used to prevent tagging asknowledge.Wealsofoundthatthiswouldresultinnoadditionaltruepositives.Inaddition,fiveofthefalsepositiveswerebasedontheregulationmetaTagandalloftheseinstanceswerebracketedbytruenegative sentences. This suggests these falsepositives couldbeaddressedby taking intoaccount theclassificationofthesentencesbeforeandafter,whichalignswithhowpeopletendtoreflect,becauseanideaismorelikelytobeencapsulatedinanumberofsentencesratherthanjustone.

Forthefalsenegatives, thereappearedtobestrongcorrelationswiththesubTags,providingaway inwhichmanyofthefalsenegativescouldbeclassifiedastruepositives.Inviewofthis,weconsideredthatwemayneedtoaddressthewayregulationisarrivedatfromtheunderlyingsubTags.Howeverwealsonotedthatthiscouldhaveresultedinalargeincreaseinfalsepositives.Wefoundthat17ofthe46wouldhavebeentruepositivesifthegoalsubTagwasused,fourifthetriggersubTagwasused,and14ifthepertains phraseTag was used. The numbers provide an indication that this area of the classificationprocessrequiresfurtherrefinement,andthattheapproachwillneedtobemorecomplexthanpurelyutilizingthesubTagsandphraseTags.

Whencheckingthedata,wewerenotsurprisedtofindthatauthorstendedtowardaparticularstyleofreflection. We thought that these stylistic differences may have had a material impact on theclassification,soweseparatedtheresultsforeachauthor(seeTable10).Thiscouldbeanareaforfurtherinvestigationaspartofamoreauthor-focusedanalysis;however,forthisstudywefoundtheresultstobelessvariablethanexpected,anddidnotpursuethislineofenquiryfurther.



Table10:Resultsbyauthor.

Author1 % Author2 % Author3 % Author4 %

falsepositives 11 0.1% 2 0.5% 0 2.0% 2 1.2%

falsenegatives 23 0.9% 11 0.5% 3 0.0% 9 0.6%

truepos&neg 121 1.9% 41 2.7% 15 2.0% 34 2.6%

35 1.0% 15 1.0% 6 20.0% 15 10.8%

3.5.2CheckofmetaTagforWeakAuthors(DS-EDataset)Initiallyweselectedreflectionsfromauthorswiththreeormorereflectionscategorizedasstrong.Werecognizedthepossibilitythatthealgorithmmayshowsignsofsuccesswiththisselectionifover-fittingwasoccurringwithinthehigherqualitydata.Therefore,wefeltthatweneededtoundertakeasimilarevaluationwith sentences fromauthorswithmostlyweak category reflections. That is, a selectionofreflections based on authors with three ormoreweak reflections.We used the DS-E dataset, whichyieldedatotalof437sentencesfor22authors.Likethepreviouscheck,wemanuallycodedthedataforfalse positives, false negatives, true positives, and true negatives (see Table 11). Overall, we wereencouraged by finding very similar results to the previous study, with 80% matching, and 20% notmatching (comparedwith78%and22%respectively fromthestrongauthors).Adifferentiating factorwith the weak author reflections was that most mismatches were false positives (rather than falsenegatives).Thismakessense,asitsuggeststhatouralgorithmispossiblynotasaggressiveasitcouldbe.Amoreaggressivealgorithmwouldlikelyresultinareductionoffalsenegativesinthestrongreflectionsaswellasreducethenumberoffalsepositivesintheweakdata.

Table11:DS-EweakauthormetaTagevaluation.

Positive Negative Total %

true 53 296 0 80%

false 61 27 349 20%

Ofthe61falsepositives,42belongedtoreflectionsclassifiedasweak,meaningthatthefalsetaggingofthesesentencesdidnotmakeamaterialdifferencetotheoutcome.However,40ofthesesentencesweretaggedwithknowledge,indicatingthatifthiseffectappliedtoallreflections,thenfewerreflectionsmaybetaggedasweak.Amongtheother19falsepositives,12werepartofundeterminedreflections,whichcouldpossiblybecomeweak iftheclassificationwas improved.Moreofan issuearosewiththesevenfalsepositivesamongreflectionsclassifiedasstrong.ThesewereallbasedonaregulationmetaTag,andbecausethisisacriticalcomponentofastrongclassification,allsevenofthesereflectionswouldhavebeenclassifiedasundeterminedifthesefalsepositiveswerefixed.Theseissuessuggestedtousthatthesideeffectsofthefalsepositivesonstrongclassificationissignificantandworthyoffurtherattention.



Withtheexceptionoftwosentences,thefalsenegativeswerepartofreflectionsclassifiedasweak,soweneededtoassesstheextentoftheircontributiontothefinalreflectioncategorization.Inourassessment,allbutoneofthesesentencesshouldhavebeenclassifiedasexperience,andnoneoftheothersentencesinthereflectionsincludedaregulationtag,sotherewasnomaterialchangetothereflectionclassification.Similarlywithonethatcouldhavebeentaggedasregulation,therewerenoothersentencesinthesamereflectiontaggedasknowledgeorexperiencesoastoinappropriatelypromotethereflectiontothestrongcategory.Thissuggeststousthatdespitethefalsenegatives,thereisamoderatingeffectinthealgorithmthatarisesbytakingtagsfromallsentencesindeterminingthereflectionclassification,whichappearstobeagoodthing.

3.5.3AlgorithmChangesFrom the checks, we determined that only two changes to the algorithm would not have adverselyaffectedtheevaluationdata.Bothofthesechangesrelatedtothestrongauthorsentences.

The first was to consider the relationship between the trigger subTag and the knowledge metaTag.Lookingattheoriginalphrasepatternmatching,wededucedthattheissuemighthavepresentedduetothesimilaritybetweentheunderlyingphraseTagsofoutcome(relatedtotrigger)andcompare(relatedtoknowledge).ItappearedthatitwouldbebettertohaveincludedcomparetogetherwithoutcomeforthetriggersubTag;however,wedeterminedthatfurtheranalysisisrequiredtoascertainthatthisdoesnotadverselyaffectotherclassifications.WealsonotedthatissueswiththeknowledgemetaTagwerealsopresent in theweakauthordata.This reinforces the fact that thecompositionof this tagneedsmoredetailedexamination.

Thesecondchangeinvolvedmoderatingregulationtaggedsentencesbasedonsentencesbeforeorafter.That is, ifthesentencebeforeorafterhasnometaTag,thenremovetheregulationmetaTag.Wealsodeterminedthatitwouldbebesttonotimplementthischangeuntilitseffectshadbeenexaminedwithinalargerdatasettodeterminewhethertherewouldbeanyadversesideeffects.

3.6 Metacognitive Analysis by Author

OurfinalanalysisforthisstudywasonDS-I,thelargestofthethreedatasets.Whilethesizeofthedatasetdidnotpresentanycomputationalproblems,itdidmakeourassessmentoftheresultsproblematic.Thelackofpre-annotatedreflectivetextcorporamakesitdifficulttoundertakehigh-qualityevaluations,andalthoughwecouldchoosesmallersubsetsofDS-EandDS-Stomanuallyannotate,theDS-Idatasetwasnotquiteasstraightforward,asitdidnothaveaclearsubsettoworkwith.However,thelargersizeofDS-Idoeshavesomebenefits;repeatedpatternscanbecomemoreobvious,ascananomaliesinthedata.Ourapproachwastoexaminethedataasawholeforobviouspatternsandanomalies.

Inlinewiththeoverallresearchobjectiveofidentifyingmetacognitiveactivityinthelearner,wedecidedtotakeanauthor-centricviewofthedata.Thefirsttaskweaddressedwasawaytoranktheauthors.Wewantedourrankingalgorithmtoranktheauthorsexhibitingthemostmetacognitiveactivityhighest,andauthorsexhibitingtheleastactivitylowest.However,weknewthatauthorsusuallyhadamixofstrong,



weak,andundeterminedreflections,andwealsoknewthattherewasawidevarianceinthelengthandnumberofreflections.Wealsoknewthat,onaverage,thereweremorethanfourweakreflectionstoeverystrongreflection(seeTable7).Inordertoaddressthesefactors,wesettledonanalgorithmthatweightedstrongreflectionsas+2andweakreflectionsas-1.Undeterminedreflectionswereweightedwith0.Wesummedtheweightingsofallreflectionsofanauthoranddividedtheresultbythelogofthetotalnumberofreflections.

This algorithm proved useful in ranking the authors, and on running itwewere immediately able toidentifytwokeyanomalies:thehighestrankedauthorandthelowestrankedauthor.Bothrankingsoftheseauthorswereoutsidethealmostsymmetricrangeoftheother475authors(-7.44to+7.82)withthehighestrankedauthorat+9.232andthelowestrankedauthorat-27.126.Whatwasinterestingaboutthese two authors, however, was the significant difference in their reflections to the closest rankedauthors.

Thehighestrankedauthorwroteverylongreflections,thelongestbeing53sentencesandtheaveragebeing25.9sentences.Whilesomeauthorswrotelongreflections,theytendedtowriteveryfew,whereasthisauthorwrote15reflectionswithonly fiveof themhavingfewerthan20sentences.However, thelargeamountoftextforthisauthoralsohighlightedpatternsinthephraseTagsthatwerenotimmediatelyobvious with smaller reflections. For example, there appeared to be a very large number ofgeneralPronounVerbtags, indicatingthat related filteringmaybetooaggressiveandthatmorewordswereneededintheassociatedlexiconstominimizephrasesfallingthroughtothegeneraltagratherthanbeingcaughtbymorespecifictags.ThelongertextsalsorevealedanunusuallyhighnumberofpertainphraseTags,oftenfourtimesgreaterthanthenexthighesttag.Thiscouldmeanthatthefilterforpertainsisnotaggressiveenoughandneedstobetightenedup.Finally,weobservedthatgoalisoftenthemostfrequentsubTagandthatitoftenco-occurswithtrigger,butregulationislessfrequent.ThisindicatesthateitherweeitherneedtoimprovemonitorControldetection,orweneedtorelaxtherulefortheregulationmetaTag.

Attheotherendoftherankings,thelowestrankedauthorwrotealotofveryshortreflections,145intotal,ofwhich113wereonlyonesentencelong.Unlikethehighestrankedauthor,wedidnotfindanyobviousimplicationsforchangingthealgorithmusingthisauthor’sreflections.

Basedonthesefindings,wesettherangeofrankingsfrom-8to+8,whicheffectivelyexcludedthesetwoanomalousauthorsfromthedataset.TheresultsforauthorsinthisrangeareprovidedinTable12.

Aninterestingfindingfromtheseresultsisthecomparisonofthethreeclassificationsovertherankingrange.Figure3showsthattheundeterminedclassificationappearstohaveagreatereffectontheweakclassificationthanonthestrong,andthatthiseffectstrengthensinthetophalfoftherange.Thissuggeststhatwemightneed to fine-tune theclassificationalgorithmaround theboundarybetweenweakandundetermined. This is not surprising, as the algorithm is looking for the presence of patterns whenclassifyingasstrong,resultinginarelativelyclearboundarycondition,butitistheextentoftheabsence



ofthosepatternsthatresultsinthedivisionbetweenweakandundetermined.Wemayneedsomemorepositiveindicatorsoflackofmetacognitiveactivityinordertostrengthenthealgorithminthisarea.

Table12:DS-Sanalysisbyranking.

Figure3:PercentageofmetaTagsforeachrankingrange.

Bottom Lower Middle Upper Top All

Rank -8.0to-4.7 -4.69to-3.1 -3.09to-0.01 0to2.22 2.23to8.0 -8.0to8.0

Authors 47 100 173 106 49 475

Reflections 876 1,161 1,621 785 559 5,002

Meanrefs 20.33 12.99 11.6 12.7 13.59 13.85

Minrefs 12 5 3 1 3 1

Maxrefs 33 31 26 27 25 33

StdDev 6.06 4.62 5.11 6.12 4.6 6.12

Strong 16 24 179 238 304 761

Weak 790 991 1,043 359 137 3,320

Undeterm. 70 146 399 188 118 921

Strong% 1.8% 2.0% 11.0% 30.3% 54.4% 15.2%

Weak% 90.2% 85.4% 64.3% 45.7% 24.5% 66.4%

Undeterm.% 8.0% 12.6% 24.7% 24.0% 21.1% 18.4%



Alsoofinterestintheseresultsweretheminimumandmaximumnumberofreflectionsforeachrankingband.Overall,lowerrankedauthorswrotemorereflectionsthanthoseinthehigherranking.Thissuggeststhatmorewritingmaynotnecessarilyprovidebetterevidenceofmetacognitiveactivity.Therelationshipbetween the number of reflections and overall metacognitive activity could benefit from furtherinvestigation,asthismayhaveimportantimplicationsfortheteachingofreflectivewriting.

4 IMPLICATIONS FOR LEARNING ANALYTICS

WecontendthatourworkholdsanumberofimplicationsforthefieldofLearningAnalyticsthatwouldbeworthyoffuturepursuit.

First,anumberoffindingsarisingfromthedevelopmentoftheconceptualmodelcouldimpactthefield.Wefounditessentialtoclarifytherelationshipbetweenmetacognitionandreflectionpriortotheworkonthecomputationalalgorithm.Weseethisclarificationasafirststep,withalotofpotentialforfurtherdevelopmentofthemodel,particularlyaroundtheexpansionofthemodelintotheexternal/socialarea.

Asourinterestwasfundamentallyonmetacognitiveactivity,wedidnotexaminethevariousmodelsofreflectivewriting,northerelationshipsbetweenthemandthecognitiveactofreflection.Webelievethatthe conceptual model would benefit from further work in this area. A greater understanding of therelationships between cognition and writing may also inspire new approaches to reflective writinganalytics.

Thelackofanannotatedcorpusofreflectivewritingturnedouttobeasignificanthandicapinundertakingthisstudy.Weareconsciousthatafullandproperevaluationofthevariousheuristicswasnotundertaken,andthatbeingabletotestthealgorithmsagainstapre-existingannotateddatasetwouldhavebeenasignificantbenefit.However,wealsonotethatthelackofthistypeofresource,andtheethicalissuesincreatingone,offersanopportunityforthedevelopmentofnewapproachestoevaluatinganalytics.Onesuchmethodcoulduseautomatedfeedbackasamechanismtocollectvalidationfromtheauthor;thatis,providinginformationbacktoauthorsabouttheirreflectionsandrequestingaresponseastotheextenttowhichitisaccurate,reasonable,fair,useful,etc.Thiswouldenableanautomatedannotationofthecorpus,whichcouldbeusedtotrainmachinelearningmodelsforcomputationalanalysis.

Relatedtothisidea,istheautomatedgenerationofformativefeedback.Fromthereflectiondata,wecaneasilyidentifykeydomainphrasessuchas“mygroup,”“ourproject.”Wecanalsoidentifydeficienciesinthetypeofreflection,suchasalackofexpressionofpersonalfeelings.Puttingthesetwoaspectstogetherwouldallowustogeneratefeedbackalongthelinesof“Perhapsyoucouldwriteabouthowyoufeelaboutyourgroup.”

Finally,asidentifiedabovewithrespecttotheconceptualmodel,reflectionisneitherrestrictedtowritingnortobeingexpressedlinguistically.Weconsiderthecomputationalanalysisofnon-linguisticreflectionto be a particularly interesting area for future research, with potential benefits in applying learninganalyticstosportsperformance,musicalimprovisation,andothernon-linguisticlearningactivities.



5 CONCLUSION

Inthisstudywehaveexaminedarangeofconceptualizationsofmetacognitionandreflectionandbroughtthemtogetherintoaunifiedmodelforthepurposeofprovidingclarityastohowthefeaturesmaybeanalyzed computationally from reflective writing. Based on this conceptual model, an algorithmwasdevelopedthatcategorizesreflectionsbasedonmetacognitiveactivity.5.1 Limitations

Becauseoftheexploratorynatureofthisstudy,threesignificantlimitationsshouldbeaddressedinfuturework.First,wedidnotconductafullempiricalstudyofthealgorithmssotheresultsindicatedinthispaperareindicativeofthepotentialoftheapproachratherthanaconfirmationofitsvalidity.Weacknowledgethatinorder toestablish thealgorithms’ truepotential to the field,anumberofadditional considerationswouldneedtobetakenintoaccount.Inparticular,wewouldconsiderthefollowingnecessary:1)theuseofdifferentdatafortheevaluationthanthatusedtodevelopthealgorithm;2)manualannotationoftheevaluationdatawithmultiplehumanannotatorstoestablishinter-raterreliability;3)makingexplicitanydifferencesbetweenannotationcriteriaandalgorithmcriteria;and4)wherejustified,theapplicationofconventionalstatisticalanalysistotheresults.

Second,thedatasetusedforthisstudywasnotnecessarilyrepresentativeofreflectivewritingingeneral.Itinvolvedthecollectionofmostlybrief,unstructuredreflectionsonaregularbasisoveraperiodoftime.Thistypeofreflectionmaybesignificantlydifferentthanthelonger,morestructuredreflectivewritingtypicalofstudentassignments.Theeffectofthesedifferencesontheperformanceofthealgorithmisunknown.

Third,thecombinationofstructuralpatternsandlexicalfilteringnecessarilymeansthatsometextwillbeomitted or inadvertently included, depending on the parameters and underlying lexicons. Weacknowledge much room for improvement in disambiguation, pattern matching, and morecomprehensivelexiconsandfilteringprocesses.

5.2 Future Work

Theselimitationsprovidesomekeyareasforfuturework.Weseevalueinafullempiricalstudybasedontheapproachoutlined inthispaper.Weseepromise inboththeconceptualmodelandthealgorithmdesign,andbelievethatamorecompletestudywithrigorousevaluationwilladvanceboththeconceptualandappliedaspectsofthiswork.Wealsoseepotentialinevaluatingdifferentstylesofreflectivewriting;inparticular, investigatingthedifferencesbetweenshort, recurrentreflectionand longer form,single-instancereflection.Thealgorithmitselfpresentsmanyopportunitiesforfurtherrefinement,andwouldbenefit fromthe incorporationofnovel techniques for lexiconexpansionandpatterndisambiguation.



Anotherpotentialavenueofresearchisintheareaofgeneratingformativefeedbackfortheauthor.Thisisalargeandcomplexspacethatthisstudyhasonlyhintedat.

5.3 Final Remarks

Overall, the approach presented here to discovering learner metacognition through the analysis ofreflectivewritingshowsdemonstratedpotential,andweconsideritworthpursuinginfurtherresearch.Whilethiscouldbeuseddirectlyforthepurposeofgeneratingreflectivewritinganalytics,weidentifiedpotential for significant impact in the fieldof learninganalyticsbydeveloping the software further toincludefeedbacktothelearner.Morebroadly,thispaperisaninitial,yetsignificant,steptowardsaformofanalyticscentreduponmetacognitionandreflection.Wehavepresentedawayinwhichthedetectionof metacognitive activity in an author of reflective texts might be automated, and shown how thisapproachmightbeusedasaformativefeedbacksteptoencouragemetacognitivethinkinginstudents.Weconsiderthistobeakey21stcenturycompetency,andholdthatitisessentialthatwefindwaystonurturethisskillinourlearners.Itisourambitionforthispapertoopenupafruitfulavenueforresearchinthisdirectionoflearner-centredlearninganalytics.

REFERENCES

Amsel,E.,Klaczynski,P.A.,Johnston,A.,Bench,S.,Close,J.,Sadler,E.,&Walker,R.(2008).Adual-processaccount of the development of scientific reasoning: The nature and development ofmetacognitive intercession skills. Cognitive Development, 23(4), 452–471.http://dx.doi.org/10.1016/j.cogdev.2008.09.002

Boud,D.,Keogh,R.,&Walker,D. (Eds.), (1985).Reflection:Turningexperience into learning. London:KoganPage.

Committee for Economic Development of Australia (CEDA). (2015). Australia’s future workforce?Retrievedfromhttp://www.ceda.com.au/research-and-policy/policy-priorities/workforce

Cornford,I.R.(2002).Learning-to-learnstrategiesasabasisforeffectivelifelonglearning.InternationalJournalofLifelongEducation,21(4),357–368.http://dx.doi.org/10.1080/02601370210141020

Dewey,J.(1916).Democracyandeducation.NewYork:TheMacmillanCompany.Efklides,A.(2008).Metacognition:Definingitsfacetsandlevelsoffunctioninginrelationtoself-regulation

and co-regulation. European Psychologist, 13(4), 277–287. http://dx.doi.org/10.1027/1016-9040.13.4.277

Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature ofintelligence(pp.231–236).Hillsdale,NJ:Erlbaum.

Flavell, J. H. (1978). Metacognitive development. In J. M. Scandura & C. J. Brainerd (Eds.),Structural/process theories of complex human behavior. Alphen a. d. Rijn, The Netherlands:Sijthoff&Noordhoff.

Flavell, J. H. (1979).Metacognition and cognitivemonitoring: A new area of cognitive-developmentalinquiry. American Psychologist, 34(10), 906–911. http://dx.doi.org/10.1037/0003-066X.34.10.906



Gibson,A.,&Kitto,K.(2015).Analysingreflectivetextforlearninganalytics:Anapproachusinganomalyrecontextualisation.Proceedingsofthe5thInternationalConferenceonLearningAnalyticsandKnowledge(LAKʼ15),275–279.http://dx.doi.org/10.1145/2723576.2723635

Herber,H. L.,&Nelson-Herber, J. (1987).Developing independent learners. JournalofReading,30(7),584–588.

Koriat, A. (2000). The feeling of knowing: Some metatheoretical implications for consciousness andcontrol.ConsciousnessandCognition,9,147–171.http://dx.doi.org/10.1006/ccog.2000.0433

Lehmann,T.,Hähnlein,I.,&Ifenthaler,D.(2014).Cognitive,metacognitiveandmotivationalperspectivesonpreflectioninself-regulatedonlinelearning.ComputersinHumanBehavior,32(C),313–323.http://dx.doi.org/10.1016/j.chb.2013.07.051

Mair,C.(2012).Usingtechnologyforenhancingreflectivewriting,metacognitionandlearning.JournalofFurther and Higher Education, 36(2), 147–167.http://dx.doi.org/10.1080/0309877X.2011.590583

McCallum, A., Rohanimanesh, K., Wick, M., & Schultz, K. (2008). FACTORIE: Efficient probabilisticprogrammingforrelationalfactorgraphsviaimperativedeclarationsofstructure,inferenceandlearning.PaperpresentedattheNIPS*2008WorskshoponProbabilisticProgramming,Whistler,Canada.Retrievedfromhttps://people.cs.umass.edu/~sameer/files/factorie-nipsws08.pdf

Mezirow,J.(1990).Fosteringcriticalreflectioninadulthood:Aguidetotransformativeandemancipatorylearning.SanFrancisco,CA:Jossey-Bass.

Moon,J.A.(1999).ReflectioninLearningandProfessionalDevelopment:Theoryandpractice.NewYork:PsychologyPress.

Paul,R.(1990).Towardsanewmeasureofsuccess:Developingindependentlearners.OpenLearning:TheJournal of Open, Distance and e-Learning, 5(1), 31–38.http://dx.doi.org/10.1080/0268051900050106

Pennebaker,J.W.,&Chung,C.K.(2011).Expressivewriting:Connectionstophysicalandmentalhealth.InH.S.Friedman(Ed.),Oxfordhandbookofhealthpsychology.NewYork:OxfordUniversityPress.

Quirk,M.E.(2006).Intuitionandmetacognitioninmedicaleducation.NewYork:Springer.Reidsema,C.,Goldsmith,R.,&Mort,P.(2010).Writingtolearn:Reflectivepracticeinengineeringdesign.

Paper presented at the 9thAnnual American Society for Engineering Education (ASEE)GlobalColloquium on Engineering Education, 18–21 October 2010, Singapore. Retrieved fromhttps://www.academia.edu/501660/Writing_to_Learn_Reflective_Practice_in_Engineering_Design

Ross,J.(2011).Tracesofself:Onlinereflectivepracticesandperformancesinhighereducation.TeachinginHigherEducation,16(1),113–126,http://dx.doi.org/10.1080/13562517.2011.530753

Ryan,M.(2011).Improvingreflectivewritinginhighereducation:Asocialsemioticperspective.TeachinginHigherEducation,16(1),99–111.http://dx.doi.org/10.1080/13562517.2010.507311

Ryan,M. (2013). The pedagogical balancing act: Teaching reflection in higher education. Teaching inHigherEducation,18(2),144–155.http://dx.doi.org/10.1080/13562517.2012.694104

Sanders,J.(2009).Theuseofreflectioninmedicaleducation:AMEEGuideNo.44.MedicalTeacher,31,685–695.http://dx.doi.org/10.1080/01421590903050374



Schön,D.A.(1983).Thereflectivepractitioner:Howprofessionalsthinkinaction.NewYork:BasicBooks.Takanishi,R.(2015).Dooureducationsystemsdoenoughtoenablelearnerstoflourishasindependent,

autonomous and well-balanced individuals? European Journal of Education, 50(2), 123–127.http://dx.doi.org/10.1111/ejed.12113

Tang,R.,&John,S.(1999,December).The‘I’ in identity:Exploringwriter identity instudentacademicwritingthroughthefirstpersonpronoun.EnglishforSpecificPurposes,18,Supplement1,S23–S39.http://dx.doi.org/10.1016/S0889-4906(99)00009-5

Ullmann,T.D.,Wild,F.,&Scott,P.(2012).Comparingautomaticallydetectedreflectivetextswithhumanjudgements. In A.Moore, V. Pammer, L. Pannese,M. Prilla, K. Rajagopal,W. Reinhardt, T.D.Ullmann, C. Voigt (Eds.), Proceedings of the 2nd Workshop on Awareness and Reflection inTechnology-Enhanced Learning (AR-TEL ʼ12), Saarbrucken, Germany (pp. 101–116). Retrievedfromhttp://ceur-ws.org/Vol-931/paper8.pdf

Wharton, S. (2012). Presenting a united front: Assessed reflective writing on a group experience,Reflective Practice: International and Multidisciplinary Perspectives, 13(4), 489–501.http://dx.doi.org/10.1080/14623943.2012.670622

Wilson, M. D. (2015). Pracademia: The future of the lifelong learner. About Campus, 20(2), 28–31.http://dx.doi.org/10.1002/abc.21189

Zimmerman,B.J.(2002).Becomingaself-regulatedlearner:Anoverview.TheoryintoPractice,41(2),64–70.http://dx.doi.org/10.1207/s15430421tip4102_2



APPENDIX

Alivedemonstrationversionofthesoftwarecreatedaspartofthisstudy,aswellaslinkstothesourcecodeandvarioustechnicaldocuments,canbefoundat:http://nlytx.io/2016/metacognitionAdditionalexamplesfortheprocessinvolvedinderivingthemetaTagsfollow:Example1Text IknowI cancount on[name]todo

herbitandIknowshe willdo itwellasthe

recoverypartisnotnews

tous,EnviroSciencechicks.

posTag PRPVBPRP MDVB INNNTOVBPRPNNCC

PRPVBPRP MDVB PRPRBINDTNNNNVBZRBNN

TOPRPNNPNNPNNS

match start(PRP)&repeat(RB,VB,CC,IN)start(PRP)&any&end(PRP)


start(IN,TO)&repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)




start(IN,TO)repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)

filter containsAny(know)

startsWithAny(can)

startsWithAny(on)

containsAny(know)

startsWithAny(will)

containsAny(well)

startsWithAny(to)

phraseTag ConsiderselfReflexive

definite pertainsothers-Possessive

consider definite emotive pertains

match temporalOR(pertainsANDconsider)anticipateORdefiniteORpossibleemotiveORselfReflexive

temporalOR(pertainsANDconsider)anticipateORdefiniteORpossibleemotiveORselfReflexive

subTag monitorControl,goal,experience monitorControl,goal,experience

metaTag regulation,experience

Example2 Text Theyallseemed likelovelypeople andareinterested inparticipatingto

achievethebestpossiblemark.

posTag PRPDTVBD INJJNNS CCVBPJJ INVBGTOVB DTJJSJJNN

match start(IN,TO)repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)

start(IN,TO)repeat(PRP,JJ,NN,VB,MD,PDT,W,TO)

filter startsWithAny(like) startsWithAny(in)

phraseTag

compare pertains

match selfPossessiveORcompareORmanner

subTag knowledge

metaTag knowledge

towards the discovery of learner metacognition from reflective … · reflective writing, an...

Documents