blackboard learning analytics research update
TRANSCRIPT
Learning Analytics Research Findings Update
“Dr.John”WhitmerDirector,Analy6csandResearchJISCRemotePresenta6on|2.22.2017
1. LearningAnaly6csOverview&BbDataScience
2. MajorFindingsin20171. Varia6oninLMS“Effec6veness”
2. ToolUse3. CourseCategories4. StudentPercep6onsofDataDashboards(6me
permiYng)
3. BlackboardAnaly6csSolu6onsPor\olio4. Discussion
Learning Analytics Overview
Educational Technology Assessment Hierarchy
Doesitimpactstudentlearning?(LearningAnaly6cs)
Howmanypeopleuseit?(Adop6on)
Doesitwork?(SLAs)
What is Learning Analytics?
Learning and Knowledge Analytics Conference, 2011
“...measurement,collec6on,analysisandrepor6ngofdataaboutlearnersandtheircontexts,forpurposesofunderstandingandop2mizinglearningandtheenvironmentsinwhichitoccurs.”
Meta- questions driving our Learning Analytics research @ Blackboard
1.Howisstudent/facultyuseofBbpla\orms(e.g.Learn,Collab,etc.)relatedtostudentachievement?[orsa6sfac6on,orrisk,or…]
3.Whatdataelements,featuresets,andfunc6onalitycanwecreatetointegratethesefindingsintoBbproductstohelpfacultyimprovestudentachievement?
2.Dothesefindingsapplyequallytostudents‘atpromise’duetotheiracademicachievementorbackgroundcharacteris6cs?(e.g.race,class,familyeduca6on,geography)
Techniques
• Simula2onifX,whatY?(“WiththisUltraLearningAnaly6cstriggerrule,howmanystudentswouldtripno6fied?”)
• Hypothesistes2ng:inves6gateifaspecificrela6onshipistrue(“What’stherela6onshipbetween6mespentinacourseandstudentgrade”?)
• Datamining:analyzeunderlyinglatentpakernsindata(“WhattypicalpakernsintoolusecharacterizesBBLearncourses?”)
KeyDataSources
• LearnManagedHos6ng
• LearnSaaS
• CollaborateUltra
Main Big Data Sources & Techniques
Commitment to Privacy & Openness
• AnalyzedatarecordsthatarenotonlyremovedofPII,butde-personalized(individual&ins6tu6onallevels)
• Shareresultsandopendiscussionproceduresforanalysistoinformbroadereduca6onalcommunity
• Respectterritorialjurisdic6onsandsafeharborprovisions
Major Findings in 2016 (and one from 2017)
Relationship Student Use Learn vs. Grade
Bb Study: Relationship Time in Learn & Grade
• Distribution in Time Spent is highly skewed toward low access
• Transforming data (log
transform) can produce normal curves for analysis
• Of course, huge variation of
quality within that time spent (of course materials, of student activity)
Findings: Relationship Time in Learn & Grade • Question: what is the
relationship between student use of Learn and their course grade?
• Investigate at student-course level (one student, one course)
• 1.2M students, 34,519 courses, 788 institutions
• Significant, but effect size < 1%
Finding: Tool Use & Grade TooluseandFinalGradedonothavealinearrela6onship;thereisadiminishingmarginaleffectoftooluseonFinalGrade
Interpreta6ons• Studentsabsentfromcourseac6vityareat
greatestriskoflowachievement.• Thefirst6meyouread/seeaPowerPoint
presenta6on,youlearnalot,butthesecond6meyouread/seeit,youlearnless.
• GeYngfroma90%toa95%requiresmoreeffortthangeYngfroma60%toa65%.
Logtransforma2onshowsstrongertrend
But strong effect in some courses (n=7,648, 22%)
What makes some for a stronger or weaker relationship? Tools used? Course design? Quality of activity/effort?
Learn Tool Use vs. Grade
Investigation Grade by Specific Tools Used Ques6on:whatistherela6onshipbetweenuseofLearnandstudentgrade,basedonthetoolused?AnalysisSteps1. Filterdataforcourseswithpoten6almeaningful
use(>60minaverage,enrollment>10<500,gradebookused)
2. Iden6fymostfrequentlyusedtools3. Separatetooluseintonouse&quar6les4. Dividestudentsinto3groupsbycoursegrade
• High(80+)• Passing(60-79)• Low/Failing(0-59)
Finding: MyGrades Ateverylevel,probabilityofhighergradeincreaseswithincreaseduse.Causal?Probablynot.Goodindicator?Absolutely.
Finding: Course contents Moreisnotalwaysbeker.Largejumpnonetosome;thennorela6onship
Finding: Assessments/Assignments Studentsabovemeanhavelowerlikelihoodofachievingahighgradethanstudentsbelowthemean
Implications
• MovebeyondLMSuseasproxyforeffort(wheremoreisalwaysbeker),andgetatfiner-grainedlearningbehaviorsthataremoreuseful(e.g.studentswhoarestrugglingtounderstandmaterial,studentswhoarenotprepared).
• Majormissingelementsfromresearch
– fine-grainedunderstandingofac6vityover6me(e.g.crammingvs.consistenthardworking)
– qualityofcoursematerialsandcoursedesign
Patterns in Course Design
Research Questions Ques2ons
1. Aretheresystema6cwaysthatinstructorsuseLMStoolsintheircoursesthatspaninstructorsandins6tu6ons?
2. Whatrecommenda6onscanbedrawnforfaculty,instruc6onaldesigners,andotheracademictechnologyleadersseekingtoincreasetheimpactofLMSuseattheirins6tu6on?
Methods
1. Usesamefiltereddatasampleofstudent-coursedata
2. Calculaterela6vestudent6mepertool(as%oftotalcourse6me),forcomparisonbetweencourses
3. Clusterbypakernsinthebalanceof6mespentineachtool(unsupervisedmachine-learning;kmeansclusteranalysis)
4. Adddataasrelevanttopakernsaboutenrollment,total6me,etc.
5. Makeupcoolnamesforeachclusterandinterpretmeaning
Distribution of Courses by Type
Finding: Discussions with low/high avg use Comparecourseswithlowforumusetocourseswithforumuse>1hour/studentaverage
Summary & Future Directions for DS Research
Summary
• Tremendousvaria6oninuseofLearn;mostuseskewedtowardlow/verylowuse.
• Importanceof6mespentinLearnforlearningisalsotremendouslyvaried(“necessary”and“effec6ve”useofLearn)
• Cri6caltoaccountforthisvaria6ontounderstandpoten6alimportanceofLearnac6vity
FutureDirec2ons
• Analyzequalityofac2vityingreaterdepth(e.g.contentofassignments,wordsinforumposts)togetinsightsintoqualityofinterac6ons
• Conduct6me-seriesanalysis(quan6ta6vemethods,designalsoneeded);whensomeoneaccessesismoreimportantthaniftheydo.
• Createproxies/derivedvaluesforbehavior(aboveaverage,ataverage,etc.)bytool
3. Blackboard Analytics Portfolio
Blackboard Analytics – Product Naming
Blackboard Analytics Data warehouse products
Blackboard Analytics Suite of analytics products
BlackboardAnaly.cs
Product portfolio
Blackboard Intelligence • Analy6csforLearn–LMSdata• StudentManagement–SISdata• Finance,HR,Advancement–ERPdata
Blackboard Predict • Predic6veanaly6csandearlyalertsforreten6on• Providesdataforfacultyandadvisorsaboutat-riskstudents• FormerlyBlueCanary
X-Ray Learning Analytics • Classroomengagementdataforfaculty• Ac6vityaggregatedinto30+visualiza6ons• CurrentlyavailableforMoodlerooms&Self-HostedMoodlersonly
Pastview Currentview Futureview
Pastview Currentview Futureview
Pastview Currentview Futureview
Blackboard Analytics Solution Portfolio
X-RayLearningAnaly6cs
BlackboardAnaly6cs
BlackboardPredict
Analy6csforLearn
StudentManagement Finance HR Advancement
BlackboardIntelligence
Blackboard Analytics – Our Approach & Philosophy
Productsthatprovideinsightintotheteachingandlearningprocess
OurPhilosophy:Datacomplements
humandecision-making
Corecompetency:Learningso{wareandacademicdata
Ateamofexpertsintheanaly6csfield
Discussion Thank you!
John Whitmer, Ed.D. [email protected]@johncwhitmerwww.johnwhitmer.info/research