evaluang a province-wide early grade mathemacs intervenon...
TRANSCRIPT
Evalua'ngaprovince-wideearlygrademathema'csinterven'on:
ThecaseofR-Maths
7September2018NickyRobertsandEleanorHazell
Presenta'onoverview
1. R-Mathsprojectdesign2. R-MathsevaluaEondesign3. Testdesign4. FindingsfromtesEng:
• Subjectadvisors• Teachers• Learners
5. WhatarereasonableexpectaEonsforshiPsintestaQainment?
6. Andso?2
Spoileralert1:Stuffweknewin2007
Spoileralert2!Morestuffweknewin2007:
1. Schoolchangetakes'me.Ittakesaboutthreeyearstoseechangesinaprimaryschool,andfiveyearstoseechangesinasecondaryschool,dependingonsizeandcomplexity.
2. Thecoreofeduca'onistheteacherintheirclassroom.Thisisthehardestparttochange.Tomakerealchangesthiscoremustbereached.
3. Thereisnoprovenwayofchangingthedynamicsofindividualschoolsotherthanworkingcloselywiththem.
4
Paraphrased from Christie, Butler and Pottering (2007) Schools That Work, Ministerial Report to the Minister of Education, South Africa
1.R-Mathsprojectdesign
5
R-Maths
• LedbytheWesternCapeEduca'onDepartment
• IncollaboraEonwiththeUCTSchoolsDevUnit(SDU)
• Programmetraining&materialsdevelopedbytheSDUincollaboraEonwiththeWCED
• Fundedandsupportedbydonors
6
Goal: Improve the conceptual understanding
and Mathematical skills of Grade R learners in the Western Cape
R-Maths‘Form’:Modifiedcascade
TheR-Mathstraining–includingcoursematerialsandclusternotes–isbackboneofR-MathsSA’sreceive30hrsblocktraining,UCT/SACEassessment/PoE/accreditaEon,+/-30hourssupport(‘dryruns’)Teachersreceive14hrsclusterw’shops,30hrstraining,SACEaccreditaEonprovidedbytheSA’s,reflecEonworkshop,PLConR-Maths.FidelityofimplementaEonissupportedthroughfullsetofteacherresources
7
Process theory
Impact theory
Development of teacher training &
resources
Capacitation of district officials
Training of Grade R teacher/
practitioners
Resource use for teacher training & classroom teaching
SAs: Enhanced knowledge & skills to support Grade R
teacher/ practitioners
Teacher/practitioners: Enhanced knowledge & skills to
teach Grade R maths
Improved learner performance in mathematics
(readiness for Grade 1)
Effective teaching
Short-term
M
edium-
term
Long-term
(impact)
Process theory Impact theory Outcomes
Support to Grade R teacher/
practitioners
2016 2017 2018
Training and Materials Development:
70 SA training (30 hrs) UCT/
SACE accredited
J F M A M J J A S O N D
1500 Teacher training (30 hrs) SACE
accredited
1500 Teacher training (30 hrs) SACE
accredited
Cluster w
’shop (2 hrs)
2x Cluster
w’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
PLC
Reflection
Reflection
PLC
Dry run
Dry run
R-Maths‘Dosage’
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
Cluster w
’shop (2 hrs)
R-Mathstrainingandsupporttosubjectadvisors “dry-runs”
priortoclusterworkshops(2hoursx7)&blocktraining
Blocktraining:5days/30hoursx1
R-Mathsteachermaterials
11
R-Maths:‘substance’
• ExplicittheoryofGradeRlearning:PlayinmathemaEcs• 7principlesofR-Mathslearning• ExplicitintegraEonofmathsindailyprogramme• AddedtoandexpandedCAPSperterm
• Integratednumberintoalltheothertopics• Focusedon1topicperweek• Specificguidanceontopicfortheweekandsmallgroupwork,
repeatedover1week• Explicitdevelopmentalframeworkfornumberwork• AfewpowerfulrepresentaEons,consistentlyapplied(egdot
cards,numeralcards,structuredbeadstring
CAPSreferencetoplayinmathema'cs
TheapproachtolearningMathemaEcsshouldbebasedontheprinciplesofintegra'onandplay-basedlearning.Theteachershouldbepro-ac've,amediatorratherthanafacilitator.MakemostoflearningopportuniEesthatarisespontaneouslyduringarangeofchild-centredac'vi'es:- freeplayinthefantasycornerorblockconstrucEonsite,sandandwaterplay
acEviEes- teacher-guidedacEviEesthatfocusonmathemaEcalconceptssuchascounEng,
numberconceptdevelopment,spaceandshape,paQerns,EmeandotheremergentmathemaEcsacEviEes
Duringfreeplaytheteachercanpromoteemergentmathema'csthroughtheappropriatestructuringofthefree-playarea(CAPSGradeRMathemaEcs,page14;Grade1–3,page13)
7PrinciplesofR-Maths
1. Context2. AcEvity3. Level4. InteracEon5. Guidance6. Inclusivity7. PracEce
14
CAPSMathema'cs:Fivetopics
18
Data handling
Measurement
Shape and space (geometry)
Patterns, functions and algebra
Number operations and relationships
• TrajectoriesforLearning(Sarama&Clements,2009;TreffersvanDenHeuvel-Panhuizened,2012;vanDenHeuvel-Panhuizen,Kuhne,Lombard,2014)
• SubiEzing(Kaufmanetal.in1949;StarkeyandCooper,1995;Clements,1999)• CounEngprinciples(Gelman&Gallistel,1978)• MulEplerepresentaEons(Leinhardtetal.,1991;Hiebert&Carpenter,1992;
Greeno&Hall,1997)• UniEzing(Dewey,vonGlasersfeld&Richards,1983;vonGlasersfeld,1981,1995)
TheoriesunderpinningR-Maths
All young children have the same implicit understanding of maths but to make that understanding explicit, they need to hear the language of maths and get the chance to think about it and use it when they speak (Clements, 2016).
• Numberwordlist(ordinality)-acousEc/oral,stories,songs,rhymes,‘washingline’• Cardinality(howmanyinaset)-resultaEvecounEng,perceptual-conceptualsubiEzing,comparethencounttofind‘howmanymore/fewer’• One-to-onecorrespondence-extendaccuracytolargersets• Numbersymbols-represenEngnumbers:concrete,iconic,symbolic
2.R-Mathsevalua'ondesign
1. Focusedon‘productandprocess’aswellas‘outcomesandimpact’
2. Purposivelyselected2districts:1urbanand1district,andcasestudyschoolswithinthosedistricts
3. PhasedimplementaEonover2yearsallowedforacounter-factuallearnersinthesamedistrict(notpossibleforSAs)
4. Mixedmethodstogettodifferentlevelsofthesystem
21
Evalua'onques'ons
22
Productandprocessevalua'onques'ons1.WhatdoestheR-MathsProjectentail?2.Whatisthecontext(e.g.rural/urban,socio-economicstatusoftheschoolcommunity,numberofteachersandlearners,backgroundofGradeRteacher/pracEEoners,etc.)withinwhichtheimplementaEonistakingplace?3.Istheprojectbeingimplementedasplanned:inputs,outputsandtargets?4.Whatisthenatureandqualityofthetraining,supportandresourcesprovidedtoFPSubjectAdvisorsandteacher/pracEEoners?5.WhatisthelevelofadopEonbyFPSubjectAdvisorsandteacher/pracEEoners?6.Whatarethechallenges/barrierstoimplementaEonbydistrictsandschools?7.Whatarethesuccessesandchallengesofthepartnershipmodelthroughwhichthisprojectisbeingimplemented?8.Whatarethekeymechanismsofchangeinthecasestudycontexts?
Outcomeandimpactevalua'onques'ons9.WhatistheimpactoftheprojectonFPSubjectAdvisors,GradeRteacher/pracEEonersandGradeRteacher/pracEEoners’teachingpracEce?10.DoestheR-MathsProjecthaveanimpactonGradeRlearners’MathemaEcalknowledgeandskills?11.WhatarethesuccessesofandbarrierstoscalabilityandembeddednessintotheWCED?12.Hastheprojectmetitsintendedoutcomesoutlinedinthetheoryofchangeandlogicmodel?
10.WhatistheimpactofR-MathsontheSubjectAdvisorsandGradeRteacher/prac''oners’andteachingpracEce?11.DoestheR-MathsprojectimpactonGradeRlearnersmathemaEcalknowledgeandskills?
Datacollec'onmethodsandsources
23
Ac'vi'es:
Level
Project/
province
District/
Subject
Advisor
HoD Teacher Learner
Keystakeholderinterviews x x
SubjectAdvisortest x x
Teachertest x x
Training & training dry-run
observa'on
x x
CT&dry-runobserva'on x
Monitoring fidelity, tracking
“dosage”
x x x
Casestudies x x x
Learnertest x x
3.Testdesign
Howdoweassesstheknowledgeof:• SubjectAdvisors• Teachers• LearnersforsupporEng,teachingandlearningGradeRmathemaEcs?…andin3differentlanguages?
24
SubjectAdvisortest
Whatisourexpected‘knowledgeforteachingGradeRmathemaEcs”,forSubjectAdvisors?• Nocommonstandards…• Noagreed/commonassessmentsitems…• Novalidatedinstruments…• Comparabilityofanytestinstrumentacrosslanguages?So,(30min)testdevelopedbasedontrainingguidesandteacherconceptguide(drawingonexpertknowledge):- MixofWCEDpolicyrequirements- SubjectmaQerknowledge- PedagogicalcontentknowledgeaboutGradeR(atthelevelofwhatwasbeingtaught) 25
ExampleSAtestitems
26
Teachertest
1. ThesameissuesasforSAtest2. Shorterandwithfeweritems.3. QuesEonedre-phrasedasfromperspecEveofateacher
– YouobserveaGradeRteacher’smathslesson….to– InyourGradeRmathslesson…
27
Learnertest
28
AdaptaEonofGermantest48itemone-to-oneoralinterviewValidatedinGautengforEnglish,AfrikaansandisiZulu(+2)HLlearners
Learnertest
29
1. Testconductedorallyinschools,onechildataEme,administeredbytrainedtestadministrators
2. ChildrencompletedthetestintheirLoLT3. BaselineinFeb/Mar2017;endlineinOct/Nov20174. IniEalsimplerandomsampleoflearnersin2districts:- 168ineachoftheintervenEonandcomparisongroups,- 168incomparisongroup- DuetoaQriEonetc.,therewasatotalof622matchedtests
4.Testfindings
Whatdidwefindabouttheknowledgeof:• SubjectAdvisors• Teachers• LearnersforsupporEng,teachingandlearningGradeRmathemaEcs?
30
SAtestresultsPre-TestResults(n=47)• Meanscoreof48.6%.• Scoresrangedfrom20%to80%
31
Post-TestResults(n=47)• Meanscoreof66.1%.• Scoresrangedfrom46%to92%• Increaseof17.5pppretopost
test=significantat95%confidencelevel
• Largeeffect(1.44sd).
Teacher/prac''onertestresultsPreTestresults(n=157)• Meanscoreof51.4%.• Scoresrangedfrom16%to88%.
32
PostTestresults(n=157)• Meanscoreof68.1%.• Scoresrangedfrom28%to96%.• Increaseof16.7pppre-posttest• Significant,largeeffect(1.37sd).
Learnerfindings:UrbanDistrict1
33
• Mostlearners(≈70-80%)wereassessedinEnglish.• AsmallminorityofthesewereELLs(i.e.wereassessedinEnglish
butwerenotEnglishHL)• AlmostallremaininglearnerswerelearningMathemaEcsintheir
HLofisiXhosa.
Notsignificant
Learnerlevelfindings:RuralDistrict2
34
• Mostlearners(≈75%intheintervenEongroup,and100%inthecomparisongroup)wereassessedinAfrikaans.
• AllremaininglearnerswerelearningMathemaEcsintheirHLofisiXhosa.
Significant,withasmalleffectsize.
Learnerlevelfindings:Generallinearmodelofwholesample–setup
Fixedeffects:- Baselinetestscores,- District(urban,rural),and- Group(intervenEon,comparison)
Dependentvariables:- Marko-Dendlinetotalscores,and- Marko-Dendlinescores,bylevel(L1,L2,L3,L4,L5)
Factorsincluded:- Gender(Male;Female),- QuinEle(1;2;3;4;5),- LoLT(E,A,X),and- Age
35
Generallinearmodelonwholeresult:findings1
1. Greatest(medium)effectsonMarko-Dperformancewere:a. LoLT(isiXhosa-andAfrikaans-speakersimprovedbest;
English-speakersleast)–foralllevelsandtotal,andb. District(urbanlearnersimprovedmore,atL3toL5,and
total)2. Weaker(small)effectonMarko-Dperformancewere:
a. Group(intervenEongrouplearnersperformedbeQeratL2,L3andL5andonwholetest),and
b. Age(olderlearnersperformedbeQeratL2,L3andL4)3. NosignificanteffectsonMarko-Dperformancefor:
a. Genderb. QuinEle
36
Generallinearmodelonwholeresult:findings2ImpactofgroupInterven'ongrouplearnersperformedbeQeronwholetestandatL2,L3andL5
37
Generallinearmodelonwholeresult:conclusions• BiggesteffectswereonLevels2and3oftheMarko-D.
• Grade R children in the intervenEon groupwere performingsimilarlytothoseGradeRlearnersinthecomparisonschoolswhoweresixmonthsolder(whenassessedatLevels2and3ofMarko-D).
• ExpecEnglargedifferencesinimprovementsinlearnerscoresinashortspaceofEmeisunrealisEc.
• Overall,therefore,thefactthattheR-MathsintervenEonhada generally small but posi've effect on the MathemaEcsresults of childrenwhose teachers had been exposed to theintervenEonisencouraging.
38
5.Whenisaneffectonlearneroutcomes,goodenough?
IsastaEsEcallysignificantdifferencebetweenintervenEonandcomparisonenough?
Isitenoughwhenthedifferencesareonlyafewpp?IsitenoughwheneffectisaboutafiihofaSD?Is‘6monthsgain’,relaEveto‘ageadvantage’,good?Isitgoodover10monthsinGradeR?
WhataboutwashoutoverEme?
Meta-analysesofimpactofinterven'ons• EducaEonintervenEonsinGr1-3haveaverageeffectsizeof
0.18SD(Hill,Bloom,BlackandLipsey,2008)
• Sub-SaharanAfricanmetaanalysis(Conn,2017):• OveralleffectsizeforalleducaEonintervenEontypes:0.18SD• EducaEonintervenEonsfocusingonpedagogy:0.92SD
ComparisonswithfindingsfromotherSAstudies
1.ELOM(50-59and60-69month)Differencebetween>5y.oand<5y.onormsforemergentMathsisonly2.5pp
2.Readingcatch-upresearchproject(Grades1-3)ThestudyshowedanoverallimprovementfrombothcontrolandintervenEonschools,withintervenEonschoolsshowingaslightlybelerperformanceparEcularlyinspecificareasofreading(spellingandgrammar)
3.WCEDLitNumGrade1Numeracymean:4pp:27%(2009)to31%(2012)
4.E-Lit(2016lit/langinterven'oninGradeRacrossGrR)6monthsaPertheintervenEontrainingbegan(Mid-GradeR):0.41SDEndofGradeR:0.24SDEndofGrade1:Nosignificantdifference.
41
6.Sowhat?
WasR-Maths“successful”?WhatdoesthisR-MathsexamplehighlightaboutoureducaEonresearchlandscape?
42
43
Achieved
Partially Achieved
Not achieved
Issuesthathinder
1. QualitaEveandquanEtaEverealmsareatEmesdifferentuniverses
(cold&clinicalnumbersvswarm&fizzypeople)2. Enoughofpilots.Wecaneasilychangeafewschools.Butweneedto
getchange“atscale”.Butthen,wegobig.Immediately.Withouttakingintoaccountthestability/robustnessofourtreatment(whichissEllbeingconceptualisedand/ortheorised)
3. Changeisurgent.Thingsaredire.Sowemustimplementnow.Thebudgetisforthisfinancialyear…onlyinthisadministraEveterm…
4. Tomeasure:RCTis“goldstandard”(irrespecEveofthephaseofintervenEondesign…)
5. TeachingCAPSandnotchildren.Tightlymonitoringcomplianceonform(“supporEng”)daily/weekly/termlycurriculumpaceand‘coverage’.
44
Whereourresearchisimproving
45
Bogstandard(now…):• Enrolment(grossandneQ),drop-outs,throughput,NCS
aQainment• Totalcohort(withGHSandDBEdata)• Inequality(class:school,household,child;race,gender)
Increasingly:• Theoriesofchange,• IntervenEoninput:level,purposeand‘form’,(egcurriculum
coverage,dosage,fidelityof“uptake”)• Learneroutcomes:curriculum-basedandinternaEonal
assessments,preandpost-intervenEonorchangeoverEmeSeldom:
• Delayedposttests,or• TrackingofmorethanoneacademicyearforSAs,teachersand
learners
Whatdoweares'llnotdoingwell–notmeasuringordescribing:
• Joined-upfamilyservicesinECD• Levelofthesystemandwhy• Costs&costeffecEveness• DifferenEaEon/remediaEonatschoollevel• Sensiblesamplingforsystem-widefeedbackonlearning• OurinstrumentaEonforstandardisedmeasurementforlearning
outcomes/knowledge• Validityoftheinstrumentsweuse• Standard/commonassessmentscrossstudies• Comparabilityofourinstrumentsacrosslanguages
• The‘substance’andnotjusttheform,ofourintervenEons46
Whatisthe‘substance’?
Ourtheoryoflearning:• Howourchildrenofthisagelearnx?• Ourchildren’sexpecteddevelopmentaltrajectoryforx?Ourpedagogictheory:• OurenvisagedandactualroleandcapacityofourSubject
Advisorsforx(whatdotheydo,knowandfeelaboutx?)• DiQoforHODs:Whatdotheydo,knowandfeelaboutx?• DiQoforteachers:Whatdotheydoknowandfeelaboutx?OurREALISTICexpectedlearningoutcomesforchildrenandcaringadults:• Knowledgeforx–eg“teachingmathstoGradeRs(inthis
schoolcontext)”
